Rendered at 20:46:25 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nojs 17 hours ago [-]
We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
calgoo 14 hours ago [-]
I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed.
The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.
mobelkh 12 hours ago [-]
what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?
tacomagick 11 hours ago [-]
Moving from CC I can agree OpenCode seems very simple. I don't see what is too much about it honestly.
verdverm 6 hours ago [-]
look into the prompts, there are some choices in there
disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi
pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level
esperent 6 hours ago [-]
> look into the prompts, there are some choices in there
Such as?
verdverm 5 hours ago [-]
Instructing to use the `gh` CLI, not ideal for those of use using BBC at work
They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
esperent 4 hours ago [-]
What is BBC? Other than the tv network.
4 hours ago [-]
verdverm 4 hours ago [-]
Bitbucket Cloud (Atlassian)
sn0n 15 hours ago [-]
A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.
sandeepkd 13 hours ago [-]
Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses.
At this point when all the models have been trained on all available data with the similar algorithm,
1. either you get more data which is not feasible,
2. or get a better algorithm - a possibility ,
3. or write a more targeted harness.
Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task
I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.
artdigital 7 hours ago [-]
Very curious as well to see comparisons between Claude Code, Codex, Pi, Grok, OpenCode, Crush, etc
Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most
They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade
ebrahimisoheil 9 hours ago [-]
[flagged]
leemysw 15 hours ago [-]
[flagged]
general_reveal 13 hours ago [-]
Any harness will always be privately and secretly shaped and created by those selling models, especially coding models. It is literally a “selling point”, and unless the government steps in to oversee the tests like in the car industry, then there is absolutely no way lizard satanists like Altman and Musk are going to exercise their native ethical traits (“native” loosely assumes they procured it divinely and quite recently, because,
we simply haven’t observed it prior). Short of that, these tests will be fabricated, a lot, for money.
Tell a horny monkey not to jerk off. How the fuck … would that even be possible? God, only God can stop this godless train.
There is no sincere discussion to be had here. HN has been a cesspool for marketing and it reeks in here lately.
Edit: I am not punching down, it’s gotta be crooks from those companies down-voting.
lukax 15 hours ago [-]
What matters more is that you use the tools that the target model was fine-tuned on.
E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).
It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.
So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.
raffael_de 13 hours ago [-]
> or make them similar.
identical. similar is what you tried and it didn't work.
kouteiheika 12 hours ago [-]
No. You don't need identical. For example, I have extra arguments in my edit file tool that the model can specify, and that works just fine.
dns_snek 10 hours ago [-]
Backwards compatible then.
drdexebtjl 4 hours ago [-]
In practice, if you’re using both Anthropic and OpenAI models, how can you configure Pi (or any other harness) to use these native tools?
I’ve seen this before and the conclusion ends up being “use the native harness”, which I don’t think is an acceptable solution.
lukax 4 hours ago [-]
OpenCode checks model name and registers the appropriate tools.
Pi uses its own tools, like Armin wrote in the linked article.
sn0n 15 hours ago [-]
Doesn’t that have more to do with the templating of tool-calls and how using them are presented to the models?
Or is that just why my model likes to break out of the sandbox, going strait to exec shell command and editing files using python on the cli?
nativeit 15 hours ago [-]
I’m not really a dev, so hefty pinch of salt with this take, but doesn’t this feel like we’re just inventing new “fuzzy” regex with much more required compute?
I can understand this take 4-5 years ago but I have no idea how that's your position in 2026
imtringued 13 hours ago [-]
This is correct. People seem to get the wrong idea about why agentic coding is even a thing in 2026. The naive AI techno optimist which has basically displaced the vast majority of opinions on HN, thinks that the models got "smarter" [0]. No, the training distribution shifted towards training on agentic sessions which made certain forms of agentic coding "in-distribution".
We are still witnessing the same underlying problems of transformers.
[0] Think back to all the publicity stunts like the Hugging Face. They are meant to convince you that the agents have somehow progressed past the transformer limitations when those publicity stunts are actually expressions of transformer limitations.
wollowollo 8 hours ago [-]
There's a widespread perception that models are worse for prose and creative writing now. That would track.
TedDoesntTalk 13 hours ago [-]
You think the hugging face incident was a stunt? Can you explain?
atwrk 8 hours ago [-]
OpenAI started fearmongering way back with GPT 2, arguing that model was too dangerous to release freely. That model was barely coherent enough for using it as a twitter bot. Anthropic just hopped onto that later. Conveniently, calling for regulation now would ease the competition from open Chinese models, opening the chance for both companies to eventually reach positive ROI, with consumers paying the price.
The burden of proof that this isn't just a publicity stunt again is squarely on them.
nottorp 5 hours ago [-]
[dead]
slopinthebag 4 hours ago [-]
yea, plus they have been rlhf'ed to an inch of their lives as well. hard to tell if frontier models can solve more problems because of that or not.
corv 19 hours ago [-]
My own findings are in line with this research:
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
ygouzerh 14 hours ago [-]
One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output.
It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.
Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.
After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent
corv 13 hours ago [-]
I believe Hermes offers something comparable now.
Personally, I’m not convinced that a cascade of LLM judges lets me sleep soundly at night.
My agents stay in their sandbox and escalating changes to that env or prod is solely in the (human) operator’s domain.
Obviously this is less convenient but I think it’s a necessary tradeoff for security and correctness.
faangguyindia 10 hours ago [-]
I believe it simply uses Sonnet as a classifier.
yearolinuxdsktp 6 hours ago [-]
Codex has an “Approve for me” mode (“Guardian” internally) that does the same now.
anon373839 11 hours ago [-]
> Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.
corv 7 hours ago [-]
I initially went with OpenCode's v2 client-server rewrite until I noticed how that wasn't ready for prime time.
Turns out I didn't need a fancy TUI at all when desktop, web and mobile interfaces are already provided by or available for Hermes.
Otterly99 7 hours ago [-]
The most astonishing thing to me is that Pi harness is basically as efficient as the Codex/Claude.
I wonder if the same is true for the smaller models in the 9-32B range? I would expect that these models need more steering, but again I was not expecting this result either.
illiac786 38 minutes ago [-]
It’s more efficient, no? Token wise at least.
Supermancho 20 hours ago [-]
The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc.
Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.
It is a starting point.
corv 16 hours ago [-]
While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.
visarga 12 hours ago [-]
A big part of that responsibility can be put in code tests.
Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.
A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.
So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.
grigio 2 hours ago [-]
I also test harness with "Ship Harness Bench", jcode with its browser integration gives the best results
Yashjain413 16 hours ago [-]
I think it’s really important, especially when you look at everything the tool does, from the execution loop and context management to feedback. The harness is basically the underlying source of truth.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
x312 19 hours ago [-]
Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).
In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
vb-8448 5 hours ago [-]
The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.
pllbnk 3 hours ago [-]
Anthropic is the only provider trying to lock in their users by using the dirty tactics:
- Prevent subscription usage on third party harnesses
- Append "Co-authored by Claude ..." to commit messages
mikeocool 5 hours ago [-]
I believe you can use your codex subscription with Pi, and it’s officially endorsed by OpenAI and uses your subscription usage.
Though for cc you are correct, using it with your sub draws from extra usage.
wyre 5 hours ago [-]
Pi supports OpenAI login for ChatGPT subscriptions.
Every 3rd party open-source harness I know of supports ChatGPT subscriptions.
vb-8448 3 hours ago [-]
Isn't this against TOS of OAI?
wyre 3 hours ago [-]
No. It’s against Anthropics TOS though.
It isn’t explicitly allowed in OAI’s TOS, however they publicly support Pi and OpenCode’s usage of their Oauth, and because Codex is open-source, it means the machinery to support Oauth login is open-source under Apache 2.0
logicchains 5 hours ago [-]
Deepseek harness can if using Deepseek V4.1 Flash from the official API, it's cheap enough to be competitive even without a subscription.
spike021 19 hours ago [-]
Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.
Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?
CaveTech 17 hours ago [-]
Your question is worded kind of confusingly, but all caching is handled on the inference layer, and by all major providers. In short, caching should work as long as you are sending requests to the same model and provider.
spike021 16 hours ago [-]
Apologies. I guess my main question is just is there a benefit to routing directly to a provider through its own harness vs just sending chat requests to those providers using OpenRouter or similar as a proxy?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
vintagedave 14 hours ago [-]
> Models may perform better with other harnesses than with their own. So it turns out that your Claude models may not need Claude Code…
This is interesting. We built our own harness (CodeBot, an agent for Delphi) and it currently uses OpenAI models; we tuned it for specific behaviours and patterns and I find its behaviour better than Codex. Same models underneath.
nichyjt 7 hours ago [-]
it seems like accuracy isn't affected much by the harness. can someone help me understand what's the value or the secret sauce in closed-source harnesses like claude code then?
ed_mercer 20 hours ago [-]
Does this extend to open models like GLM 5.3? This would mean that simply changing the harness to Pi reduces cost in half?
In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.
ebrahimisoheil 9 hours ago [-]
[dead]
roywiggins 19 hours ago [-]
I've found the experience of using Pi with local models feels a lot snappier than both OpenCode or Claude Code.
lexicalmathical 18 hours ago [-]
If this is mostly because of the size of the system prompt, then perhaps in long horizon tasks the "tax" will be less obvious.
snehesht 16 hours ago [-]
I have been using jcode for past two weeks, honestly I feel its much better compared to Opencode.
mikert89 19 hours ago [-]
As the model gets smarter, you need to tell it less
big-chungus4 13 hours ago [-]
It goes the opposite way too. Small models need minimal harness like pi or they get overwhelmed by long system message and tool descriptions
NewJazz 19 hours ago [-]
As the black box gets bigger, you need to stop expressing yourself even more.
malnourish 18 hours ago [-]
Perhaps, but then the value of each thing you tell it goes to.
edgyquant 8 hours ago [-]
No not really, it’s able to fill it more gaps for sure but the way it fills them is totally random and almost always not what you want it to do
sn0n 15 hours ago [-]
It matters about as much as where you leave the electrician, he’s gonna use what tools he has to get the job done with what he has.
kittikitti 7 hours ago [-]
I find that in many cases, you ARE the harness. Case in point, Terence Tao utilized the simple chat interface to find the Jacobian Conjecture counterexample. This is arguable no harness at all. I also find that many people will disagree with me but most of the time, they just want a button to press that will solve the problem. If that's the work you'd like to accomplish, then the harness matters, but as the article alludes to, it doesn't need to be complex.
jedbrooke 5 hours ago [-]
I like to say with LLMs the real intelligence was inside us all along.
That said, system prompt (which to your point also comes from the human) matters a lot too. For example a coding harness that doesn’t a have a plan mode will keep trying to write underbaked ideas to code instead of working with the user on the plan. That said, a “system prompt” can also be the first prompt you send to the agent in a vanilla chat window, so I do like your point about the user being the real harness
robinpie 19 hours ago [-]
claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product
nativeit 15 hours ago [-]
I’m still baffled as to where these 100x valuations came from. Amazon’s market cap is $2.65T, and I don’t see how any AI company comes within a faint whiff of that sort of value.
CharlieDigital 8 hours ago [-]
Claude Code (CLI) has been really bad with handling window resizing and layout changes. For long sessions, it will freeze while redrawing. On an M3 Max. With 64 GB or memory. Codex CLI does not have this problem, nor does OpenCode. Every time I split the pane, I'm dumbfounded by how bad this is.
satnhak 4 hours ago [-]
Claude Code is an absolute dog. They hijack the scrollbar. Keybindings don't work properly on WSL. A jumble sale of slash commands that are almost all useless. Stupid flair like "Boondoggling...". I despise everything about it.
Compare this to the Codex CLI that is no fluff, minimal and just does it's job. Open AI models are in a totally different league to Anthropic (in the real world) so no loss really.
topaz0 17 hours ago [-]
It's pretty great to not use, conversely.
jswelker 20 hours ago [-]
Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.
"Why pay the waste disposal tax? Dumping into the ocean is free!"
Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.
Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.
roywiggins 19 hours ago [-]
It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens.
(I do think Pi should ship with more than zero builtin sandboxing though)
zackify 18 hours ago [-]
I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic.
Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.
skirmish 14 hours ago [-]
I tried putting my agent in an LXC / Incus container but then I also needed it to process videos stored on a NAS, and that was not easy at all. I tried a FUSE-based NFS client [1] but it was very flaky and would randomly report files having zero length and fail processing.
I'm looking at changing my AI harness and am definitely interested in this
seanmcdirmid 19 hours ago [-]
You can sandbox with tool hooks, at least in antigravity, I think a lot of other harnesses support similar features. Sandboxing via the prompt is just dangerous and unreliable.
imtringued 13 hours ago [-]
The developers of pi.dev are geniuses.
They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via
pi --no-tools --tools tool1, tool2, tool3
except..., that's not enough to sandbox pi.dev, because remember, it's a minimalist coding agent! So what does a minimalist coding agent do? Of course! It loads extensions by default and do you know what extensions can do? they can add default tools that bypass --no-tools!
So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.
Here is how to run pi.dev under its most minimal configuration under a sandbox:
pi --no-tools --no-extensions --tools tool1, tool2, tool3 -e ./your-sandboxing-extension
Claude Code's initial instructions to a model are a dump of 80kb of text. And none of it has anything to do with security. The "security" is handled by the evaluator model that sometimes denies some tool uses. I'm sure that in itself can be done very cheaply.
sandeepkd 13 hours ago [-]
Wonder if you have any source to back that data, unless you are referring to the security code to detect the user and their usage itself to prevent against unauthorized usage patterns?
kouteiheika 13 hours ago [-]
> Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes.
Um, no? Have you actually read Claude Code's system prompt and tools?
There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.
The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).
shshsjsj 19 hours ago [-]
[dead]
epolanski 9 hours ago [-]
Okay but...closed source harnesses change multiple times per week, sometimes per day.
And they also tune the inference of the models behind it.
You're using different tooling every day. Hard to benchmark.
edgyquant 8 hours ago [-]
A benchmark would be a series of prompts + tests that check their results you theoretically run every day
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.
disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi
pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level
Such as?
They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
At this point when all the models have been trained on all available data with the similar algorithm,
1. either you get more data which is not feasible,
2. or get a better algorithm - a possibility ,
3. or write a more targeted harness.
Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task
I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.
Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most
They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade
Tell a horny monkey not to jerk off. How the fuck … would that even be possible? God, only God can stop this godless train.
There is no sincere discussion to be had here. HN has been a cesspool for marketing and it reeks in here lately.
Edit: I am not punching down, it’s gotta be crooks from those companies down-voting.
E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).
It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.
So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.
identical. similar is what you tried and it didn't work.
I’ve seen this before and the conclusion ends up being “use the native harness”, which I don’t think is an acceptable solution.
const usePatch = model.modelID.includes("gpt-") && !model.modelID.includes("oss") && !model.modelID.includes("gpt-4")
Pi uses its own tools, like Armin wrote in the linked article.
Or is that just why my model likes to break out of the sandbox, going strait to exec shell command and editing files using python on the cli?
not RegEx ... more like a "fuzzy" DSL.
I can understand this take 4-5 years ago but I have no idea how that's your position in 2026
We are still witnessing the same underlying problems of transformers.
[0] Think back to all the publicity stunts like the Hugging Face. They are meant to convince you that the agents have somehow progressed past the transformer limitations when those publicity stunts are actually expressions of transformer limitations.
The burden of proof that this isn't just a publicity stunt again is squarely on them.
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.
Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.
After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent
Personally, I’m not convinced that a cascade of LLM judges lets me sleep soundly at night.
My agents stay in their sandbox and escalating changes to that env or prod is solely in the (human) operator’s domain.
Obviously this is less convenient but I think it’s a necessary tradeoff for security and correctness.
That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.
Turns out I didn't need a fancy TUI at all when desktop, web and mobile interfaces are already provided by or available for Hermes.
I wonder if the same is true for the smaller models in the 9-32B range? I would expect that these models need more steering, but again I was not expecting this result either.
Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.
It is a starting point.
Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.
A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.
So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
- Prevent subscription usage on third party harnesses
- Append "Co-authored by Claude ..." to commit messages
Though for cc you are correct, using it with your sub draws from extra usage.
Every 3rd party open-source harness I know of supports ChatGPT subscriptions.
It isn’t explicitly allowed in OAI’s TOS, however they publicly support Pi and OpenCode’s usage of their Oauth, and because Codex is open-source, it means the machinery to support Oauth login is open-source under Apache 2.0
Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
This is interesting. We built our own harness (CodeBot, an agent for Delphi) and it currently uses OpenAI models; we tuned it for specific behaviours and patterns and I find its behaviour better than Codex. Same models underneath.
In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.
That said, system prompt (which to your point also comes from the human) matters a lot too. For example a coding harness that doesn’t a have a plan mode will keep trying to write underbaked ideas to code instead of working with the user on the plan. That said, a “system prompt” can also be the first prompt you send to the agent in a vanilla chat window, so I do like your point about the user being the real harness
Compare this to the Codex CLI that is no fluff, minimal and just does it's job. Open AI models are in a totally different league to Anthropic (in the real world) so no loss really.
"Why pay the waste disposal tax? Dumping into the ocean is free!"
Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.
Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.
(I do think Pi should ship with more than zero builtin sandboxing though)
Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.
[1] https://github.com/facebookarchive/nfusr
I'm looking at changing my AI harness and am definitely interested in this
They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via
except..., that's not enough to sandbox pi.dev, because remember, it's a minimalist coding agent! So what does a minimalist coding agent do? Of course! It loads extensions by default and do you know what extensions can do? they can add default tools that bypass --no-tools!So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.
Here is how to run pi.dev under its most minimal configuration under a sandbox:
Pretty neat, huh?Sources: https://github.com/earendil-works/pi/issues/555
Um, no? Have you actually read Claude Code's system prompt and tools?
Here's Fable's system prompt: https://github.com/navanchauhan/agent-autopsy/blob/6d9c00e54...
There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.
The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).
And they also tune the inference of the models behind it.
You're using different tooling every day. Hard to benchmark.