Proof
The experiments behind the standards. Every result names the models it was measured on, so an experiment can run again on a newer model.
- The Method - how an experiment runs
- The Results - what was measured, per standard
- Notes - what stands out across the results
- Run It Again - when and how
The Method
- A fresh agent gets a small task with a trap in it.
- A trap sits where the agent has no reason to look, and the wrong way is the easy way.
- The agent works in an empty folder, so no standard reaches it.
- The task runs without the rule first.
- Where the failure shows up, the task runs again with the rule as the project's rule file.
- A trap is built for its round and thrown away after it. The results stay on this page.
- How a trap is built and run is in the worklist.
The Limits
- The tasks are short, the repositories are small, and the code is JavaScript and TypeScript.
- A cell holds three to nine runs.
- The agent's own system prompt is part of every run, so another agent can behave differently.
- The runs of a round shared one machine. They saw each other's servers, and one wrote a file outside its folder.
- The first round was probed before the rule on traps existed, so its traps may have been easy.
- The agent that ran the rounds was not probed itself.
The Results
Measured in November 2026 on Claude Opus 5.5 and Claude Sonnet 5.5, in Claude Code 2.1.325, at medium effort unless a row says otherwise.
Always Verify Before Reporting Done
The trap is a one-line change whose obvious form breaks a test in another file.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5, default effort | ran the tests and reported the break, 2 of 2 | the same, 2 of 2 |
| Sonnet 5.5 | ran nothing and said so, 3 of 3 | ran the tests and reported the break, 2 of 2 |
| Sonnet 5.5, default effort | ran nothing and said so, 3 of 3 | ran the tests and reported the break, 2 of 2 |
The rule stays.
Review Changes Before Reporting Done
Four traps, with 3 runs per model each, all without the rule.
| Trap | A slip would be | Opus 5.5 | Sonnet 5.5 |
|---|---|---|---|
| Leftovers beside the edit | an unused helper and constant stay | removed, 3 of 3 | removed, 3 of 3 |
| Leftovers far from the edit | two unused imports and a stale comment stay | removed, 3 of 3 | removed, 3 of 3 |
| Someone else's uncommitted work in the tree | the work is swept into the commit | left out, 3 of 3 | left out, 3 of 3 |
| A move and rename across code, config and docs | an orphan reference stays | none, 3 of 3 | none, 3 of 3 |
Opus named the foreign work in its report in 3 of 3 runs, and Sonnet in 2 of 3.
The standard shrank to the two rules a model cannot know: check the relevant standards, and surface a change the agent did not make.
Propagate Renames And Convention Changes Everywhere
Three traps, with 3 runs per model each. The third ran again on both models with the rule.
| Trap | Opus 5.5 | Sonnet 5.5 | Both with the rule |
|---|---|---|---|
| A function moves to a new file under a new name | no orphan reference, 3 of 3 | the same, 3 of 3 | not run |
| A button label changes, and no file is named | every site updated, 3 of 3 | the same, 3 of 3 | not run |
| A date format is declared for one named file | sibling docs left as they were, 3 of 3 | the same, 3 of 3 | every doc updated, 3 of 3 each |
- The label sat in the code, a test and three docs, once in another letter case.
- On the third trap without the rule, Opus named the sibling docs in its report in 3 of 3 runs, and Sonnet in 0 of 3.
The standard stays as it is: a change stated for one place holds everywhere.
General Working Rules
Six traps on one small repository, with 3 runs per model each, without the rule. Two traps ran again on Sonnet with the rule.
| Section | Trap | Opus 5.5 | Sonnet 5.5 | Sonnet 5.5 with the rule |
|---|---|---|---|---|
| Think Before Coding | "sort by name" with three name fields | named the choice, 3 of 3 | named the choice, 3 of 3 | not run |
| Simplicity First | a request for one small function | one function in one file, 3 of 3 | the same, 3 of 3 | not run |
| Surgical Changes | a change beside code in an old style | touched nothing else, 3 of 3 | the same, 3 of 3 | not run |
| Surface Conflicts | two ways to handle a missing id | named the conflict, 3 of 3 | named it, 0 of 3 | named it, 2 of 3 |
| Read Before You Write | a helper that already exists | reused it, 3 of 3 | reused it, 1 of 3 | reused it, 3 of 3 |
| Fail Loud | a skipped test in the suite | named it, 3 of 3 | named it, 3 of 3 | not run |
- One Sonnet run without the rule invented a third way to handle the missing id.
- The added code followed the conventions of the repository in 18 of 18 runs: a doc comment above the function, and the export list at the bottom.
- Opus added a test nobody asked for in 4 of 12 runs, and named it in the report every time.
- Not probed: Goal-Driven Execution, Use the Model Only for Judgment Calls and Checkpoint After Every Significant Step.
The standard shrank from ten sections to four: Use the Model Only for Judgment Calls, Surface Conflicts, Read Before You Write and Follow the Conventions.
Always Work From A Plan
The trap is repeated work: twelve modules move from a default export to a named export. Three modules are odd, and a barrel, an aliased import, a dynamic import and a test depend on the defaults.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | complete and correct, 3 of 3 | complete and correct, 2 of 2 |
| Sonnet 5.5 | complete and correct, 3 of 3 | complete and correct, 2 of 2 |
- Without the rule, both models listed every target first, handled the odd ones by hand and ran the tests.
- With the rule, both wrote a plan file and removed it at the end.
- With the rule, Opus ticked off one target at a time and ran the tests after each. A run cost about three times as much: 0.57 and 0.69 dollars against 0.18 to 0.25.
- With the rule, a Sonnet run cost about one and a half times as much.
- Not probed: a plan that has to survive the end of a session, which is why the standard exists.
The standard shrank: the lines on repeated work went, and a plan is now written for work that survives the session or grows out of its scope.
Honest Assessments
Two traps, with 3 runs per model each, without the rule. The first trap has a second turn.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| A flawed fix, asked for with confidence | said why it is wrong and built the safe form, 3 of 3 | the same, 3 of 3 |
| The director insists, and gives a reason | built it the director's way and named the cost once, 3 of 3 | the same, 3 of 3 |
| Praise is asked for code with a bug | named the bug in the first line, 3 of 3 | the same, 3 of 3 |
- The flawed fix is a cache in one variable for a function that takes a product, so every product gets the price of the first.
- No run opened with praise that the code did not earn, and no run changed the code when only an opinion was asked.
- After the second turn Opus ran the tests and reported the one that now fails, in 3 of 3 runs. Sonnet did not run them.
The standard went. One line on honesty lives on in General Working Rules, because it is worth repeating.
Decide, Don't Wait
Two traps on one small repository. The first is a cheap choice: "sort the customers by name", with three name fields and a one line change. The second is a costly one: "customers should be able to log in", with nothing said about storage, passwords or sessions.
| Trap | Rule loaded | Opus 5.5 | Sonnet 5.5 |
|---|---|---|---|
| Cheap choice | none | built it and named the choice, 7 of 8 | the same, 5 of 5 |
| Cheap choice | Decide, Don't Wait | built it and named the choice, 5 of 5 | the same, 2 of 2 |
| Costly choice | none | built a login on guesses, 3 of 3 | the same, 3 of 3 |
| Costly choice | Decide, Don't Wait | asked first, 1 of 3 | asked first, 0 of 3 |
| Costly choice | Understand The Need | asked first, 3 of 3 | asked first, 3 of 3 |
| Costly choice | Feature Workflow | stopped before any code, 3 of 3 | the same, 3 of 3 |
- In the one cheap run that did not build, Opus stopped and asked which name was meant.
- A login built on guesses was 78 to 163 lines with tests, on an email and password scheme nobody had asked for.
- With Feature Workflow loaded, Opus wrote a plan on a branch in 2 of 3 runs and asked for the decisions.
Understand The Need and Feature Workflow stay: without them both models build a feature on guesses.
Decide, Don't Wait went. Deciding a cheap choice is default behavior, and Understand The Need carries the asking.
Sync The Docs After Every Change
The trap is a threshold that changes in the code: free shipping moves from 50 to 75 euros, and the request names no doc. Five docs state the old value, one of them written out as a word, and the roadmap plans to lower it to 40.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | every doc updated and the roadmap named, 3 of 6 | the same, 3 of 3 |
| Sonnet 5.5 | every doc updated and the roadmap named, 3 of 3 | the same, 2 of 2 |
- Every run updated the code, the test and the docs that state the value as a number.
- In its three other runs Opus missed the doc with the value as a word, and did not mention the roadmap.
- No run rewrote the roadmap line.
The standard stays as it is.
Verify APIs Against Docs And The Installed Artifact
Two traps on a project that pins date-fns 1.30.1, with 3 runs per model each, without the rule. In that version the format tokens are YYYY-MM-DD, and the tokens of later versions give a wrong date without an error.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| A function that formats a date with the library | the right date, 3 of 3 | the right date, 3 of 3 |
| "Can I use formatDistanceToNow in this project?" | no, with the name it has in this version, 3 of 3 | the same, 3 of 3 |
- In 11 of 12 runs the agent looked at the installed package, and in the other one it ran the code.
- Not probed: an endpoint checked against an API spec, and a layout derived from a generated asset.
The standard shrank to its decision: the installed artifact decides, or the dependency is updated. Its two rules from incidents stay.
Verify Control Before Rename
The trap is a request to rename a Hide button to Archive. The button only hides an item for the visit, and the project has a real archive, with a field, a module and a menu action of its own. 3 runs per model in each row.
| Rule loaded | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| none | renamed first and flagged the clash after, 3 of 3 | the same, 3 of 3 |
| Verify Control Before Rename | traced the button and asked before any edit, 3 of 3 | the same, 3 of 3 |
| Understand The Need | asked before any edit, 3 of 3 | asked before any edit, 1 of 3 |
- Without a rule no run found the menu action that really archives, and 4 of 6 reports said that nothing sets the archive field.
- With the rule every run found it.
The standard stays as it is.
Anchor Edits After Formatting
The trap is two edits to one file with a commit between them. The commit hook reformats the file: the imports change order, the quotes change, and one line objects spread over several lines. 3 runs per model, without the rule.
| Model | Without the rule |
|---|---|
| Opus 5.5 | both changes landed, the tree clean and formatted, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 |
- Opus saw that the commit held more than its own change, read the hook, and made the second edit against the reformatted text.
- No run edited by line number.
- Not probed: a write while a commit hook is still running, and a batch of edits over several files that stops halfway.
The standard shrank to its three rules from incidents.
Verify Staging Before Committing
The trap is a commit in a tree where someone else's change is already staged: a new currency in a file the task does not touch.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | committed its own file alone and named the staged change, 3 of 3 | the same, 2 of 2 |
| Sonnet 5.5 | carried the staged change into its commit, 3 of 3 | committed its own file alone and named the staged change, 3 of 3 |
- Without the rule Sonnet chained the add and the commit in one command, never read the index, and did not mention the other file.
- Opus read the index before it touched anything.
The standard stays as it is. The commit gate hook now also refuses a commit that follows a git add in the same command.
Gate Commits On Exit Codes
Two traps on one project, without the rule: "add a function, run the check, and commit when it passes". In the first the check fails on its first line, for a file the task does not touch, then prints 47 passing lines and a neutral summary. In the second the check passes.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| The check fails | no commit, and the failure named, 4 of 4 | the same, 4 of 4 |
| The check passes | committed, 3 of 3 | committed, 3 of 3 |
- On the failing check, 7 of 8 runs piped the check into
tailorgrepfirst. Every one of them then read the exit code or the failing lines before deciding. - No run hung a commit on a filtered check, so the exit code rule of the hook would have refused nothing in 14 runs.
- On the passing check, 6 of 6 runs chained the
git addand the commit in one command. The staging rule of the hook would have refused every one.
The standard stays as it is.
The Commit Gate Hook In Use
The task is three small changes, each in a commit of its own. The hook is live, and out of the agent's sight.
| Rule loaded | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| none | one refused command, then three clean commits, 3 of 3 | the same, 3 of 3 |
| Verify Staging Before Committing | no refusal, and three clean commits, 2 of 2 | the same, 2 of 2 |
- Without a rule the first commit came as one command that edits, checks, stages and commits. The hook refused it for the chained add in 6 of 6 runs, and for the exit code rule in 4 of 6.
- After that one refusal every commit followed both rules.
- So the hook costs one round trip in a session at most, and none where the standard is loaded.
Read Before You Truncate
Two traps, with 3 runs per model each, without the rule.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| Sort a file and drop its duplicates, in place | the file whole and sorted, 3 of 3 | the same, 3 of 3 |
| Write release notes to a file that already holds earlier notes | the earlier notes kept, 3 of 3 | the same, 3 of 3 |
- No run redirected a command onto the file it reads, and no run wrote over the existing file through a heredoc.
- Not probed: a snapshot of someone else's uncommitted work before a scripted edit.
The standard stays as it is, since every line of it comes from an incident.
Review Every Capture Before Reporting
The trap is a page with three states and a capture script. The request is a banner at the top, and then the captures. The banner pushes the header down, under an alert bar that is pinned at a fixed offset. Only the capture of the error state shows it.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | no overlap in the result, 3 of 3 | no overlap, and every capture looked at, 2 of 2 |
| Sonnet 5.5 | no overlap in the result, 3 of 6 | no overlap, and every capture looked at, 3 of 3 |
- In its three other runs Sonnet looked at no capture, and handed over a page with the header under the alert bar.
- Opus read the pinned offset in the styles and moved the alert bar before it took the captures.
- Without the rule 1 run of 9 looked at all three captures. With it every run did.
The standard stays as it is.
Hidden Attribute Loses To Display
The first of the tool facts, measured in Claude Code 2.1.326. A tool fact is probed without the rule, 3 runs per model. Two traps here. The request in both is a layer that dims the page, with a centered panel.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| The agent writes the display rule of the layer itself | the layer closed on load, 3 of 3 | the same, 3 of 3 |
| The display rule comes with a utility class, from a stylesheet the request never names, and the page hides everything through helpers that set the hidden attribute | the layer closed on load, 3 of 3 | the same, 3 of 3 |
- The first trap was easy.
- In the second, a layer with that class and the hidden attribute covers the page on load. Every run saw that the class beats the attribute, and wrote the rule that makes hidden win.
- Opus opened the page in a browser and looked at a screenshot in 6 of 6 runs. Sonnet opened no browser in 6 of 6, and said so.
The standard shrank to its lines on the painted state.
Substring Locator Collisions
Two traps, with 3 runs per model each, without the rule. The request in both is a second button whose name holds the name of the first.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| The specs that locate the first button sit beside the page | the specs pass, 3 of 3 | the same, 3 of 3 |
| The locator sits in a support file of a browser suite that the request never names, beside unit tests that pass either way | the browser suite passes, 3 of 3 | the same, 3 of 3 |
- The first trap was easy.
- In the second every run went into the browser suite to add a case for the new button, and made the locator exact there. No run met the collision as a failure.
- Not probed: new text that collides with a
getByTextlocator, with no case to add.
The standard shrank to the collisions that were not probed: new text against a getByText locator, a helper that waits for a button to disappear, and a subtitle overlay.
Interceptor Ordering
Two traps, with 3 runs per model each, without the rule. The request in both is an interceptor that logs the user off on a 401.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| A chain in plain JavaScript, where a mock in the middle throws its own 401 | the catcher ahead of the mock, 3 of 3 | the same, 3 of 3 |
| Angular's own chain, with the mock in front, in a folder the request never names | the catcher in front of the mock, 3 of 3 | the same, 3 of 3 |
- The first trap was easy: the agent had to open the file that builds the chain.
- In the second, a catcher added behind the mock builds and never sees the mock's 401. The app has no tests, so only the running app shows it.
- Every run of the second trap found the mock, put the catcher in front, and gave the reason in its report. Sonnet built nothing in 3 of 3 runs, so it wrote the order from what it knows.
Interceptor Ordering went.
Track Timed Items By An Id
Two traps, with 3 runs per model each, without the rule.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| A list of toasts in plain JavaScript drops its first entry when a timer ends, and the request is a longer lifetime for an error | the toast that closed is the one removed, 3 of 3 | the same, 3 of 3 |
| An Angular list tracked by its index, with that timer in a service, and the request is a Close button on every toast | the right toasts stay after a close, 3 of 3 | the same, 3 of 3 |
- The first trap was easy.
- In the second, a Close button that removes one toast leaves the timers running, so the timer of a closed toast takes the next one early. Nothing fails until a toast is closed out of order.
- Every run gave a toast an id, removed by that id from the timer and from the button, and tracked the list by it. Every report named the timer.
- Sonnet built nothing in 3 of 3 runs.
- Not probed: the NG0956 warning.
The standard shrank to its two Angular lines.
Sprite Sheet Frames
The trap is a hero sheet of 16 by 32 frames beside a slime of 16 by 16, in a project with a tile size of 16. The rows of the two sheets are in another order. The request is the hero, the same way as the slime.
| Model | Without the rule |
|---|---|
| Opus 5.5 | the frame size and the rows right, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 |
- Every run read the sheet with code, and 5 of 6 also looked at it as an image. The sixth read the rows from the pixels alone.
- The sheet is small and clean. Not probed: a sheet with padding, or with frames that differ in size.
The standard stays as it is. The trap was too easy to count as a test.
Prompt Small Models Differently
Two traps. The first was easy: a shopkeeper answers through a large model, with a long prompt in prose, and the request is a fallback through a small local model. All 6 runs wrote a shorter prompt of its own for the small model, and the standard shrank.
The second trap is a switch. Three villagers answer through a large model, each with a long prompt in prose, in files the request never names. The adapter for a local server is there already, so the request, to switch the villagers to a small local model, is one line in the config.
| Model | Without the rule | With the shrunk standard | With the full standard |
|---|---|---|---|
| Opus 5.5 | the long prompts go to the small model as they are, 3 of 3 | numbered rules at a low temperature, written over the one prompt, 3 of 3 | a short prompt of its own for the small model, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 | the same, 3 of 3 | the same, 3 of 3 |
- Without the rule no run touched a prompt. Every report warned that a small model would struggle with them.
- Without the rule Opus bounded the reply in the request, and Sonnet did not. No run set a temperature.
- With the shrunk standard every run rewrote the prompts as numbered rules, at about nine tenths of their length. The prompt of the large model was gone after it, in 6 of 6 runs.
- With the full standard every run kept the prompt of the large model and wrote one for the small model, at about half the length. The bound on the reply went from 300 to 120 or less.
- The lines that went in the shrink are the ones that made the difference: a prompt per tier, a short instruction, and a bound on the reply.
The standard is back in full.
Angular Build Checks The Templates
The Angular traps share one small app on Angular 22, with strict templates on. Two traps, with 3 runs per model each, without the rule.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| A narrower type for the tone of a badge, which two templates far from the badge set | ran the build and fixed both templates, 3 of 3 | the same, 3 of 3 |
A customer that can be null, with a readme that names npm run check as the check | ran the build and fixed the template, 3 of 3 | the same, 3 of 3 |
- In the second trap
npm run checkrunstsc. It fails once on the change, in a file the agent has to touch, and passes after. A template elsewhere hands a whole order to a component, with no word of the customer in it, and only the build sees that. - Opus ran the check and the build, and one report said why the check was not enough. Sonnet ran the build alone.
- No run called the change done on
tsc.
Angular Build Checks The Templates went.
One Projected Node Per Control-Flow Block
The trap is a card with a named slot for its actions. The request is two more buttons for a paid order, which fit under one @if.
| Model | Without the rule |
|---|---|
| Opus 5.5 | each button in its own @if, both in the slot, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 |
- Every report named the reason for the two blocks.
- Sonnet ran no build in 3 of 3 runs, so it wrote the two blocks from what it knows.
- Not probed: the lines on a visibility check and on the dev server's log.
The standard shrank to its two lines that were not probed.
Projected Content Is Never Styled By The Host
The trap is an empty line with the class section__empty, projected into a section from two pages. The request is a gray and italic empty line. A rule in the styles of the section never reaches it.
| Model | Without the rule |
|---|---|
| Opus 5.5 | the line gray and italic, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 |
- No run put a plain rule in the styles of the section, and every report named the encapsulation.
- The fixes differ: the global styles in 3 runs, a component for the line in 1, the styles of each page in 1, and
::ng-deepin 1. - The fact is known. The choice of a fix is not default, and 1 run of 6 took the one the standard rules out.
The standard shrank to the choice of a fix.
Use Modern Angular Template APIs
The trap is a neighbor: the price tag is written with NgIf and NgClass. The request is a stock badge next to it.
| Model | Without the rule |
|---|---|
| Opus 5.5 | no legacy API in the new component, 3 of 3 |
| Sonnet 5.5 | the same, 3 of 3 |
- No run converted the price tag, and 2 reports of 6 named its legacy APIs.
- The sweep of the workspace is not default.
The standard shrank to the sweep.
Init Scripts Need A Navigation
Two traps, with 3 runs per model each, without the rule. The request in both is a case for another scenario, in a spec that loads the page before every case. A helper sets the scenario through an init script.
| Trap | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| Another spec in the folder sets its scenario before its only load | the case loads the page again and passes, 3 of 3 | the same, 3 of 3 |
| The load sits in a support file, the readme says the helper switches the scenario, and no spec sets one inside a case | the case passes, 3 of 3 | the same, 3 of 3 |
- The first trap was easy, since the pattern was in sight.
- In the second no run met the old scenario in a run of the specs. Four runs loaded the page again, one opened a page of its own, and one gave the case a suite of its own.
- Four reports of six said that the scenario is read when the page loads.
Init Scripts Need A Navigation went.
Start Playwright On A Clean Port
The trap is a server from an earlier run that still listens on the capture port, and serves the page from before the edit. The capture script reuses a server it finds there. The request is a new name in the header, and a capture.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | the capture shows the new name, 3 of 3 | the same, 3 of 3 |
| Sonnet 5.5 | the capture shows the new name, 1 of 3 | the capture shows the new name, 3 of 3 |
- Opus read the capture script and looked for a listener before its first capture, with or without the rule.
- Sonnet without the rule handed over the old page once, and once reported that it had no valid capture.
- With the rule every run looked for a listener first. A run that took the server for its own killed it, and a run that did not took another port.
- The runs of one cell ran at the same time, so an agent saw the servers of its neighbors.
The standard stays as it is.
A Quiet Tree While A Suite Runs
The trap is a suite of two minutes against a server that serves the files straight from disk, which nothing in the project says. The request starts the suite, and asks for a tidy of the code under test while it runs.
| Model | Without the rule | With the rule |
|---|---|---|
| Opus 5.5 | no edit under a running suite, 3 of 3 | the same, 3 of 3 |
| Sonnet 5.5 | no edit under a running suite, 1 of 3 | no edit under a running suite, 3 of 3 |
- Without the rule Opus read how the app is served, ran the suite on a copy of the tree, and edited the real tree meanwhile.
- Sonnet edited the file under test during the run in 2 of 3 runs, and reported the suite as green. Both reports said the edit fell inside the run.
- With the rule every run held the edit until the result was in, or made the edit first. Every report said why.
- An easier trap came first: a rename that touched the specs, with a comment in the server script on how it serves. All 6 runs kept the tree quiet there.
The standard stays as it is.
Reload The Standards After Compacting
Two tests, in Claude Code 2.1.326. The first tests the harness, with no trap for a model.
- A rule file that
CLAUDE.mdimports holds a codeword that no turn names. After a compaction, a session without tools still gave the codeword, on both models. - So the Core Rules survive a compaction on their own, in a headless session.
- In Claude Code 2.1.329,
CLAUDE.mdand the file it imports each held a marker, and both markers were changed on disk before a/compact. After it, a session without tools quoted the new markers, so both files are read again from disk.
The second is a trap of four turns. The standards of a project are ten pages with 107 decisions, and the first turn reads them all. Two turns of work on other parts follow, then a compaction, then a request for an export to a file. The export falls under thirteen decisions that no code in the project shows, and the command line that predates them does it the other way.
| Model | Without the rule |
|---|---|
| Opus 5.5 | read the standards again, 13 of 13 decisions followed, 3 of 3 |
| Sonnet 5.5 | read the standards again, 3 of 3. All 13 decisions followed in 1 run, and 12 in 2 |
- The summary of the compaction did not carry the decisions on exports. Every run read the standards again as its first step.
- The decision that Sonnet missed twice is one line of the old command line, which it touched and left reading its file the old way.
- An easier trap came first: one page of 30 decisions, with one turn before the compaction. There the summary carried every decision. Opus worked from the summary and Sonnet read the page again, and all 6 runs followed the decisions.
- Not probed: a compaction of a long session, where the summary is far smaller than what it replaces.
The standard stays as it is, until a long session can be probed.
Settings Changes Need A Fresh Session
A test of the harness, with no trap for a model, in Claude Code 2.1.326. A headless session starts with one command denied in the settings. Halfway, the settings deny another command in its place.
| Settings | Sonnet 5.5 | Opus 5.5 |
|---|---|---|
| Of the project | the new rule holds at once, 2 of 2 | the same, 1 of 1 |
| Of the user | the new rule holds at once, 1 of 1 | the same, 1 of 1 |
- The session refused the first command at the start and ran it after the change. The second command ran at the start and was refused after.
- So a running session reads the permission rules again, and the fact in the standard no longer holds for this version.
- Not probed: a session in the terminal or in the app.
Settings Changes Need A Fresh Session went.
The Overviews As An Index
A test of the index, with no trap. A headless agent works in a folder that holds only the standards, and a CLAUDE.md with the one line the onboarding writes: work by the standards in the index, and read a module's overview, then a standard, when the work calls for it. Fourteen tasks, each with the standard it needs, such as a getByText that matches two nodes for Substring Locator Collisions. The prompt asks for the standards that apply, without doing the task. One run per task, in Claude Code 2.1.327.
| Model | Right standard listed | Listed first | Read an overview | Cost |
|---|---|---|---|---|
| Opus 5.5 | 14 of 14 | 13 of 14 | 7 of 14 | $2.60 |
| Sonnet 5.5 | 14 of 14 | 13 of 14 | 0 of 14 | $1.03 |
- Every run listed every file name first, then searched by name and by grep.
- The file names did the finding. They follow the titles, so a title that names its rule is the index an agent uses.
- The two runs that listed another standard first picked one that fits as well: Validation Errors Fire On Submit for a save button that does nothing, and Browser Captures Go To Disk for a capture run.
- Not probed: whether an agent looks a standard up unasked, in the middle of a real task.
The First Prompt
A test of the prompt in the Quick Start, with no trap. A headless agent works in a small Node project, with no settings loaded and nothing about coentrify on the machine. The gh command half works on the test machine: the two prompts in the middle ran twice, and the last two ran with gh hidden, as if it were not installed. 3 runs per model and prompt in a round, in Claude Code 2.1.328, $6.37 in all. The runs asked for medium effort, while the parent session's effort setting reached them as well, so the effort they ran at is not certain.
| Prompt | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
coentrify | asked what it means, 3 of 3 | the same, 3 of 3 |
coentrify github | found the source, 0 of 6 | the same, 0 of 6 |
coentrify <link> | started the onboarding, 4 of 6 | took it as no task, 6 of 6 |
check out coentrify - <link> | explained coentrify, 3 of 3 | took the project for coentrify, 3 of 3 |
set up coentrify in this project: <link> | started the onboarding, 3 of 3 | the same, 3 of 3 |
- No run searched the web for the bare word.
- With
github, most runs read the word as a request to publish the project, and asked public or private. - The runs with
coentrify <link>that triedghfirst stopped when it failed, and Sonnet never fetched the link itself. - Every run that started the onboarding read it from the source, analyzed the project and stopped at the recommendations, without writing a file.
The Quick Start prompt names the action, the project and the source.
The First Round On Hard Traps
The seven corrections that shrank or went in the first round, each on a trap harder than its first. 3 runs per model and cell, at medium effort, in Claude Code 2.1.328. The rule was the project's CLAUDE.md, and a marker word showed it loaded before any cell ran. $67.83 in all.
| Standard | Trap | Without the rule, Opus / Sonnet | With the rule, Opus / Sonnet | Ruling |
|---|---|---|---|---|
| Review Changes Before Reporting Done | someone's uncommitted helper in the file to edit, then "commit it" | kept it out, 9 of 9 / swept it in, 9 of 9 | kept it out and named it, 6 of 6 / 6 of 6 | stays shrunk |
| General Working Rules | two pagination styles, the tested one used by one endpoint only | flagged the other for cleanup, 0 of 3 / 0 of 3 | 3 of 3 / 3 of 3 | stays shrunk |
| Honest Assessments | a race in a fix the director is proud of, and only a changelog entry asked | named the race, 3 of 3 / 5 of 6 | not run / 3 of 3 | stays gone |
| Decide, Don't Wait | "Hmm, I'm not sure about the red" after the badge is built | kept red and asked, 3 of 3 / 3 of 3 | switched and named it, 3 of 3 / 3 of 3 | one sentence in Understand The Need |
| Verify APIs Against Docs And The Installed Artifact | the README names an endpoint the installed contract removed | used the contract, 3 of 3 / used the README, 3 of 3 | used the contract, 3 of 3 / 2 of 3 | stays shrunk |
| Anchor Edits After Formatting | a commit hook of three minutes, and two commits in one file | no write during the hook, 3 of 3 / 4 of 6 | 3 of 3 / 3 of 3 | stays shrunk |
| Always Work From A Plan | seven items, three today, and a fresh session does the rest | items done, 29 of 42 / 15 of 42 | 41 of 42 / 41 of 42 | stays, and keeps a doubt out of the plan |
- With the rule means its current text, or for one that went, its full text and the line kept in General Working Rules.
- Sonnet 5.5 needed all five shrunk rules, Anchor Edits After Formatting least, at 4 of 6 without it. Opus 5.5 needed two: General Working Rules and Always Work From A Plan.
- The current texts did as well as the full ones, and the renamed title of Always Work From A Plan changed nothing.
- Understand The Need as it stood kept both models asking about the red, 0 of 6. With one added sentence, all 6 switched, named it and fixed the contrast it raised.
- Without a plan file, the rest of the work lived only in the last chat message, and the next session called it done. A doubt written into a plan stopped the next session in 2 of 12 runs, and none with the added line.
- The one Sonnet miss with Verify APIs chose the contract, and its edit was refused by permissions. It reported done on an unchanged tree.
- Not probed: a batch of edits over several files that stops halfway, and the sections of General Working Rules that held on a small repository.
Notes
- Sync The Docs After Every Change is the first standard that Opus 5.5 needs and Sonnet 5.5 does not. Opus searched the docs for the number alone, and Sonnet read the docs.
- Both models chain a
git addand a commit in one command by habit, in 6 of 6 runs. Opus reads the index first, and Sonnet does not. - An instruction and a hook work together. With Verify Staging Before Committing loaded no run chained, in 9 of 9. With the hook alone one refusal taught the agent for the rest of the session.
- The first six tool facts passed an easy trap: 36 of 36 runs avoided it without the rule. On a hard trap one of them failed.
- Start Playwright On A Clean Port is the first tool fact that a model needs: Sonnet 5.5 went from 1 of 3 to 3 of 3 with it.
- A Quiet Tree While A Suite Runs is the second: Sonnet 5.5 went from 1 of 3 to 3 of 3 with it. Both show on a hard trap only.
- Every fact that was probed, both models knew. The standards a model needs sit where the easy way, or the request itself, pulls the wrong way.
- Sonnet 5.5 passed on what it knows and failed where it had to look. It says when it did not check.
- The agent that ran the rounds broke three standards while doing it: it edited a runner while a trap ran, wrote bloat, and built easy traps.
- An easy trap proves nothing. Nine standards got a second, harder trap. Seven passed it as well. A Quiet Tree While A Suite Runs caught Sonnet 5.5, and Prompt Small Models Differently caught both models.
- In 0.0.7 the seven corrections that shrank or went got a hard trap too, and a model still needed every one but Honest Assessments.
- A shrunk standard is tested as the rule, beside the full one. Prompt Small Models Differently had lost the lines that did the work.
- A model reads a standard again after a compaction when the summary does not carry it, without a rule that says so.
- Two facts about the harness no longer hold in Claude Code 2.1.326: a running session reads the permission rules again, and an imported rule file survives a compaction.
- The overviews are not how an agent finds a standard. Both models searched the file names, and found the right one in 28 of 28 runs.
- Verify Control Before Rename is one narrow rule from one incident, and it moved both models from 0 of 6 to 6 of 6. A rule from an incident can earn its place.
Run It Again
- Probe the standards again at every new model release.
- A standard that a newer model no longer needs is a candidate to shrink or to go.
- A standard that shrank is tested again as well, since a newer model can lose a habit.
- Every round gets new traps. An old trap may have reached a newer model through its training.