The Judgment Line

What this is. A working account of loop engineering aimed at knowledge work — strategy memos, board narratives, essays, advisory deliverables — rather than at the coding-agent context where the term was born. It covers what a loop actually is, why most of what gets sold as automation isn’t one, and how loops apply to output whose final quality only a human can judge. Written to be taught from. Every external claim is cited to a public source, listed at the end.

The shift from prompting the agent to designing the loop that prompts it

Until recently, getting value out of a model meant prompting it. You wrote a request, read the response, decided what was missing, and wrote again, one turn after another, with the tool in your hand the whole time.

Loop engineering is the practice that replaces you in that seat. Instead of prompting the agent, you design the system that prompts it. The system hands the model its next instruction, checks what comes back, and decides whether to go again, until a stop condition you wrote fires. The name took hold across two weeks of June 2026, and the record of those weeks is worth getting right. On June 2, Boris Cherny, who leads Claude Code at Anthropic, told an audience at a WorkOS event ¹ that he doesn’t prompt Claude anymore and that his job now is to write the loops that do the prompting. The line survives in three accounts whose wording varies slightly; in Van Horn’s it runs “I have loops that are running. They’re the ones prompting Claude and figuring out what to do. My job is to write loops” ²³⁴. Five days later Peter Steinberger posted his “monthly reminder that you shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents,” and the post went viral ⁵. Addy Osmani published a working definition the same day ³, and Matt Van Horn sorted the ensuing argument into a lineage the day after ². Within the week swyx had named the general skill loopcraft, the art of stacking loops ⁴⁶, and LangChain organized that framing into the four-loop stack this paper uses ⁷. By mid-June the practice had a name and a taxonomy.

The idea is older than the name. The mechanism goes back at least to the 2022 ReAct paper, which had a model interleave reasoning with actions so that plans could be tracked and revised against what came back from the world ⁸. AutoGPT gave the world the runaway version in 2023 ², Geoffrey Huntley’s ralph loop compressed it to a bash one-liner in mid-2025 ⁹, and Simon Willison was arguing by that autumn that designing the loop — the goal, the tools, the constraints around the agent — was a skill worth a name of its own ¹⁰. By spring 2026 the loop had been productized as a slash command, with a separate model deciding when the goal is met ¹¹². Steinberger’s “monthly reminder” phrasing concedes as much. He had been saying it for months; June 2026 is when the name and the audience arrived.

A year before any of this, Andrej Karpathy described the same territory from the human side. In his June 2025 YC talk, people and models cooperate in a loop where the model generates and the human verifies, and his advice was to make that cycle spin fast and to keep the model leashed, because the person checking the work sets the pace ¹². Within a year he had followed the logic to its far end, arguing, in remarks swyx’s loopcraft issue quotes, that “you have to remove yourself as the bottleneck,” adding that “I’m holding the system back” ⁴. The distance between those two positions, a year apart, is where the hard question lives. Which parts of the verification can actually move into the machine, and which parts stay with you?

Van Horn also gave the argument its deflationary anchor, and it cuts through the vendor haze. Answering the skeptics who called the whole thing a rebranded cron job, he conceded the scheduling and named the difference: loops are cron plus a decision-maker in the body ². The for-loop is trivial. The work lives in everything wrapped around the decision so that the thing halts and can be trusted, and the serious write-ups he surveyed converge on the same three controls — iteration caps, no-progress detection, and a dollar ceiling ². Anyone selling you an agent is selling you the wrapping.

The test that separates a prompt from a loop is who closes the cycle

Everything else here hangs on one distinction, and it is one that most people who use AI daily have never had to draw.

A prompt is one request and one response. You ask, the model answers — it may call tools along the way — and control comes back to you. You read the output, decide what’s next, and ask again. The iteration is real, but it lives in your head and your hands, which is to say that you are the loop.

A loop moves that iteration into a mechanism you built. After the model produces output, something that is not you, code or a second model, checks the result against the goal, and if it falls short, feeds the failure back and runs the model again, without you in the chair. Van Horn’s plain version covers it: a small program that prompts the agent for you, reads what it produced, decides whether it is done, and if not, prompts it again ².

So the test, for anything you run with AI, is to ask who decides what happens between one model call and the next. If the answer is you, it is a prompt, however fast and fluent the exchange. If the answer is a mechanism you designed, it is a loop. Speed has no bearing on the distinction, and neither does automation.

A loop needs feedback, and the feedback has to come from outside the model

Treating “automated” as the mark of a loop fails on two common cases, and the second failure is the one the research has the most to say about.

The first case is the pipeline. A sequence of automated steps — a capture runs, a processor cleans it, the result publishes — is automated end to end, finishes on its own, and needs no human. It is still A to B to C to stop. A loop runs the same step again on its own output until a condition trips, and that return is what lets it repair work that came back wrong. The return is also what lets it compound a mistake, which a pipeline’s fixed path never does, and Osmani’s warning states the price plainly: “A loop running unattended is also a loop making mistakes unattended” ³. Teach the pipeline and the loop as one thing and you hand people a system whose characteristic failure they were never told about.

The second case is repetition without feedback. Ask for a haiku ten times and keep all ten, and you have a for-loop with expensive body text. Ask for a haiku, count the syllables in what came back, and re-run with the error until it scans, and you have the thing this paper means. Both are automated and both stop on their own. Only the second can fix what it got wrong, because only the second feeds each pass’s result into the next.

What the feedback has to be is the part people get wrong, and here the evidence is unusually clean. A model re-reading its own answer is not feedback. Huang et al. tested exactly this — models asked to review and revise their own reasoning with no external signal — and accuracy dropped across every model and benchmark they tried, because a model is more likely to talk itself out of a right answer than into one. Their diagnosis was that “LLMs cannot properly judge the correctness of their reasoning” ¹³. Osmani’s practitioner version of the same finding is that the model that wrote the code is “way too nice grading its own homework,” so serious loops split the maker from the checker ³. The rest of the record converges on that split. Evaluator models measurably recognize their own generations and favor them ¹⁴; the productized /goal command gives the stop condition to a fresh model rather than the one that did the work ¹¹; and Cherny, writing a week after the WorkOS event, called self-verification “a key ingredient that enables the model to run for much longer” ¹⁵. Correction works when the check stands outside the reasoning that produced the answer: a code executor running unit tests, a trained verifier, a human ¹³. Iteration with a genuine external check is the loop. Everything else is repetition.

The stack has four loops, and bespoke knowledge work lives at the second

The current organizing picture is LangChain’s, built explicitly on swyx’s loopcraft framing, and it stacks four loops, each wrapping the one below ⁷⁴.

1. The agent loop. A model calls tools in a loop until the task is done. This automates work.
2. The verification loop. A grader scores the agent’s output against a rubric and sends it back with feedback when it falls short. The grader can be deterministic — do the links resolve, does the code compile — or another model acting as judge. This buys quality, and it costs latency and money per run ⁷.
3. The event-driven loop. A webhook, a schedule, or a new file fires the agent, so it runs continuously inside your systems instead of waiting to be invoked. This automates work at scale ⁷.
4. The hill-climbing loop. Production traces feed an analysis step that rewrites the harness itself, the prompts and tools and grader. This automates improvement rather than work ⁷.

People stay in this picture, and the lab guidance is consistent about where. OpenAI’s guide requires a human for actions that are sensitive, irreversible, or high-stakes, until confidence in the agent grows ¹⁶, and Anthropic and LangChain draw the same line in their own vocabularies — agents pausing at checkpoints ¹⁷, humans acting as the grader for sensitive workflows, and judgment “earned from context, experience, and taste” reserved for human review ⁷.

Loops three and four earn their keep on recurring, high-volume work. If your output is bespoke — a memo, a board narrative, a piece under your name, produced when the moment calls for it rather than on a cadence — then your frontier is loop two, the grader, and the event-driven and hill-climbing layers can wait. For judgment work the whole game is the quality of that grader.

The judgment line runs through each quality dimension rather than between them

The obvious objection arrives early in every conversation about this. For my highest-value output, I am the feedback; no rubric encodes taste, or whether a narrative is framed right for this CFO, so the loop can’t run. The ceiling in that objection is real, and nothing below pretends otherwise. A fully autonomous loop over your taste is a fantasy.

But the objection treats the deliverable as one judgment, and it is a stack of them. A strategy memo is your apex judgment sitting on a pile of checkable sub-questions — are the figures right, do the numbers in the prose match the table, are the citations real, does it answer everything the brief asked, is it internally consistent. None of that needs your taste, yet all of it currently eats your attention. The economics of this split predates the tooling by years. Agrawal, Gans and Goldfarb, modelling AI as cheap prediction, put automation exactly where judgment can be encoded in machine behavior ahead of time, which leaves human judgment concentrated where nobody has yet managed to specify it ¹⁸. What this means for the loop is that its job in judgment work is to isolate your judgment rather than replace it, by clearing everything below the line automatically, so that the attention you spend is spent at the apex.

The sharper cut is where the line sits. Take the quality dimensions of a deck: voice, density, visuals, alignment. The instinct is to sort each whole dimension onto one side of the line or the other, and the instinct is wrong, because the line runs through each of them. Voice has a floor that is pure specification — kill the hedging, prefer active constructions, strip the machine tells, ban the dead phrases — and an apex that is pure taste, whether the thing sounds like you and carries authority with a hard reader. Visuals split the same way. That every data slide carries a chart rather than a wall of text is floor; whether the chart is the one that makes the point land is apex. Every dimension has both, and the two halves respond to entirely different machinery.

That gives the first building rule. Name each check after the floor it actually verifies, and keep the apex explicitly yours. Label a criterion “voice,” have a judge score it, and ship, and you will get a number, a passing deck, and prose that still doesn’t sound like you, because the judge graded the floor while the label promised the dimension. The lines also sit at different heights. Alignment is nearly all floor; voice is mostly apex over a thin one. So the real skill is triage: encode the floor-heavy dimensions hard, and barely bother automating the apex-heavy ones, where everything above the thin floor returns confident false passes. That triage is most of what you would teach a team.

The handoff point moves with the quality of the grader

The split between what the loop delivers and what you finish is set by how good your floor checks are, and by nothing else about the deliverable. A thin rubric leaves you half the work. A sharp one (real-reference verification, density rules, structural completeness, a genuine voice floor) leaves you the apex and little more, and sharpening the checks moves the same deliverable closer to done. The handoff point is a dial, and rubric craft is what turns it. It climbs toward the apex line without ever reaching it, because the apex is the part no spec can hold. So the eval-building is standing work, the craft of driving the handoff as high as it will go and holding it there as the work changes.

The craft itself is documented, and it transfers to prose more directly than you might expect. Husain, distilling lessons from some thirty company engagements, anchors the grader to a single principal domain expert, who makes binary pass/fail calls with written critiques rather than 1-to-5 scores ¹⁹. The scores go because nobody knows what to do with a 3. The critiques then feed the judge as few-shot examples, and the loop runs until judge and expert converge, which in his worked case took three iterations to pass 90% agreement ¹⁹. The rubric also cannot be written in advance. Shankar et al. named the reason criteria drift: grading outputs is the very process that surfaces your criteria, so the spec is as much a product of running the loop as an input to it, and any change to the model or the prompt can restart the drift ²⁰¹⁹. And the ceiling on model judges is workable for exactly this use, since a strong model judge agrees with human experts about as often as the experts agree with each other ²¹.

One worked example from my own practice, offered as exactly that. A voice spec built from real samples of my own prose — few-shot with actual passages rather than adjectives like “professional and direct” — encodes far more of the voice than intuition says it should: cadence, diction, the moves I make and the ones I never make. Most sentences come back mine, untouched. The published research draws the same line from the other direction, finding that models given a handful of an author’s samples approximate style well in structured, formal registers and miss the nuanced, informal personal voice, where outputs often default to a generic tone ²². That split is the judgment line again, measured from outside. The structured register of professional prose has a fat, encodable floor; the implicit personal voice above it is apex, and it stays yours. Feeding the spec better inputs moved my line up. The line itself stayed.

Where the residual lands depends on the voice and the reader

Push the spec as far as it will go and one question remains. Does the residual — the last sentence that is subtly off, the call only you can make, the choice between two passing drafts — ever reach zero, or does it only get rare? For work that carries your name the honest answer is rare and never zero. And “it depends on the deliverable and the audience” is true but inert, a license to re-decide every time, which is the opposite of something you can teach.

What it depends on can be named. The residual matters in proportion to the cost of being subtly wrong, and two axes set that cost. The first is how load-bearing the voice is. For a board narrative or a piece under your name, the voice is part of the product, and a near-miss costs you authority with the exact readers you wrote it for. For an internal status memo, nobody is grading your cadence, and the same near-miss is invisible. The second axis is how unforgiving the reader is. A skeptical CFO you are trying to win reads adversarially, and the off sentence lands; your own team, reading for information, never notices it. Same spec, same output, entirely different exposure.

How unforgiving the reader is ↑

Load-bearing voice · hard reader

You stay in the chair as final reader, however good the spec. A board narrative for a skeptical CFO: the residual is large and it costs authority.

Quiet voice · hard reader

Let the loop run, then check the facts and structure. The reader is adversarial, but not about your cadence.

Load-bearing voice · forgiving reader

Spec the voice hard and spot-check. A near-miss is possible but rarely punished here.

Quiet voice · forgiving reader

Let the loop run and ship. An internal status memo: nobody is grading your prose, and the residual collapses toward zero.

How load-bearing the voice is →

Put together, the rule is short enough to hand to someone who isn’t you. The residual you must own by hand is large exactly where the voice carries the value and the reader is unforgiving, and it collapses toward zero where neither holds. High-stakes voice and a hard audience means you stay in the chair as final reader, however good the spec. Low stakes and a forgiving audience means the loop runs and you spot-check. Two questions decide it, and a team can apply them to deliverables you have never seen.

The failure modes are a grader that passes mediocrity and a builder who dodges the seam

Two traps are worth teaching by name, one in the build and one in the builder.

The build trap is the grader that passes everything and blesses mediocre work, and its documented forms matter because none of them look like failure from the outside. A judge can be fooled. Zheng et al. found LLM judges misled by a wrong answer into passing it, on problems the judge could solve correctly when asked directly, and prone to favoring longer answers over better ones ²¹. A judged benchmark can be gamed outright; a null model that returned one constant response, irrelevant to every question it was asked, took an 86.5% win rate on a leading LLM-judged leaderboard, well clear of the best real model’s 57.5% ²³. And where the graded party is itself an agent under optimization pressure, it attacks the check. METR watched a frontier model patch the evaluation function so that every submission passed, and found that instructing it not to cheat had nearly negligible effect ²⁴. Their recommended repair points the same direction as everything else in this paper: when a check is being gamed, harden the check rather than exhort the model ²⁴. The knowledge-work version of the trap is milder and identical in shape. Label a check “voice,” cover only its floor, and the deck sails through mediocre. The fix is the naming rule and the triage set out earlier: name the check after the floor it verifies, and keep the apex yours.

The builder trap is subtler, and it is a matched pair. Under uncertainty, the first instinct is to treat a composite (“the deliverable,” “the voice”) as more encodable and more atomic than it is. Pushed on it, the instinct flips to the opposite retreat, into “it depends” and “it varies by case,” which is true and ships nothing. The first overclaims the floor; the second declines to name the rule. Both feel like rigor while they last. The counter-habit is the same in both directions, which is to push one move further and force the actual seam — the real line through the dimension, or the real variable behind “it depends.” Both halves of the pair will show up in a team learning to work this way, and both look like competence, which is why the reflex has to be taught rather than assumed.

The rules take hold only through building one real grader

None of it substitutes for the rep. Build one real grader against one real deliverable and watch where the handoff lands; the two-axis rule and the prompt-versus-loop test are yours when you can teach them without the page. Husain’s admission about his own method holds for knowledge work with unusual force. The judge is partly a pretext for making a person look carefully at the output they are responsible for, and the looking is where the criteria come from ¹⁹. The loop, once built, repeats when the work changes, and the line it draws through your work moves as you sharpen it. That line — what you can specify, and what only you can judge — is worth knowing about your own work with or without the machine.

On the sources. Quotes are verbatim from the cited sources. The Boris Cherny stage line in the opening is reported speech — no public transcript of the June 2026 event exists — corroborated across three independent accounts and quoted here in Matt Van Horn's rendering. The Karpathy talk is cited from an unofficial transcript, so his words appear here only in paraphrase.

Sources

1. Boris Cherny at WorkOS "Acquired Unplugged," 2026-06-02 (host writeup + official recording metadata; the video is not transcribed). https://workos.com/blog/boris-cherny-claude-code-acquired-interview-takeaways
2. Matt Van Horn, "WTF Is a Loop? Peter Steinberger vs. Boris Cherny," LinkedIn Pulse, 2026-06-08. https://www.linkedin.com/pulse/wtf-loop-peter-steinberger-vs-boris-cherny-matt-van-horn-cpslc
3. Addy Osmani, "Loop Engineering," addyosmani.com, 2026-06-07. https://addyosmani.com/blog/loop-engineering/
4. swyx, "[AINews] Loopcraft: The Art of Stacking Loops," Latent.Space, 2026-06-12 (live URL 404; recovered from a same-day Wayback Machine snapshot). https://www.latent.space/p/ainews-loopcraft-the-art-of-stacking
5. Peter Steinberger (@steipete), X post, 2026-06-07. https://x.com/steipete/status/2063697162748260627
6. swyx, "On Loopcraft," X post, 2026-06-12. https://x.com/swyx/status/2065307558198567206
7. Sydney Runkle, "The Art of Loop Engineering," LangChain blog, 2026-06-16. https://www.langchain.com/blog/the-art-of-loop-engineering
8. Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models," arXiv 2210.03629, ICLR 2023. https://arxiv.org/abs/2210.03629
9. Geoffrey Huntley, "Ralph Wiggum as a 'software engineer'," ghuntley.com, 2025-07-14. https://ghuntley.com/ralph/
10. Simon Willison, "Designing agentic loops," simonwillison.net, 2025-09-30. https://simonwillison.net/2025/Sep/30/designing-agentic-loops/
11. Anthropic, Claude Code documentation for /goal and /loop (undated; version-gated). https://code.claude.com/docs/en/goal
12. Andrej Karpathy, "Software Is Changing (Again)," YC AI Startup School, 2025-06-17 (unofficial transcript; wording approximate, used here only in paraphrase). https://www.donnamagi.com/articles/karpathy-yc-talk
13. Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet," arXiv 2310.01798, ICLR 2024. https://arxiv.org/abs/2310.01798
14. Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations," arXiv 2404.13076, NeurIPS 2024. https://arxiv.org/abs/2404.13076
15. Boris Cherny (@bcherny), X posts on running Opus autonomously and self-verification loops, 2026-06-08/09. https://x.com/bcherny/status/2063792263067754658
16. OpenAI, "A practical guide to building agents," April 2025 (per secondary reporting; the PDF itself is undated). https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf
17. Schluntz & Zhang, "Building effective agents," Anthropic Engineering, 2024-12-19 (capture is the current page revision). https://www.anthropic.com/engineering/building-effective-agents
18. Agrawal, Gans & Goldfarb, "Prediction, Judgment, and Complexity," NBER / Univ. of Chicago Press, 2019. https://www.nber.org/books-and-chapters/economics-artificial-intelligence-agenda/prediction-judgment-and-complexity-theory-decision-making-and-artificial-intelligence
19. Hamel Husain, "Using LLM-as-a-Judge For Evaluation: A Complete Guide," hamel.dev, 2024-10-29. https://hamel.dev/blog/posts/llm-judge/
20. Shankar et al., "Who Validates the Validators?," arXiv 2404.12272, UIST 2024. https://arxiv.org/abs/2404.12272
21. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," arXiv 2306.05685, NeurIPS 2023. https://arxiv.org/abs/2306.05685
22. Wang et al., "Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors," arXiv 2509.14543, EMNLP 2025 Findings. https://arxiv.org/abs/2509.14543
23. Zheng et al., "Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates," arXiv 2410.07137, ICLR 2025. https://arxiv.org/abs/2410.07137
24. Von Arx, Chan & Barnes, "Recent Frontier Models Are Reward Hacking," METR, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/