Fast, Good, or Cheap: First Principles for Choosing the Models Behind Your Agents

Fast, good, or cheap: pick two. How do you know which two your model actually picked?

In June I thought I knew. I evaluated 8 vision models for a pair of small production classifiers, chose the smallest and cheapest one, and it scored 100 percent. Perfect accuracy. Zero variance across 60 repeated runs. Half a second per inference. Fast, good, and cheap, all three.

A month later I audited that eval. The score was real. It was also wrong three different ways, and the model I had deployed was silently missing a third of what mattered. This paper is what fell out of catching that, fixing it, and re-running everything properly, with confidence intervals this time.

The systems involved are small. They are home cameras answering one question, "should this alert," running on hand-me-down Mac minis in my basement. The principles are not small. If you are standing in front of a model catalog trying to decide what should power an agent, these are the decisions you own too. My lab just has cheaper mistakes.

Six principles

1. A score is a claim about your test, not your system. My 100 percent was a true statement about 30 hand-picked frames. It said almost nothing about my driveway. When the test set grew to 115 independent frames with the hard cases deliberately left in, the same model, same prompt, same settings measured 70.8 percent. Nothing about the model had changed. The claim changed because the test changed. Every number on your eval dashboard works this way, and the first question to ask of any score is not "is it high" but "what test is it a claim about."

2. Write down what good looks like before you measure. Then freeze it. If you grade after you build, the model defines the bar. What changes across tasks is the form the definition takes, not the discipline. For tasks with physical ground truth, gold is labeled examples, and the hard part is honest sampling. My June set had quietly excluded ambiguous frames to keep the labels clean, which is a polite way of saying I defined good as what was easy.

As tasks get more subjective, gold changes form. For policy tasks, it is a written judgment: one frozen file that serves at once as the model's prompt, the labeling standard, and the metrics spec, with the unforgivable error gated at zero rather than averaged away. Concretely, for a support-response task that file might say: cite a source document, never promise a refund, stay under 120 words, match the tone of these 3 exemplars, and inventing an account detail is the unforgivable error. The model reads that file. The labelers score against it. The metrics mean it. For graded work, gold is dimension weights decided before any model runs. For taste, gold is exemplars plus a contract, and the verdict collapses to one binary question, asked of a reader who has no idea what you meant to say: would you share this? Labels, then rubrics, then weights, then exemplars. The form changes. The freezing does not.

3. Every eval has two uncertainties. Report both. Stability is whether the model gives the same answer twice: run-to-run deviation and per-item flip rate. Coverage is how much your test set can support the claim at all: a confidence interval over items. They are different numbers and one never substitutes for the other. My perfect score came with 0.00 percent standard deviation, which was true, and irrelevant, because a perfect score on 30 items is not 100 percent. The exact binomial math says the honest claim was "at least 88 percent with 95 percent confidence." With only 30 items, even a flawless streak is consistent with a true error rate near 1 in 9. There is a 2025 paper titled "Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints," and my zero-width error bar is its textbook pathology. Two practical corollaries. Under a fixed inference budget, more items beats more repeats. And flip rate is the judge metric nobody reports: the model I retired flipped its verdict on one identical image 27 times out of 60. A judge that flips on the same input is not a judge. Meanwhile my deployed model was almost perfectly stable while being wrong on a third of the hard set. Consistent is not correct.

4. Perception and rule-following are different skills, and neither scales with price. My rubric contains a negation rule: the family Tesla parked in the driveway is not notable, ignore it. The 30-billion-parameter models described the scene most richly and still broke that rule, alerting on a car they had correctly identified, scoring 72 to 78 percent in the June screening rounds. The 4-billion-parameter model followed the rule perfectly and could not see small figures at the frame edges, at any resolution I fed it. Research backs both halves: instruction adherence and perception are separable competencies, benchmark leaders on one routinely rank low on the other, and negation-shaped rules are a documented systematic weakness. It goes further than model choice. I re-serialized the identical rubric into three formats and measured: plain prose 91.5 percent (the production configuration, a number you will meet again in the results table), markdown 90.1, XML 79.1. The XML advice that is correct for one vendor's models cost mine 12 points on my task. The transferable finding is the variance, not the 12. Format, size, and price are all empirical variables. Test, don't inherit.

5. Diagnose before you fix: spec, ceiling, or pipe. When an agent fails its eval, there are three kinds of failure and three different fixes. A spec failure means the rule itself is wrong. The tell is that you find yourself arguing with the rubric. The fix is editing the prompt. A capability ceiling means the model understands the rule but cannot execute it. The tell is prompt edits that trade errors instead of removing them. I once rewrote a prompt five times to fix one false alarm, and every version broke something else. Whack-a-mole means ceiling, and the fix is swapping the model. An input failure means the information is not there. No prompt and no model recovers it. But test the pipe before blaming it: I was certain my misses came from a 640-pixel downscale, so I re-ran the whole eval at 1280 pixels, and it recovered 2 frames out of 32. The hypothesis died in one controlled experiment and saved me a useless config change. One discipline binds all three: move one lever per change, because any edit to labels or rules invalidates every number you had. I once changed a truth label 7 minutes after committing eval results and quoted the stale score for a month.

6. Everything drifts, and the refresh ends with a human. Gold sets, rubrics, and the meaning of "good enough" all have expiration dates, so a scorecard carries a timestamp whether you print one or not. But the refresh is a review, not an automatic rewrite. My eval flagged a rubric rule as apparently stale: the system alerts on a silver Highlander that is, by any reasonable guess, a household car. I nearly fixed it. The owner review said the rule was right. That car belongs to my ex-wife, and an alert when it arrives is not a false positive, it is the feature. An eval can tell you a rule looks stale. Only the owner knows whether it still encodes the life it was written for.

The loop

The principles compose into one loop. Define what good looks like and name the unforgivable error. Build the gold set honestly and freeze it. Score every candidate at production configuration, one fixed rubric. Read both uncertainties. Diagnose failures by class. Fix one lever. Re-run. Ship with a timestamp and a monitoring plan, and route what drifts back to the top.

What the loop did to me

Here is the loop run against my own June decision, so the method has a body of evidence instead of a diagram.

The audit found three defects: the statistics above, test frames that shared provenance with the frames I had tuned prompts on, and that un-re-run label change. So I rebuilt the gold set: 115 camera frames drawn from a time period the prompts had never seen, because reusing the frames I had tuned prompts on would be train-test contamination. Hard cases stayed in. Every label the deployed model disputed was re-verified adversarially, by fresh AI reviewers instructed to refute it with zoom-crop evidence and by me adjudicating. 36 of 37 disputed labels held. The whole labeling and QA pass took one afternoon. Then every model ran 10 to 15 times over the frozen set at true production configuration, on both camera tasks: the alert classifier, and a sibling that answers whether the trash bins are out at the curb.

The table inverts. The model my June eval crowned lost by 22 points on independent frames. The model it retired turned out to see everything and flip constantly, with the exact false-alarm habit that got it retired. And the winner was the one June ranked second: statistically tied with the seer on accuracy, tied with the rule-follower on discipline, best on the second task outright. It went to production with a one-line change, a measured cost of about 1.5 seconds per inference, and a rollback file. Frame-level accuracy is 91.5 percent, but a real event fires several frames and one alert is enough. The number a resident experiences is event-level recall: 94.6 percent. The whole campaign was roughly 5,000 inferences, run overnight on hardware I already owned. On a managed platform that is a few dollars of small-model inference, not a project.

One smaller run closed the loop: the same harness settled the prompt-format question in an evening, because the gold set already existed.

The agent is more than the model

My cameras are one-shot classifiers: one input, one verdict, done. Agents act. They take many steps, call tools, and change state, and the 2025-2026 evaluation literature treats that as a categorically different problem. Tools are interdependent, so a call is only valid if earlier calls produced the right state, and an early error compounds through everything after it. A static prompt-response benchmark shows a model can demonstrate a capability in principle. It does not show the agent deploys it reliably over a long horizon. Four findings from the current research matter most, and each one is a principle from above meeting a harsher environment.

Score the trajectory and the outcome, and expect no standard to save you. Outcome metrics ask whether the task succeeded. Trajectory metrics ask how: tool-call correctness, step-level progress, whether the path was safe and efficient. You need both, because outcome-only scoring misses what one paper calls corrupt success, the right answer reached through a path you would never approve. The field has not even standardized what "success" means: one benchmark uses a resolved flag, another a success field, another an implicit reward of 1. And final-outcome pass rates saturate: turn-aware progress metrics have produced outright ranking reversals between models that pass-rate scoring called a tie. My flip-rate principle survives translation, by the way. It just gets expensive: for agents you re-run whole tasks, not frames, and reliability over repeats is the metric.

Trajectory judges cannot detect false success. Ground the verdict in state. The sharpest 2026 result: across 5 judge models and 5 prompting strategies over roughly 12,000 trajectories, no configuration beat an AUROC (area under the ROC curve, where 0.5 is chance and 1.0 is perfect) of 0.65 at telling false successes from honest failures, because judges anchor on the agent's confident closing language instead of checking what actually changed. Reasoning-mode judges did not help. The prescription is blunt: the final verdict must come from a programmatic check of environment state, the database row, the file, the booking, and the judge is for diagnosis, not adjudication. My cameras taught the same lesson at smaller scale. The model says CLEAR with perfect confidence. Truth comes from looking at the frame.

The simulated user is an eval component too. Agent benchmarks put an LLM on the other side of the conversation, and holding the agent constant on tau-bench's retail domain, task success swings up to 9 points depending on which model plays the user. Calibrate the simulator the way you calibrate the judge.

The frozen gold set survives, but it changes shape. You cannot freeze a transcript, because the agent's second turn diverges from whatever reference conversation you curated. What freezes instead: the environment, the task specification, the programmatic verifier, and the user persona template. And the specification layer rots exactly like my rubric did. Even the most-cited agent benchmark needed a verified fork, published by Amazon's own AGI team, because task definitions did not align with the stated policies. That is truth QA at benchmark scale, and it is the same move as my 37-frame adversarial re-check: before you trust a score against gold, audit the gold.

Where the human goes

Everything above leans on judges, verifiers, and automation, so the fair question is where the human reviewer belongs. The evidence supports a sharper answer than "spot-check sometimes," and it starts with two negatives.

The human does not go everywhere, because blanket review does not scale and much of it is theater: a reviewer skimming confident output they have no independent basis to judge is approving, not reviewing. And the human does not go where the judge feels unsure, at least not by the judge's own report, because judge self-confidence systematically overestimates agreement with humans even for the strongest models. A judge's feeling about its verdict is not a signal you can route on.

What works, with measured results, is calibrated escalation. Collect human labels on a slice of real cases and calibrate the judge's confidence against them, so "confident" now means "empirically agrees with humans here." Then let cheap judges keep everything above the calibrated bar and escalate the residue up a ladder: stronger judge, then programmatic verification where it exists (tests, state checks, compilation), then a person. The ICLR 2025 result behind this pattern reached 80 percent-plus human agreement while covering 79 percent of cases, with 88 percent of the work done by models a fraction of the price. The human is the escalation tail, not the default reviewer. One warning before you trust any judge enough to calibrate it: correlation is not agreement. A judge can track human ratings at r = 0.95 while scoring systematically lower than they do, which is bias, not validity. Measure agreement and bias, not correlation.

None of this is new. It is the scoring floor of a state testing company, rediscovered. Before AI, my world was high-stakes student assessment. I was CTO of a company that scored state exams for Tennessee, New York, Utah, and Mississippi, where a human rater's judgment on an essay carried real consequences for real kids. The raters themselves were evaluated with a discipline the LLM-judge world is now reinventing paper by paper. No rater touched a live response before passing a qualification set. Scoring leaders did read-behinds, re-scoring a sample behind every rater on a schedule. High-stakes responses got a blind second read, with disagreement routed to adjudication, not averaged. Pre-scored validity papers were seeded into the live queue, indistinguishable from real work, so drift was caught while it was happening rather than at the next audit. Agreement was measured in exact-plus-adjacent rates and weighted kappa (a stricter agreement statistic), never correlation. A drifting rater was recalibrated or unqualified. Swap "rater" for "judge model" and that is the architecture this section just derived from 2025 papers, except assessment ran it for decades, at millions of responses a year, under contractual scrutiny. The LLM judge is a rater. Treat it like one, and half your eval-operations design is already written.

Two places the ladder cannot replace the person. First, ambiguity is not error. When judges disagree or confidence collapses, sometimes the case is genuinely ambiguous, and humans disagree on it too. Routing it to a labeler misses the point: an ambiguous case is unwritten specification, and the human it needs is the owner, deciding policy. My June eval quietly excluded ambiguous frames to keep labels clean, which threw away exactly the cases that define the rubric. Second, flagged rules go to the owner, because staleness is not decidable from data. The Highlander rule from principle 6 is the proof. The eval could flag it. Only the owner could know the alert was the feature. No judge, at any price, held that fact.

So the rule I would defend: a human must look when calibrated confidence is low and no programmatic check exists, when the case is ambiguous enough to be specification rather than data, when a rule is flagged as stale, and before any action you have named unforgivable. Sampling suffices for the confident bulk, as a scheduled calibration audit rather than continuous review. And prudence, not evidence, says gate irreversible agent actions on human approval. The published guidance on that point did not survive my verification bar, so I state it as judgment. That judgment is informed by the one regulated domain I worked in, where a second human read on a consequential score was contractual, not optional. It is the one place in this paper where I will accept review that looks like theater.

Doing this on Bedrock

I run this loop on Mac minis because my workloads are cameras and my budget is electricity. If your workloads run on Amazon Bedrock, the loop is the same and most of the machinery is managed. What follows is the mapping. The principles stay yours. No feature owns them for you.

Choosing from the catalog is the bake-off. The catalog gives you providers, sizes, and prices per token. A leaderboard orders them in general. Your frozen gold set orders them for your task, and as my table shows, those two orderings can disagree by 22 points. Build the gold set before you shortlist models, not after.

Model Evaluation, including LLM-as-a-judge, is the harness for the model layer. Bedrock's evaluation jobs will run your dataset across models, and the LLM-as-a-judge mode scores open-ended output against built-in quality and responsibility metrics (Correctness, Completeness, Faithfulness, Harmfulness, and others) or custom metrics you define, with judge models spanning several providers. Managed machinery does not repeal principle 3. Run the job more than once and look at what flips. Treat a small dataset's score as an interval, not a number. Calibrate the judge itself with a human spot-check, because a judge model is still a model, with its own flip rate and its own price-performance curve. Note what this machinery is for: responses. The agentic layer above needs the trajectory discipline from the previous section, and most of that verifier layer is still yours to build.

A first eval job, concretely. The round trip is small enough to show. Your gold set is a JSONL file in S3, one line per case, in exactly the shape principle 2 asked you to write anyway:

{"prompt": "Customer: my June invoice shows the same charge twice...",
 "referenceResponse": "Acknowledge, cite the billing policy doc, open a dispute ticket. Never promise a refund.",
 "category": "billing"}

One call starts the job. The skeleton, trimmed to what matters:

aws bedrock create-evaluation-job
  --job-name support-eval-week1
  --application-type ModelEvaluation
  --role-arn arn:aws:iam::...:role/bedrock-eval
  --inference-config '{ generator model under test, or your own captured responses }'
  --evaluation-config '{ judge model + metrics: Builtin.Correctness,
      Builtin.Completeness, Builtin.FollowingInstructions,
      or a custom metric that encodes YOUR rubric }'
  --output-data-config '{"s3Uri": "s3://your-bucket/eval-results/"}'

Results land in that bucket and in the console: a score per case, with the judge's explanation of each one. Two details from the docs earn special mention. You can supply your own captured responses instead of a generator model, which means Bedrock can judge output from systems that do not run on Bedrock, including whatever you are running today. And a prompt router can sit in the generator seat, so the routing policy itself gets evaluated like any other model, which is exactly what the routing paragraph below demands. Then apply this paper: run the same job 3 times and diff the scores, because a case whose verdict moves between runs is your flip rate. Compute the interval over your case count before you quote any average. And spot-check the judge's explanations against a human read on a sample, because the judge is a rater, and you know what to do with raters.

AgentCore Observability is the trajectory layer's raw material. Amazon Bedrock AgentCore ships observability that, per the official docs, offers detailed visualizations of each step in the agent workflow, letting you inspect an agent's execution path, audit intermediate outputs, and debug failures. That is exactly the substrate trajectory evaluation needs: step-level traces you can score, replay, and mine for the judge-disagrees-with-outcome cases that localize failures. What it is not, on its own, is a verdict. Pair the traces with programmatic state checks, because the false-success result above says a judge reading a confident transcript will bless failures. One honesty note: AWS agent-tooling ships fast and feature status churns on a scale of weeks, so verify names and capabilities against current docs before you architect around anything in this section, including this paragraph. Treat vendor claims the way this paper treats model claims.

Prompt management is where format discipline lives. Store prompts as versioned artifacts and treat serialization as an empirical variable. My identical rubric spanned 12 points across three formats on one model family. Whatever format guidance you have inherited was calibrated on someone else's model. One evening against your gold set answers it for yours.

Intelligent Prompt Routing is the triangle negotiated per request. Routing easy traffic to cheap models and hard traffic to strong ones is the right instinct, and it is exactly the fast-good-cheap trade this paper opened with. But a routing policy is itself a model making a gated decision, and principle 4 applies to it directly: evaluate the router the way you evaluate the routed.

Model Distillation is the end of the cost curve, and it inherits your eval. Bedrock distillation transfers a teacher's behavior into a smaller, cheaper student. Everything in this paper becomes load-bearing at that moment, because the teacher's labels are model-as-judge in production: audit the teacher against a stronger model before training, hold out data the student never saw, and keep a scheduled audit after deployment. I ran that audit on my own pipeline before its student trains: a stratified 50-frame replay agreed 50 out of 50, which by this paper's own math means at least 93 percent agreement with 95 percent confidence. That interval is exactly why the audit is scheduled rather than retired. Trust the teacher the way you trust any confident employee who is occasionally wrong: verify on a schedule.

Guardrails are the unforgivable error, productized. Decide which failure your system is never allowed to make, gate it at zero, and measure it separately from average quality. An average hides exactly the errors you cannot afford.

Which two did you pick?

So, fast, good, or cheap. The honest answer to the opening question is that you do not get to pick. You get to measure. I believed I had all three and was quietly running two. After the re-run I paid 1.5 seconds for the third, knowingly, at a price I had measured. Later, a distilled student will make the same task fast, good, and cheap at once, and it will stay trustworthy only because a bigger model audits it on a schedule.

The triangle is negotiable. But only for people who can read their own corner, and the only instrument that shows it to you is an eval you froze before you had an opinion, scored with intervals, diagnosed by failure class, and re-ran when anything moved. A perfect score is a claim about your test, not your system. Mine said 100. The instrument said 70.8. I know which one I trust now, and it is the one with error bars.

Your first eval, this week. Pick the one question your system answers. Write the rubric before you look at any output, and name the unforgivable error. Label 100 real cases, the ugly ones included. Run every candidate over them 5 times at production settings. Report the interval and the flip rate, never the bare point score. For a task with checkable ground truth, that is a week of part-time work (the footer scripts compute the intervals and flip rates). If your ground truth needs expert labels, the labeling is the project, and you should budget for it rather than skip it. Either way, it is the difference between choosing a model and inheriting one.

Epilogue, added the day this published

Hours after this essay went live, the rubric got reweighted. Living with the system, I decided a false alarm costs me a glance at my phone, but a missed event costs me the only thing the system exists for. False positives: acceptable. Misses: not. And latency matters.

Under those weights, the same frozen eval data re-decided the model with zero new inference. The 9B this essay describes retiring has the lowest miss rate of the three (3.8 percent of true-event frames versus the 8B's 8.2), and its known flaw, over-alerting on the family car, is now a priced and accepted cost. It took the camera workload back the same evening. The 8B keeps the second task, where it is still the outright best.

Then the timing test came back: 1.62 seconds per warm inference for the 9B, against 2.09 for the small model and 3.61 for the 8B. The model with more parameters is the fastest of the three, because its vision path is lighter. So the current production pick is simultaneously the fastest, the least likely to miss, and no more expensive to run. Fast, good, and cheap.

The table above stands as written. It was the right decision under July 7's definition of good, and it lasted one day, because the definition moved and the eval was ready when it did. That is not the system failing. That is the loop, closing on schedule.

The eval harness and statistics scripts are open at github.com/nateober/eval-harness. The labeled datasets stay home, for the obvious reason that they are pictures of my house; the repo shows the schema. Written from a June-July 2026 eval campaign: 8 models screened in June, the 3 finalists re-run on the honest 115-frame set at 10 to 15 repeats each, adversarial label QA, one refuted hypothesis, and one production swap. The agentic-evaluation section draws on adversarially verified 2025-2026 sources, including the ICLR 2026 agent-evaluation blogpost, the false-success judge study (arXiv 2606.09863), the TED turn-level framework (arXiv 2603.15483), the Counsel meta-evaluation dataset (arXiv 2606.21627), the simulated-user calibration study (arXiv 2601.17087), tau2-bench-verified, and the Bedrock AgentCore documentation. The human-review section draws on Trust or Escalate (ICLR 2025), Judge's Verdict (arXiv 2510.09738), Bias in the Loop (arXiv 2604.16790), and the human-label-variation literature (arXiv 2507.02593). Every identifier resolves as of July 2026. Check them. That is the point of the paper.