Blogpost & Paper Reviews

“A support associate would glance at the account, check the spelling, and move on.”

How would the support associate know the account information? Are they cross-referencing based on the phone number?

From the post: The post doesn't say how the associate finds the account.

Inference: The account is usually pulled up before verification starts, most often by caller ID, which matches the calling number to the number on file, or by an identifier the caller gave earlier, such as an account number. The phone number is only a lookup key, not proof of identity, because caller ID can be spoofed. That's why the associate still checks the name.

“To evaluate performance under these conditions, we built an internal benchmark using domain-specific customer service audio.”

What are the main performance metrics for this kind of data? How are they defined and measured?

From the post: Only one metric is named: utterance error rate (UER), how often an utterance has a meaning-changing error.

From web search (μ-Bench blog and README): Sierra's open benchmark also reports word error rate (WER), which is substituted, deleted and inserted words divided by the words in a human reference transcript, and 95th-percentile latency to a complete transcript. The blog's case for UER is that WER counts a dropped "uh" the same as a misheard phone-number digit. An LLM judge decides which errors change the meaning.

Inference: UER is the right primary metric for agents. Its weak point is the judge: I couldn't find how often it agrees with human reviewers.

“And unlike other conversations — where agents can infer intent from a “good enough” transcription — verification is binary: you need an exact match.“

What verification are they talking about here, and why is it binary in this case but not in others?

From the post: Identity verification, such as confirming a caller's name with a bank or insurer.

Inference: It's binary because the transcript is compared with a stored value, and the comparison either matches or doesn't: "Kaitlyn" fails against "Caitlyn." In other turns, the transcript goes to an LLM that can understand the caller's intent despite small errors. Systems can loosen the match, for example by ignoring case or matching on sound, but each loosening makes it easier for the wrong person to pass.

“If context shifts mid-conversation, the challenge gets harder.”

Shouldn't we default to using a multilingual transcription provider then? Any major downsides to that?

From web search (μ-Bench blog): No provider was best in every language, and the most accurate one was among the slowest.

Inference: A multilingual model is a good fallback but a weak default, for two reasons. It usually trails a model tuned for one language on that language. And it has to guess which language it's hearing, which is hard on short inputs like a spelled name.

“Standard transcription pipelines process audio in isolation, without awareness of what the conversation is about or what the caller is likely to say next.”

Similar to the way we give LLMs context at the start of their prompt to inform subsequent generation, are there any architectures/models for voice transcription that can do the same thing?

From web search (Deepgram): Deepgram's Nova-3 accepts up to 100 key terms per request to improve their recognition, without retraining.

Background knowledge: Yes, three kinds. Conventional speech recognition can favor a supplied word list, like Deepgram's. Models with a text decoder, such as Whisper, accept a text prompt treated as the preceding transcript, which is closest to your LLM analogy. Speech LLMs, where an audio encoder feeds a language model, take context exactly like an LLM prompt. The shared risk is that the model outputs a favored word the caller didn't say.

“Instead of picking one provider and managing its weaknesses, we built an ensembler that queries multiple providers in parallel and combines their outputs.“

A major consideration in voice AI is latency. If we're introducing an ensemble layer, what happens to our latency?

Inference: Because providers run in parallel, the added latency is roughly the slowest provider you wait for plus the ensembler's own processing, not the sum of all providers. Two design choices keep it small. A deadline lets the ensembler use whatever has arrived and drop slow providers. A fast path accepts the transcript immediately when providers agree, and runs the slower resolution step only where they differ.

“Cross-reference outputs and identify where providers agree or diverge“

What does this mean concretely?

Background knowledge: The classic method, ROVER (1997), aligns transcripts word by word and votes at each position.

Inference: Sierra likely aligns the transcripts to find the specific words where providers differ, such as a name returned as "Caitlyn," "Kaitlyn" and "Katelyn." It then resolves only those words, using information a vote ignores: each provider's accuracy in that language, confidence scores, and the name on file. Giving an LLM the candidates and the context is a plausible way to do that last step.

“Incorporate signals from earlier turns in the conversation.”

What does this mean concretely?

From the post: The only earlier-turn signal the post mentions is the conversation's language. Its name and address examples come from the customer record, not earlier turns.

Inference: The two most useful signals are probably the agent's last question, which predicts the type of answer (a date, a spelled name, digits), and entities already mentioned, such as an order number or product name that's likely to come up again.

“On our internal benchmarks, we have found that ensembling can cut utterance error rate (how often an utterance has a meaning-changing error) by ~25% on average versus the best single provider, and by up to 37% in languages with more headroom for improving transcription.“

How to weigh this performance improvement versus the presumed latency increase?

Inference (the numbers are my illustration): Compare expected time, because errors cost time too. If ensembling cuts errors from 8% to 6% of utterances, and each error costs a 6-second re-ask, it saves about 120 milliseconds per utterance on average. So it pays off if it adds less than that. Failed verifications and transfers to a human cost far more than a re-ask, so the case is strongest on high-stakes turns like verification and weakest on small talk.

“Rather than asking the transcription model to guess from the full space of possible utterances, we feed it context from the conversation.”

Is the context fed to the transcription model or the agent in a post-processing layer?

From the post: To the transcription side, not the agent. The ensembling diagram also shows context going into the ensembler.

Inference: The context is probably used twice: as bias terms for each provider, and by the ensembler when choosing among their outputs. One risk is that biasing toward the value on file makes a near-miss more likely to be transcribed as an exact match, which weakens identity checks.

“For our financial services agents, context-aware transcription improved input verification rates by over 25%”

What is input verification?

Inference: It most likely means checking that a value the caller says, such as a name, date of birth or account digits, matches the record, with the rate being the share of attempts that pass. Two things are unclear: whether "25%" is a relative increase or percentage points, and whether some of the extra passes are callers who shouldn't have passed. The false-accept rate is the missing number.

“Sierra voice agents have improved resolution rates by up to 1%, which translates to tens of thousands of resolutions a week, and reduced major transcription errors by up to 15%.”

Why is the improvement in transcription errors so much higher than the improvement in resolution rates?

Inference: The 15% is a relative cut in errors, while resolution is measured over all calls, and most calls have no major error. Of the errors that do occur, many don't change the outcome because the agent asks again or understands anyway. And most unresolved calls fail for other reasons, such as policies that require a human. As a rough check, if transcription errors caused failures on F% of calls, a 15% cut would recover about 0.15 × F points, so a full point would need F near 7%. That suggests 1% is a best case.

“the system dynamically reconfigures the transcription pipeline — selecting a different ensemble of providers optimized for that language”

How is the ensemble selected?

From web search (μ-Bench blog): Sierra benchmarks 79 locale variants across 42 languages and more than 13 providers internally, and says this is what lets it pick models per language.

Inference: Most likely, the best ensemble for each locale is chosen offline on accuracy, latency and cost, and stored in a lookup table. When the system detects the caller's language, it switches to that entry.

“No amount of ensembling or context injection will produce a reliable transcription from unintelligible audio. In those cases, the best response is to ask for clarification, just as a human would.”

Is this done using log probs or something else?

Inference: Probably several signals rather than log probs alone. Speech recognition models do return per-word confidence scores, their equivalent of log probs, but those are often poorly calibrated. They're likely combined with disagreement between providers, which the ensemble provides for free, and with checks such as a date that's impossible or a value that doesn't match the record.

How would you improve this system if you were building it today?

Inference: I'd make one change at each stage of the pipeline:

• Routing. Ensemble only on high-stakes turns such as verification and data capture, and use the fastest accurate single provider elsewhere (#9).
• Decoding. For structured inputs, restrict output to the expected format, such as digits, dates or letters, and accept the phonetic alphabet ("B as in boy").
• Verification. Keep "what was said" separate from "does it match," log how much each match relied on biasing, and track the false-accept rate (#10, #11).
• Learning. Store provider disagreements with human-reviewed corrections, use them to train the ensembler, and alert when the disagreement rate jumps, since that often means a provider has degraded.
• Evaluation. Report end-to-end task success, for example from τ-voice simulated calls, alongside error rates, because #12 shows transcription gains don't translate directly into resolutions.

“Picking a pharmacy. How hard could that be?”

Why is Forus involved in picking the pharmacy at all? Isn’t that something Maria would do herself?

From the post: Forus handles the step after the prior authorization (PA) is approved. Maria's preference is one input, and the agent can “request patient preferences.”

Inference: For a specialty drug the choice isn't free. Her plan often requires a specific specialty pharmacy, and the manufacturer may limit which pharmacies can stock the drug. Most patients don't know these limits. Also, the prescriber's side sends the prescription, and Forus does that work for practices. Maria's preference only decides among the valid pharmacies.

“[W]hich eligible pharmacies are in-network can change without warning. Often, we find out when a routing fails.”

No API connection for checking in-network?

From web search: Drug Channels says there is “no master database” of BIN/PCN/Group codes.

Inference: Contracts between specialty pharmacies and PBMs (the companies that process drug claims for insurers) are private, change on their own schedule, and have no public data feed. The most reliable signal is a rejected claim, which is why a failure is how Forus finds out.

“Insurance routing depends on BIN, PCN, and Group. We may only have two of those three codes.”

What’s the difference between these three and why might you only have two of them?

From web search (Drug Channels): The BIN identifies the insurer or PBM processing the claim. The PCN identifies the benefit package within that payer. The Group identifies the specific plan, such as one employer's plan.

Inference: There are two ways to end up with only two codes:

1. The code doesn't exist. Some processors don't use a PCN, and some cards have no group number.
2. The code exists but wasn't captured, for example because the EHR has the wrong card, the photo is blurry, or a form field was skipped.

“Two pharmacies may both be valid choices, so accuracy against a single label can mislead.”

Spectrum of validity or binary?

From the post: Validity is yes-or-no for each pharmacy. The issue is that several pharmacies can each be valid, so the right answer is a set.

Inference: It's binary for correctness: a pharmacy is either in network and able to dispense the drug, or it isn't. It's a spectrum for quality: speed, distance, copay and the patient's preference. Re-route rate only measures the binary part, so a valid but slow pharmacy still counts as a success.

“Unresolved cases went to manual review.”

What does manual review entail?

From the post: Forus's clinical experts do it, using the same tools the agent later got: searching pharmacies, reading documents, checking insurance, contacting the patient and scheduling calls.

Inference: A reviewer likely reads the case documents and works out which pharmacies the plan and manufacturer allow. When unsure, they call the PBM or the pharmacy to confirm network status. They check the patient's preference if it matters, then send the prescription. The phone calls are what make it slow.

“14-case if/else cascade for retail drugs and ~30 hardcoded insurance rules for specialty drugs”

What's the difference between retail drugs and specialty drugs? Why would the rules for them differ?

Background knowledge: Retail drugs are cheap, common and stocked almost everywhere. Specialty drugs are expensive, often biologics that need special handling, and payers or manufacturers often restrict them to specific pharmacies.

From the post: The retail rules compare the pharmacy listed in the EHR with past prescriptions. The specialty rules are insurance rules.

Inference: The two rule sets answer different questions. For retail drugs, almost any in-network pharmacy works, so the question is where the patient likes to go. For specialty drugs, the question is which pharmacies the plan allows, and that depends on the insurance codes.

“Our data was tabular with lots of missing values, which is exactly where gradient-boosted trees shine.”

How do they handle text data? Embedding conversions?

From the post: V2 used no free text; all nine features are short structured fields. Free text (letters, notes, patient texts) only came in with V4. LLM extractors pull a pharmacy name out of the text and fuzzy-match it to Forus's pharmacy database.

Inference: The text fields were most likely treated as categories, which XGBoost handles directly, rather than as embeddings. Extraction fits free text better than embeddings because the useful content is one specific pharmacy, and an embedding would blur it.

“We set per-pharmacy confidence thresholds to target >97% accuracy on automated selections.”

1. How is the threshold determined and why is it different for each pharmacy?
2. Reasoning for 97% target?

Inference (1): For each pharmacy, they likely picked the lowest score at which accuracy on held-out data stays above 97%. Thresholds differ because the scores aren't equally reliable. A score of 0.8 for a frequently used pharmacy may be right far more often than 0.8 for a rarely used one.

Inference (2): It's a business tradeoff between the cost of a wrong routing (a patient waiting days) and the cost of manual review, probably benchmarked against how accurate humans are. At 97%, about 3 in 100 automated routings are wrong.

“Instead of a frozen feature snapshot, the model uses a live history of routing outcomes, weighted by recency.”

So the routing outcomes are features?

From the post: Not directly. Matching past routings are summarized into features: the top pharmacy, how much of the evidence agrees on it, how fresh it is, how many routings there are, and how much the rules agree.

Inference: The key change is where the knowledge lives. In V2, the link from insurance codes to pharmacy was built into the model, so it only changed when the model was retrained. In V3, that link lives in the routing history, which updates with every routing. The model only learns how much to trust a given body of evidence, and that changes slowly.

“We search every combination of at least two available features for matching successful routings.”

What does this mean exactly? How are the matching successful routings used?

From the post: They take the fields that are filled in and form every combination of two or more of them. Each combination is a “rule,” for example BIN + PCN. A rule's evidence is the past successful routings that match on those fields. Those routings are weighted by age, summarized into features, and scored by XGBoost.

My arithmetic: Four filled-in fields give 11 rules, which matches the diagram's “10 of 11 matching rules.”

Inference: It's a back-off scheme: when a specific combination has little history, broader combinations still supply evidence.

“The powerset approach substantially outperformed approaches that treated missing features as unknowns.”

By what metric?

From the post: The post doesn't say and gives no numbers.

Inference: It's probably re-route rate, or the share of cases that can be automated at the 97% target. Here is why the powerset approach would win. If “missing” is treated as its own value, a prescription with no Group only matches past cases that also had no Group, which splits the evidence for no real reason. The powerset approach ignores missing fields and uses every routing that shares the fields that are present.

“We keep every matching rule.”

What do they mean by matching rules, and what does it mean to keep them?

From the post: A matching rule is a combination of fields that has past successful routings behind it. Keeping every rule means they don't keep only the most specific one; every rule gets scored, because “when broad and specific rules disagree, that disagreement is useful.”

Inference: Throwing away the broad rules would lose two things. First, a specific rule with 3 routings may be less reliable than a broad rule with 900. Second, disagreement between broad and specific rules signals that something is changing.

“The model uses evidence across the matching rules, including how much they agree.”

How do you measure how much they agree?

From the post: The diagram shows agreement as a count, “10 of 11 matching rules favor RxGate,” meaning the share of rules that pick the same top pharmacy.

Inference: Two variants would probably work better. One weights each rule's vote by how many routings it has. The other compares the most specific rule directly with the broadest one, since the post singles out that split as a sign of change.

“we give recent evidence more weight using a time decay function.”

How are the parameters for the time decay function determined?

From the post: The post doesn't say. Its chart shows a weight near 1.0 for the first week, 0.5 around day 25, and near zero by day 90. It frames the choice as a tradeoff between throwing away useful history and keeping stale routings.

Inference: The parameters were most likely tuned by backtesting: replay past prescriptions in time order and pick the curve that gives the lowest re-route rate. The curve could also be tuned separately for payers whose networks change often.

“For each pharmacy, sum the weights of its historical routing cases. The pharmacy with the highest total becomes that rule’s top pharmacy.“

Why not retain the scores for each pharmacy and use those as features?

From the post: They partly do, in summary form: percent match is the winner's share of the weighted total.

Inference: A separate feature for every pharmacy would tie the model to specific pharmacies. A network change would then require retraining again, which V3 was built to avoid. Your concern still holds: keeping only the winner throws away the runner-up, which matters when the winner has just left the network. One fix is to score each candidate pharmacy within a rule. A cheaper one is to add the gap between first and second place as a feature.

“Time-weighted percent match: How much of the weighted evidence agrees on the winner? A rule at 95% is a much stronger signal than one at 55%.”

What does it mean for a rule to be at 95% versus 55%? What are those percentages measuring?

From the post: The post only defines it: “How much of the weighted evidence agrees on the winner.”

Inference: For one rule, divide the weighted evidence for the top pharmacy by the rule's total weighted evidence. At 95%, nearly all recent routings for that rule went to one pharmacy. At 55%, the top pharmacy barely leads, which suggests two pharmacies are both valid or a switch is underway. Percent match ignores volume, so 95% from 2 routings is weak evidence. That's why routing count is a separate feature.

“One XGBoost model scores each matching rule using its weighted results, evidence freshness, routing count, and the number of fields in the rule, along with agreement across rules. The score estimates how likely that rule’s top pharmacy is to be a valid choice.”

What does it mean for an XGBoost model to score a rule? I don’t really get the statement about how it “estimates how likely that rule’s top pharmacy is to be a valid choice”.

From the post: The highest-scoring rule's pharmacy becomes the prediction, and its score becomes the confidence.

Inference: Think of each rule as a witness that names one pharmacy. XGBoost rates how reliable each witness is, based on its features: percent match, freshness, volume, specificity and agreement. Its output is the probability that sending the prescription to that witness's pharmacy will work. It was probably trained on past prescriptions, with each rule labeled 1 if its pharmacy turned out to be valid and 0 if not.

I’m having trouble reconciling the diagram with the description. Clarify how the XGBoost pipeline works and how the rules fit into it.

From the post: The diagram differs from the text in two ways. It lists routing count as one of each rule's four features and shows agreement separately, because agreement is computed across rules. It also draws a single XGBoost box, but the model scores each rule separately.

Inference: The pipeline runs in this order:

1. Form every combination of two or more available fields; each combination is a rule.
2. Pull each rule's past routings and weight them by age.
3. Compute each rule's features, plus agreement across rules.
4. Score each rule with XGBoost.
5. Use the top-scoring rule's pharmacy as the prediction.

“Each extracted name is fuzzy matched against our pharmacy database to select a specific pharmacy.”

And then what? Is that the final decision or another feature somehow?

From the post: By V4, the extracted pharmacies fed into an 11-step cascade with a fixed priority order: “Doctor note beats insurance rule. Insurance rule beats routing history.”

Inference: The extracted pharmacy was most likely a decision at its own step, not a model feature. If a PA letter names a pharmacy, that step fires and the cascade stops there. This is the weakness the post describes, because a year-old note can override fresher evidence.

“At this point, our cascade had 11 steps. Signals often conflicted. The waterfall resolved them using the priority order we set. Doctor note beats insurance rule. Insurance rule beats routing history.”

I thought they were using the XGBoost model at this point, not the cascade?

From the post: They used both. XGBoost never replaced the cascade: “To improve the cascade, we added an XGBoost model.”

Inference: Through V4, the model was one step in the cascade, and “routing history” in the priority order most likely means the model's prediction. Each piece worked on its own, but the code that combined them was still a fixed priority list. The agent replaced that combining step.

“Those tools let it search pharmacies, read documents (including insurance cards via vision), check dispensing restrictions, look up insurance details, request patient preferences, schedule phone calls, store a pharmacy selection, or escalate to an expert. It reads everything, decides which pieces of information to rely on, investigates further if it needs to, and then acts.“

Do we have any latency constraints? This feels like a time-consuming process.

From the post: The post mentions no latency limit. Its time scale is days: a wrong pick delays treatment by days.

Inference: The time budget is hours, not milliseconds, because this runs in the background after the PA is approved. A few minutes of agent work is tiny compared with a re-route. The real constraints are cost per case and human wait time, since asking the patient or calling a pharmacy adds real delay. A sensible design is to act right away when the signals agree and investigate only conflicts.

“Ask Maria for updated information. Escalate if selection remains unclear.“

What does escalation mean here?

From the post: Escalation hands the case to a Forus clinical expert, the same people who did manual review earlier.

Inference: “Unclear” probably means Maria doesn't reply, or her answer still doesn't lead to a valid pharmacy. The expert then calls pharmacies or the PBM and decides. Ideally the expert gets the evidence the agent already gathered, so they don't start from scratch.

“The first version of the agent performed worse than the waterfall in many cases. But when it got things right, its reasoning reflected the kind of cross-signal thinking that our clinical experts exhibit.”

How much does the latter matter given the former? Is there any good way to combine the strengths of the agent with the waterfall?

From the post: They didn't ship based on reasoning quality. The agent ran in shadow mode for weeks, and its actions were unlocked gradually.

Inference: Overall accuracy matters more, because patients bear the cost of errors. The good reasoning mattered as a sign of potential. The waterfall's flaw is built into its design, since a fixed priority order can't spot stale signals. The agent's errors, by contrast, can be fixed with better prompts and tools. Three ways to combine them:

1. Let the waterfall handle cases where the signals agree, and send conflicts to the agent.
2. Give the agent the V3 model's prediction as a tool.
3. When the two disagree, investigate or escalate.

“We couldn't A/B test with live prescriptions flowing through our system.”

Why not?

From the post: The post gives no reason.

Inference: An A/B test would deliberately route real prescriptions through an unproven system, and each wrong routing leaves a patient without medication for days. The agent's actions, such as texting patients and scheduling calls, also can't be undone. Shadow mode avoids both problems and compares the two systems on the same prescriptions. Its limit is that when the agent picks a different pharmacy, you can't see whether that pick would have worked.

“A denial from a previously valid pharmacy or a provider note pointing somewhere new can prompt it to request a patient preference, suggest a phone call to verify network status, or escalate to a clinical expert.”

How do you actually encode these conditions in code? Would a system that prompts further digging whenever a high-confidence prediction is wrong help with that?

From the post: The post shows no code.

Inference: It probably works in two layers. Ordinary code detects events, such as a pharmacy rejection or a new note, and reopens the case. The LLM then chooses a response from its tools, within guardrails such as a list of allowed actions.

Your idea would help, and it builds on the post's own trigger. One failure on a confident route should flag that route for every pending prescription and trigger a single verification call, instead of letting each patient fail in turn. That fills the gap the post admits: time decay “doesn't tell us that a network has changed.”

How would you improve this system if you were building it today?

Inference:

1. Evidence: treat one failure on a confident route as a warning for every prescription on that route (Q25), and check high-volume routes on a schedule.
2. Decision: let the V3 model handle easy cases and send only conflicts to the agent (Q23).
3. Evaluation: re-route rate misses prescriptions that stall at a valid pharmacy, so also track the time from approval to first fill.

“LLM-as-judge systems have a well-known failure mode: they tend to be overzealous, unnecessarily flagging correct answers.”

Any studies where I can find more details about this besides the source they linked?

From web search: Three more targeted sources:

• VeriFact (Chung et al., NEJM AI 2025) is the post’s own reference [6] and the closest match: an LLM judge checking clinical text against EHRs with the same three labels. It reports the judge “is biased in assigning Not Supported rather than Not Addressed verdicts when compared against humans.”
• Jing et al., “On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation” (NAACL Findings 2025), found Llama 3.1 8B “overly strict” but Claude Haiku “extremely lenient.”
• MiniCheck’s LLM-AggreFact benchmark (Tang et al., 2024) compares fact-checkers that verify claims against grounding documents.

Inference: Which way the bias goes depends on the model and the prompt, so “well-known failure mode” overstates it. Over-flagging is common in strict grounding setups like this one, but it isn’t universal.

“every extraneous flag leads to additional steps”

What additional steps typically?

Inference: The likely sequence is:

• A human reviewer (Forus operations or clinical staff) re-reads the chart to check the flagged claim.
• If the chart is ambiguous or incomplete, someone contacts the provider’s office for clarification or documentation. This is usually asynchronous and can take days.
• The answer is corrected or confirmed, possibly with provider sign-off.
• The form is re-verified and submitted to the payer.

For a false flag, all of this is wasted, because the original answer was right. The main cost is time spent waiting in the queue rather than reviewer minutes, since the PA can’t be submitted until the flag clears. AMA surveys report that PA delays often lead patients to abandon treatment (web). That’s why the post treats false flags as a clinical cost and not just an operational one.

How are the “generated PA answers” being generated in the first place?

From the post: Not described. The post only calls them “generated PA answers” and says the platform “manages PA submission end-to-end.”

From web search: Forus’s site says providers write the prescription in the EHR and “Forus automatically generates and submits prior authorization forms.” Fierce Healthcare reports that each prescription gets its own AI agent that reasons over the patient’s medical history, insurance and finances, “drawing on clinical models, specialized sub-agents and what Forus has learned from millions of cases.”

Inference: The generator is most likely an LLM agent. It pulls the patient’s record through an EHR integration, retrieves evidence for each payer form question, and writes a free-text answer or picks a multiple-choice option. The judge’s retrieve-then-judge design mirrors this. That raises a question the post doesn’t answer: if the generator and judge share retrieval components, they may also share blind spots, such as a note that neither of them retrieves.

“At Forus, we developed an LLM judge to verify generated PA answers”

This is different from standard LLM judges that are used for evaluation, right? More akin to LLM verifier

From the post: Yes. The judge runs on generated PAs to “verify generated PA answers against patient clinical records and specialty-specific guidance,” and its flags send cases to humans (“every flagged case requires human intervention”). That makes it a production gate, not an offline quality score.

Inference: It differs from a typical evaluation judge in three ways:

• When it runs: online, on every case before submission, rather than offline on a test set.
• What it checks against: retrieved patient evidence (groundedness), rather than a reference answer or a quality rubric.
• What its output does: it triggers a routing decision, so a false flag has a direct operational cost. An evaluation judge’s errors only distort a metric.

“Verifier” is a fair term. In the LLM literature, though, that word often means a model that scores reasoning steps or candidate solutions, for example to pick the best of N samples. “Groundedness checker” or “guardrail” is more precise here. This framing also explains why the post evaluates the judge the way you would evaluate a classifier.

Is verification done question by question or in bulk?

From the post: Verdicts are produced per answer. Each answer is decomposed into claims, the claims are judged independently, and “question-level aggregation” rolls them up into an answer-level result. In the Crohn’s example, though, the judge flags an unperturbed answer because it conflicts with another answer, so the judge must be able to see the other answers on the form.

Inference: The unit of verification is one question’s answer, with the rest of the form available as context. The post doesn’t say whether the calls run one at a time, in parallel, or as one form-level prompt. My guess is separate per-answer calls (which can run in parallel), with the form’s context included during decomposition, since the post says decomposition “has to draw on question context, not just the answer.”

“Our system faced a calibration problem”

If they didn’t have a dataset initially, how did they know this for? Or did they assume it was the case given it’s a problem generally?

From the post: The premise is partly off, because they did have a dataset. They had already built “a rigorous gold standard dataset for our PA evals” of correct answers. Running the judge over correct answers and counting flags gives the false positive rate directly, and that is exactly the clean run. What they lacked was examples of wrong answers on the same patients, which is what perturbation adds. The post also points to industry findings [1] that judges lean conservative.

Inference: The post doesn’t say how they first noticed. The most likely route is that human reviewers in production kept clearing flags on answers that turned out to be fine, or that an early clean-run measurement showed a high flag rate. Note that “calibration” here means the judge flags too aggressively relative to the true error rate. It isn’t probability calibration in the statistical sense, because the judge outputs discrete verdicts, not probabilities.

“Emerging work like MedAgentBench [5] is more targeted, and evaluates LLM agents in simulated EHR environments. Still, no standardized benchmark exists for PA question answering.”

What's the difference between these two use cases?

From web search: MedAgentBench (Stanford, NEJM AI 2025) has 300 physician-written tasks across 100 de-identified patient profiles in a simulated EHR that follows the FHIR data standard. Tasks are either queries (retrieve information) or actions (for example, order a medication), and models are scored on task success rate.

Inference: The two differ in three ways:

• Output: MedAgentBench asks for one fact or one action with a single correct result. PA question answering asks for answers to payer questions (diagnosis, prior therapies, severity) that must be justified by chart evidence.
• Data: MedAgentBench draws on structured records (labs, vitals, procedures, diagnoses, orders). PA answers often rest on unstructured progress notes, biopsy reports and outside records.
• Grading: MedAgentBench checks whether the task succeeded. PA answers need every claim grounded in the record, and they must match payer-specific criteria.

In short, MedAgentBench tests whether an agent can operate an EHR, while PA question answering tests whether it can build a defensible case from one.

“rigorous gold standard dataset for our PA evals through a multi-annotator consensus workflow followed by expert clinical review”

Are the annotators different from the expert clinical reviewers? Why? Cost?

Inference: This is a standard tiered labeling design, and each tier handles a different kind of error:

• Random error: several annotators label each item, and taking the consensus removes individual slips. Trained non-physician staff, such as nurses or PA specialists, can do this at lower cost and higher volume.
• Systematic error and hard cases: experts (physicians or pharmacists) review the disagreements and a sample of the agreed labels. This catches mistakes that all the annotators make together.

So yes, cost is a large part of it, because expert time goes only to the cases that need it. Expert adjudication would be needed anyway, since even clinicians often disagree on this task. VeriFact reported that the best agreement between any two clinicians was 88.5% (web).

If an answer is flagged for review, what happens next exactly?

Inference: A plausible flow is:

• The flagged answer goes to a human review queue, together with the failing claim and the evidence the judge cited.
• The reviewer checks the chart and either confirms or overturns the flag.
• If the flag is confirmed, the reviewer corrects or regenerates the answer, and asks the provider for documentation if the chart is missing something.
• The corrected form is re-verified and submitted.

Claim-level verdicts make review faster, because the reviewer checks one failing claim rather than re-verifying the whole answer. The reviewers’ decisions are probably logged, and those logs are a natural source of the “confirmed errors” the post wants to mine (see Q25).

“each answer is decomposed into atomic claims”

Is this done by an LLM? If so, is it the same kind of LLM as the generation model?

From the post: It isn’t stated outright, but it’s strongly implied. The post notes that one claim (“Humira was the most recent biologic therapy”) “appears nowhere in the answer” and is implied by the question, so “decomposition has to draw on question context.” Rule-based sentence splitting can’t produce a claim like that. The method is also adapted from VeriFact and FActScore.

From web search: Both of those use an LLM for decomposition. VeriFact used Llama 3.1 70B to extract “simple sentences that encapsulate Subject-Object-Predicate relationships.”

Inference: It is almost certainly an LLM. The post doesn’t say which model, or whether it matches the generator. Decomposition is a fairly mechanical task, so a smaller and cheaper model would be a reasonable choice, with the strongest model saved for the judgment step.

Is the judge model the same as the generation model?

From the post: Not stated.

Inference: There are two reasons to prefer a different model, ideally from a different family. There’s no evidence either way about what they did.

• Self-preference: LLM evaluators tend to rate their own outputs more favorably (Panickssery et al., 2024, “LLM Evaluators Recognize and Favor Their Own Generations”; background knowledge).
• Correlated blind spots: a model that misreads a chart pattern while generating may misread it the same way while verifying.

One point cuts the other way. Their main problem is over-flagging, not leniency, so self-preference doesn’t seem to be their main failure. Their remark that “using another LLM to judge the judge is circular” shows they are aware of correlated errors in general. The same independence concern applies to the perturbation model (see Q21).

“We expand each claim into multiple semantic search queries”

How is the expansion done? SPLADE?

From the post: The method isn’t named. The example queries are full natural-language paraphrases, such as “symptoms well controlled on Humira” and “stable disease on anti-TNF therapy.” They vary clinical synonyms, brand and generic drug names, and drug class.

Inference: This looks like LLM-generated query rewriting (often called multi-query retrieval), not SPLADE. SPLADE is a learned sparse encoder. It turns one text into a single sparse vector of weighted vocabulary terms, including related terms that weren’t in the text. It doesn’t produce several readable query sentences, so you wouldn’t describe its output as “multiple semantic search queries.” The results of the separate queries are then presumably merged, for example by taking the union and removing duplicates, or by reciprocal rank fusion. The post doesn’t say which. Phrasing several queries around the same claim also makes it more likely that contradicting notes get retrieved, like the April 2026 flare note in their example.

Is their search only semantic or includes lexical as well? Why?

From the post: Only “semantic search queries” are mentioned. Lexical search isn’t.

Inference: We can’t tell, but a hybrid design would make sense, because each method covers a weakness of the other:

• Semantic (dense) search handles the synonym problem the post emphasizes, such as “eczema” versus “atopic dermatitis.”
• Lexical search (for example BM25) handles exact tokens that dense embeddings represent poorly. These include ICD codes (K50.90), lab values (CRP 12 mg/L), doses (90 mg every 8 weeks), dates and rare drug names, and PA questions depend heavily on exactly these.

VeriFact, which their pipeline is adapted from, compared dense search, hybrid dense-plus-sparse search, and hybrid search with a cross-encoder reranker (web). Two other points apply. Structured data such as medication lists and labs may be fetched by direct lookup rather than search. And the LLM query expansion partly substitutes for lexical recall, because it writes both brand and generic names into the queries.

Any domain adaptation for semantic expansion?

From the post: Nothing beyond the examples. They show clinical synonyms (“eczema” for “atopic dermatitis,” “failed therapy” for “inadequate response”) and drug-name variants (Humira, adalimumab, anti-TNF therapy).

Inference: Domain knowledge could enter at three points, none of them confirmed:

• Query generation: an LLM prompted with clinical instructions and examples. This is the cheapest option and the most likely.
• Vocabulary normalization: mapping terms to medical ontologies, such as RxNorm for drugs and SNOMED CT or ICD-10 for diagnoses, so that “Humira” and “adalimumab” resolve to the same concept.
• Embeddings: a biomedical retrieval model such as MedCPT, or a general embedding model fine-tuned on their own claim-to-evidence pairs, which their gold standard data could supply.

My guess is that prompting does most of the work, possibly with an ontology layer for drug names.

What granularity/chunking strategy for retrieved evidence?

From the post: Not stated. The diagrams show evidence as short excerpts labeled with source type and date, such as “Progress note · Apr 2026: ‘flare; symptoms recurring’,” “Lab results · Apr 2026” and “Medication list.”

Inference: That suggests sentence- or passage-sized chunks that carry metadata (document type and date). Structured data such as labs and medications is probably stored as records rather than text. Dates matter because many PA claims are about time, such as “most recent biologic” or “sustained remission.” The judge can only see that an April 2026 flare contradicts a January 2026 remission note if each chunk carries its date.

From web search: VeriFact, their reference design, split notes into 128-token chunks along semantic boundaries, then broke those into sentences or atomic propositions. It retrieved the top N facts per claim. It tested N up to 50, and performance was still improving at 50.

“Not Supported: the claim conflicts with the evidence, either through direct contradiction or through a significant omission or fabrication.”

How is the omission part different from Not Addressed?

From the post: Not Addressed means “the evidence is silent on the question.” The judge’s rationale for the adherence example, which appears when you click the verdict, draws the line: silence “isn’t fabrication either, since the claim asserts no specific finding the chart should contain.”

Inference: The difference is whether the record says anything on the point:

• Omission: the record does contain relevant information, and the answer leaves out the part that changes its meaning. For example, the answer says “methotrexate was tried” when the record says it was stopped for intolerance. That matters for a “tried and failed” question.
• Not Addressed: the record contains nothing on the point, so there is nothing to omit.

Fabrication sits between them. The claim asserts a specific finding, such as a lab value, that the chart would contain if it were true, and the chart doesn’t contain it. This boundary is hard even for people. In VeriFact, clinicians agreed only about a third of the time on Not Supported labels, and the LLM judge leaned toward Not Supported over Not Addressed (web).

“Worst-verdict aggregation means the one Not Supported claim flags the whole answer. But the Not Addressed claim does not”

So then what happens if there is a Not Addressed claim? Shouldn’t that still go to additional review?

From the post: A Not Addressed claim doesn’t flag the answer, and “that’s the distinction that keeps the judge from over-flagging.” Their reasoning is that records are routinely incomplete, so treating silence as Not Supported “would punish every incomplete chart.” The post doesn’t say what happens to these claims afterward.

Inference: Your concern is valid. The answer goes to the payer containing a claim nobody can document, and payers can deny or audit based on documentation. A middle path is to route these claims by how much they matter:

• Claims the payer’s criteria depend on, such as prior therapies or severity scores, go to a lighter “documentation gap” queue, or the provider is asked for the documentation.
• Incidental claims, such as adherence when the question didn’t ask about it, pass but get logged.

There’s also a separate question: why is the generator asserting things the chart doesn’t mention? The share of Not Addressed claims is worth tracking as a signal of generator quality, even if it never triggers a flag.

“the perturbed run measures the detection rate”

Detection rate = true negative rate?

Inference: It depends on which class you call positive:

• If a wrong answer is the positive class and a flag is a positive prediction, which is how the post frames it, then detection rate is the true positive rate for errors (also called recall or sensitivity).
• If you call a correct answer the positive class, detection rate becomes the true negative rate (specificity).

Both readings are internally consistent. The post uses the first, which is the standard one for error detection; under that framing, “false positive” means a correct answer that was flagged. Two details about the denominator: accidentally correct perturbations are excluded from it, and only direct detections count, with consistency-based flags tracked separately.

“because detection rate is only meaningful once we know the judge isn't excessively flagging”

Why?

Inference: Detection rate alone rewards flagging more. A judge that flags every answer has a 100% detection rate. A judge that flags answers at random with probability p has a detection rate of p, with no skill at all. So detection rate only shows skill as the gap above the false positive rate. For example, 60% detection with a 40% false positive rate means the judge barely tells wrong answers from right ones, while 60% detection with a 2% false positive rate is a useful judge. The two numbers have to be read as a pair, like recall and precision.

There’s a second reason. With a high false positive rate, some flags on perturbed answers may be for reasons unrelated to the perturbation, so the detection count overstates real catches. In principle, their claim-level rationales could be used to check that each flag points at the perturbed claim. Anchoring on the false positive rate also reflects their priority, since over-flagging is the failure they set out to fix.

“One perturbation, two flags, and the second one is, interestingly, a behavior we hadn't explicitly designed the judge to produce.”

I don’t fully understand the difference between these two flags.

From the post: The interactive demo labels the two flags explicitly:

• Flag 1, “direct detection,” is on Q1, the answer they changed. The diagnosis now says ulcerative colitis, but the chart documents Crohn’s disease, so the answer contradicts the record.
• Flag 2, “consistency-based,” is on Q2, an answer they didn’t change. “Stricturing disease with perianal fistulas; transmural inflammation” is still true according to the chart. But those features are hallmarks of Crohn’s, not ulcerative colitis. The demo’s rationale says Q2 “no longer fits the diagnosis it’s meant to support, so the judge flags the mismatch.” Perturbing Q2 instead produces the mirror case: Q1 gets flagged “even though this answer is chart-supported.”

Inference: Flag 1 checks an answer against the record, and flag 2 checks an answer against another answer. Measured against the chart alone, flag 2 is a false positive. It’s a useful one, though, because the submitted form as a whole would no longer make sense. That’s why the post counts the two kinds separately.

“Our approach was to build a perturbation model, seeded with structured context from the patient's record, so its wrong answers stay anchored to the patient's actual medical history. For free-text questions, the model generates a plausible wrong answer.”

So they’re just giving the model relevant context and asking it to generate answers that seem plausible but are wrong? Any configurations to think about beyond the context and the prompt?

From the post: Essentially yes. The model gets structured context from the record. For free-text questions it writes a plausible wrong answer. For multiple-choice questions it picks a different option that is “clinically coherent in context but factually unsupported.” The constraints are definitive assertions, specific and falsifiable claims, no attestations, and about 30% of questions perturbed per form.

Inference: Other settings worth deciding, grouped by stage:

• Before generation: which questions to perturb (at random, or stratified by question type), and which error type and level of subtlety to aim for. Ideally these come from an error taxonomy (see Q26).
• During generation: which model to use (ideally from a different family than the judge, to avoid shared blind spots), and how many candidates to sample per question.
• After generation: validity checks (is the new answer actually contradicted by the chart, and is it falsifiable?), plus logging so that runs can be reproduced.
• Across the dataset: splitting the train and test sets by patient rather than by question, so prompt tuning never sees a held-out patient’s record.

“Three ways a synthetic perturbation can fail to be a useful test”

Once they observed these, what did they do to avoid them? Prompt adjustments?

From the post: Each failure mode got a different fix, applied at a different stage:

• Accidentally correct (for example, they swapped in azathioprine, but the patient really did try it): “Detect & exclude from metrics,” which is a check after generation.
• Unfalsifiable (for example, “all alternative therapies were tried”): “Require specific, falsifiable claims,” which is a constraint on generation.
• Administrative noise (attestations): “Filter out before perturbing,” which limits which questions are eligible.

Inference: The post doesn’t say how these were implemented. The likely mix is prompt instructions for falsifiability, a question-type filter for attestations, and some kind of validator for accidentally correct cases. That validator has a trap. If it relies on the judge’s own “Supported” verdict to decide that a perturbation was accidentally correct, it will also throw out real misses and inflate the detection rate. The check has to be independent of the judge, for example a lookup in the structured medication history, or human review of every excluded case.

“In production, real errors often cascade through related questions”

What does this mean?

Inference: A PA form asks many related questions about one patient, and the generator answers all of them from the same reading of the chart. If that reading is wrong at one point, the error spreads to every answer that depends on it. For example, if the generator takes the diagnosis to be ulcerative colitis, it may also list therapies relevant to that disease, report a different severity score, and argue medical necessity for the wrong condition. So real errors are correlated across questions, while their perturbations change one answer at a time.

This matters for the consistency signal. In a full cascade, all the answers could be wrong in a way that is consistent with each other. The cross-answer check then wouldn’t fire, and only checking against the chart would catch the error. Single-answer perturbations may therefore make consistency-based detection look more useful than it really is.

“the judge flags a different, unperturbed answer because it conflicts with the perturbed one”

Is their judge actually looking for inconsistency across questions or is this a hypothetical?

From the post: It’s described as observed behavior, not a hypothetical: “The judge flags two things,” and the second is “a behavior we hadn’t explicitly designed the judge to produce.” They also say they have to separate direct detection from consistency-based detection in their evaluation, which would only be necessary if it actually happens in their runs. The sentence you quoted is how they define that second category. They don’t report how often it happens, though, and the interactive demo is scripted.

Inference: Since they didn’t design it, the judge isn’t explicitly checking for consistency across questions; the behavior comes from somewhere else. A plausible mechanism is decomposition. For Q2 (“What clinical features support the diagnosis?”), decomposition that draws on question context would produce an implied claim like “these features support a diagnosis of ulcerative colitis,” taking the diagnosis from Q1’s answer. The chart contradicts that claim, so Q2 gets flagged. If that’s right, consistency detection only works when one question explicitly refers to another answer.

“One is mining production data for confirmed errors”

How is an error confirmed?

Inference: Possible sources of confirmation, from most to least direct:

• Reviewer outcome: a human checks a flagged answer against the chart and corrects it.
• Provider correction: the prescribing office amends an answer after seeing the form.
• Payer response: a denial or a request for more information that points at a specific answer. Denial reasons are often vague, so linking them to an answer takes manual work.
• Audit sampling: running the gold-standard annotation process on random production cases, including ones the judge passed.

The first three mostly surface errors someone already noticed, and the reviewer route only confirms errors that the judge itself flagged. Errors the judge misses won’t show up unless something downstream catches them. So mining confirmed errors under-represents exactly the misses the evaluation most needs, and random audits are the way to correct for that.

“Another is building an error taxonomy from real cases so that perturbations cover the actual distribution of mistakes rather than the ones a model finds easy to invent”

How is this different from mining production data in the previous sentence?

Inference: They work at different levels:

• Mining gives you individual real errors. These can seed the perturbation model directly, for example as few-shot examples, which makes each perturbation look more like a real mistake.
• A taxonomy classifies those errors and records how often each type occurs. Example types include a wrong date range, an outdated lab value, past history stated as current, a wrong drug in the prior-therapy list, or a flipped negation. The taxonomy tells the generator how many of each type to produce, so the test set matches the real mix. Otherwise it would over-represent errors that are easy for a model to invent, such as swapping a diagnosis.

Mining makes each example more realistic, and the taxonomy controls which kinds of errors appear and in what proportions. A taxonomy also lets you report detection rate per error type, which shows where the judge is weak. You build the taxonomy from mined errors, so mining comes first.

“where the perturbation model is trained to produce errors the judge misses”

How would you actually do this? Prompt iteration, fine-tuning or something else?

Inference: Three options, from cheapest to most involved:

• Search without training: generate many candidates, keep the ones the judge misses, and feed them back as few-shot examples in the next round. No model weights change.
• Preference fine-tuning (for example DPO): train on pairs in which the missed perturbation is preferred over the caught one.
• Reinforcement learning: reward the generator whenever the judge misses its error.

Every option needs a validity check that doesn’t depend on the judge. Otherwise the generator learns the easiest way to beat the judge, which is to produce accidentally correct or unfalsifiable perturbations, the same failure modes the post documented. A perturbation should count as a success only if it meets three conditions. It must be verifiably wrong against the chart (checked by structured lookup, a separate checker that sees the gold answer, or human spot checks). It must be clinically plausible. And the judge must miss it. Restricting generation to the error types in the taxonomy (Q26) also keeps the errors realistic.

What could “production monitoring” actually look like for their use-case?

Inference: Grouped by what each signal tells you:

• Over-flagging: the share of flags that reviewers overturn, broken down by drug, payer and question type.
• Missed errors: payer denials or information requests on answers the judge passed, plus random audits of unflagged answers.
• Drift: changes over time in the flag rate and in the mix of verdicts. A rising share of Not Addressed claims, for example, can mean that retrieval broke or that a new EHR integration is sending incomplete records.
• Retrieval health: the share of claims for which nothing was retrieved.
• Patient impact: time from prescription to submission for flagged versus unflagged cases, which shows what over-flagging actually costs.

A useful addition is canary testing: regularly running known perturbed forms through the live pipeline to confirm that detection hasn’t dropped after a model or prompt change.

How would you improve this system if you were building it today?

Grouped by component:

• The judge: make the cross-question consistency check a deliberate step instead of a side effect, so it can be measured and can catch full cascades (Q23). Route claims that payer criteria depend on separately, so a Not Addressed verdict on one of them goes to a documentation queue (Q17).
• The flagging threshold: have the judge output a confidence for each claim, and set the threshold from the relative cost of a missed error versus a false flag, instead of using a fixed worst-verdict rule.
• Perturbations: generate them from an error taxonomy (Q26), include errors that cascade across several answers, and check for accidentally correct cases independently of the judge (Q22).
• Evaluation reporting: report detection rate by error type with confidence intervals. Also keep a small held-out set of real confirmed production errors as the final test, since synthetic detection rates probably overstate performance on real errors.
• Monitoring: add random audits of unflagged answers, which are the only unbiased way to estimate how many errors the judge misses in production (Q25).

“In the case of AI agents, Cresta suggests how to modify the agent’s prompt or other parts of its configuration and then runs tests to validate that the agent’s performance improved as a result.“

Does “Cresta” here refer to a Cresta employee or a Cresta system? In other words, is this done manually or automatically?

[Source] The post doesn't say. The sentence is built as a parallel to the preceding one about human agents, where Cresta the platform delivers hints and coaching, so the grammar points toward the system.

[Inference] The two verbs are doing different amounts of work. "Runs tests" is plainly automated — that's the evaluator toolkit and simulation suite described later. "Suggests how to modify" is the softer claim, and most likely means the platform surfaces a diagnosis (this turn failed this behavior, here's why) while a person decides the actual edit. The post's own Lessons Learned section supports that reading, since its entire argument is that humans stay in the loop on targeted decisions. So the honest summary is automated detection and automated validation, with a human-authored change in between. "Suggests" is the kind of verb that lets a vendor describe either arrangement.

Are Cresta’s agents primarily text-based or voice?

[Source] This post is modality-agnostic. It refers to "chat and call transcripts" and mentions both inbound and outbound conversations.

[Search] Voice is clearly the emphasis across the rest of their writing. Their Voice Virtual Agent announcement says voice is "the top used channel at over 70% of interactions" and criticizes competitors for "focusing exclusively on chat," and their latency engineering post is entirely about the voice pipeline — speech recognition, voice activity detection, speech synthesis.

[Inference] Both channels are supported, but the engineering problems they choose to write about are voice problems. The "interruptions" metric listed in this post only makes sense for voice at all. I'd read the post as voice-first with chat along for the ride.

“Cresta fixes this cold start problem by grounding the design in historical data”

Doesn’t cold start imply no historical data?

[Inference] You're right that the term is being stretched. Cold start conventionally means having no data about a new user or item. Here the cold start belongs to the agent, not the enterprise: the new AI agent has no interaction history of its own, while the enterprise has millions of transcripts. So what they're describing is a transfer-learning move — bootstrap the new agent from the behavior of the incumbent human agents. That's a reasonable idea, but calling it "fixing cold start" is marketing language; a stricter description would be a warm start from a related distribution. And the distributions genuinely differ: customers ask different things of a bot than of a person, and they behave differently once they know which one they're talking to.

What would you do if you actually didn’t have any historical data and wanted to launch something like this?

[Inference] Roughly in order of cost, I'd do five things. First, mine the written artifacts that exist even without transcripts — standard operating procedures, help-center articles, canned agent responses, past ticket subject lines, and abandoned queries typed into a chat widget. Those give you intents without needing dialogue. Second, have subject-matter experts record twenty to fifty think-aloud role-plays, which becomes the seed corpus your personas are drawn from. Third, generate synthetic conversations from the procedure documents, but validate them against the role-plays so you aren't only testing the agent against your own assumptions. Fourth, launch deliberately narrow, or run the agent in shadow mode alongside human agents so it produces responses that get scored but never sent. Fifth, instrument heavily, because the first few thousand real conversations are the actual asset and you want the relabeling loop to be cheap from day one.

The thing to watch is that the first three all encode the same prior. Only real traffic breaks it.

“finds the major clusters of customer issues”

How?

[Search] Their Insights announcement is more specific: Topic Discovery uses "unsupervised LLM clustering on conversation reasons," and generates "an exhaustive taxonomy and up to three levels of subcategories" without teams having to predefine categories.

[Inference] In practice that reads as a two-stage pipeline. An LLM first extracts a short conversation reason from each transcript, compressing a long call into a normalized phrase. Clustering then runs over those compressed reasons rather than over raw transcripts — most likely embeddings plus a clustering algorithm, with an LLM naming clusters and merging them hierarchically into the taxonomy. Clustering on extracted reasons instead of full text is the important design decision, because small talk, greetings, and hold messages would otherwise dominate the similarity signal and you'd end up clustering conversations by style rather than by problem.

Why do so many of their cluster examples sound very similar to each other?

[Inference] Two explanations, and they have different implications. If the screenshot is showing sibling leaves beneath one parent node, then similarity is expected and fine — "billing, unexpected charge" and "billing, duplicate charge" are supposed to differ by one discriminating detail, and the three-level taxonomy makes that likely. The less comfortable explanation is under-merging, which is a real failure mode of LLM-driven taxonomy induction. The model finds distinctions that are linguistically genuine but operationally meaningless, because nothing in the clustering objective penalizes redundancy. The test I'd apply: if two clusters would route to the same conversation flow and trigger the same tool calls, the taxonomy is over-split for the purpose it's being used for. Your suspicion is worth holding onto.

“how human agents resolved them”

Do we know for sure that the human agents' approaches were correct? How?

[Inference] "Proven" is doing unearned work there. Transcripts record what humans did, not what was correct. There are three ways to get closer to correctness, none of them free. You can filter by outcome rather than by behavior, keeping only conversations with a good resolution signal such as no callback within some window, positive satisfaction score, or no subsequent refund reversal — which turns plain imitation into outcome-weighted imitation. You can stratify by agent and learn only from the top quartile on the relevant metric, which is the standard contact-center move and something Cresta's quality management product already scores. Or you can treat the mined flow as a hypothesis rather than ground truth and put it in front of subject-matter experts before it becomes the spec. There's also a survivorship risk worth naming: shortcuts agents use to hit handle-time targets look indistinguishable from best practice in a transcript.

“Insights surfaces real customer phrasings and look-alike intents”

What does this mean?

[Inference] They're two separate things. Real customer phrasings means the utterances people actually produce, as opposed to how the business names the intent internally — customers say "I got charged twice," while the internal taxonomy says "duplicate transaction dispute," and it's the former you need in the router's examples. Look-alike intents means pairs that sit close together in surface form but need completely different handling. "I want to cancel my order" and "I want to cancel my subscription" share most of their words and then diverge into different flows touching different systems. Surfacing those pairs during scoping tells you in advance where your router will confuse itself, so you can add a disambiguation turn or supply discriminating examples before the confusion shows up in production traffic.

“we can explicitly define what the AI agent will handle and what falls out of scope”

What are realistic examples of things that would fall out of scope?

[Inference] Exclusions tend to fall into four groups. There's regulated or liability-bearing work — debt collection promises, insurance coverage determinations, medical or legal advice, anything that creates a binding commitment on the company's behalf. There's high blast radius — refunds above a threshold, account closures, changing a payment method, anything hard to reverse. There's emotionally loaded contact — bereavement, fraud victims, threats of self-harm, and retention saves where human judgment carries the conversation. And there's the long tail, meaning intents that appear a handful of times a month where the cost of building and maintaining a flow exceeds the containment you'd gain. By raw count the long tail is the largest category. The first two are the ones that get written into the contract.

How is “automation readiness” determined?

[Source] This post lists it as something Insights quantifies, with no method given.

[Search] Their dedicated Automation Discovery post is more forthcoming: "The score is driven by structural and operational signals observed in the data, including the frequency and nature of deviations, as well as the level of inferred integration complexity."

[Inference] Unpacking those two signals, deviation frequency is a proxy for how well-structured an intent is — if most conversations follow one dominant path, a state machine can capture it, and if they scatter, the intent needs judgment. Integration complexity is a proxy for build cost, meaning how many backend systems the flow has to touch. So readiness is roughly path regularity divided by integration cost, weighted by volume. Worth noticing that this is a build-feasibility score, not a risk score. Nothing in it speaks to whether automating a given intent is safe, and the post's framing quietly blends the two axes.

“Automation Discovery reconstructs representative dialogue flows from the transcripts”

If this tool is looking at a very large number of transcripts (presumably not fitting in a single context window), how is it generating the desired information correctly?

[Source] Not addressed here. Their Automation Discovery post says only that they use "LLMs together with deterministic analysis."

[Inference] Nobody puts a million transcripts into one context, so the shape is almost certainly a four-stage pipeline. First, per-transcript extraction, running one model call per conversation in parallel, compressing each into a structured trace: the reason, an ordered list of steps, tool-like events, and the outcome. This is the only stage that sees raw text and it's parallel. Second, cluster the traces rather than the text. Third, align the step sequences within each cluster, which is closer to process mining than to language modeling and can be done deterministically with frequent-sequence mining or transition counting. Fourth, bring an LLM back in only at the end, to name states and write descriptions over an aggregate that now fits comfortably in context. Their phrase "deterministic analysis" most plausibly points at the third stage. The quality risk sits in the first: whatever schema the extractor uses silently bounds what the final state machine is capable of expressing.

“State machine skeleton: It identifies the dominant paths customers take to reach resolution”

Per intent, or more general?

[Inference] Per intent, almost certainly. A single global state machine spanning all customer issues would be close to useless, because the states for rescheduling an appointment and disputing a charge share nothing except authentication. The pipeline ordering in the post supports this: Insights clusters issues first, then Automation Discovery reconstructs flows, so the natural unit is one flow per cluster, with shared preamble states like greeting and identity verification and shared exit states like escalation factored out as common subgraphs. Their decentralized-architecture post is consistent with this, since it names sub-agents for "authentication, policy verification, troubleshooting workflows, payment handling" — which is exactly a shared-preamble-plus-per-intent-body decomposition.

Is a human the primary consumer of the output of the Automation Discovery tool to guide the development of the agent? Or is it being directly fed into a system somewhere?

[Search] Both, according to their Automation Discovery post. It supports human inspection, and it "can export a structured draft prompt derived from the workflow, serving as a scaffold for AI agents."

[Inference] "Draft" and "scaffold" are the operative words — this is a human-reviewed handoff, not a closed loop. That's the right call given what the artifact is. The mined flow encodes whatever the human agents happened to do, so pushing it straight into production would automate their mistakes at scale. The post you're reading agrees implicitly by calling the scoping output a "v1 spec," which presupposes somebody reviews it.

“Tool catalog: identifies critical data dependencies and system touchpoints”

How can you get this information just from transcripts?

[Search] Their Automation Discovery post answers this directly: "When agents say things like 'let me check your order,' 'I'll submit a claim,' or 'I need to verify eligibility,' those patterns often indicate interaction with backend systems."

[Inference] So it's linguistic evidence that a system call happened, not observation of the call itself. Two further signals are usually available and they don't mention either: long silences or explicit hold events in a call mark a lookup, and the shape of the data the agent reads back — an order number, a balance, a ship date — reveals both which system was queried and roughly what it returned. The limitation is real, though. This gives you a hypothesis list of integrations with no guarantee of completeness, no API contract, and no visibility into any system the agent used without narrating it. Screen recordings or CRM audit logs would be the ground truth; transcripts are the cheap proxy for it.

How is “Tool use and data needs” different from “Tool catalog”?

[Source] The post lists them as separate bullets and the distinction it draws is thin. Tool catalog identifies "critical data dependencies and system touchpoints" and enables function schema specification; tool use and data needs catalogues required internal systems and data such as CRM lookups and order adjustments, and informs function call specifications.

[Inference] As written these overlap heavily, which by a MECE standard is a defect in the post rather than a distinction you're missing. The most charitable reading is that the catalog is the inventory — which systems exist and get touched — while tool use and data needs is the contract, meaning what each call requires as arguments and what it returns. Inventory versus signature.

“that might merit separate specialized agents”

Is the appeal of separate agents a clean context window or different capabilities (whether through a different model, harness or skills)?

[Source] This post doesn't say.

[Search] Their decentralized-architecture post makes the case, and the reasons it gives are mostly neither of your two options. It cites avoiding a single point of failure, since if a central orchestrator "falters due to an unexpected scenario, a model update, or a spike in call volume, the entire experience may degrade"; avoiding errors that "accumulate across long conversations"; and latency, because "every decision and turn reroutes through the orchestrator." It does also describe keeping "the AI focused on the immediate step," which is your clean-context argument.

[Inference] In practice I think prompt focus is the dominant real benefit, so your first option. A sub-agent's prompt contains only the rules for its own task, which raises instruction-following accuracy and cuts token cost. Independent testability is second, and this post names it outright — "each skill can be built and tested independently." Running different models per sub-agent is easy to do and occasionally worth it, a small fast model for authentication and a stronger one for troubleshooting, but it isn't the headline reason.

“and a world state the agent must navigate: authentication, entitlements, status, partial data, the messy stuff”

Why is this in the simulated visitor spec as opposed to the test spec? Does their version of visitor both the simulated user and the task?

[Source] Yes, it bundles them. The simulated visitor spec as described carries four things: a persona with tone and vocabulary, a goal defining "what 'done' means," the world state, and behavior policies covering how the person asks, clarifies, and pushes back. Separately the post says "we know 'what should happen' in each scenario," so the expected outcome travels with the scenario too.

[Inference] Your instinct that these are separable is right, and separating them would be the better design. Persona, world state, and goal form a cross-product you'd want to sample along independently, so you could hold a persona fixed while varying account state, or the reverse. Bundling them into a single artifact means every combination is a hand-built object and your coverage is limited to whatever someone thought to write down. The likely reason it's bundled is how they're generated: if each visitor is derived from one real conversation, the persona and the world state come from the same source record and arrive already entangled.

“Because these simulated visitors are generated from de-identified real conversations”

How generated?

[Source] No method given beyond the de-identification.

[Inference] The plausible pipeline is to take a real transcript, strip personally identifying information using named-entity redaction plus format-based scrubbing for things like card and account numbers, then have an LLM read the cleaned transcript and emit the structured spec. It would infer the persona from how the customer wrote or spoke, the goal from what they were trying to achieve, the world state from facts established during the conversation — were they authenticated, did they have an active policy, was the order partially shipped — and the behavior policies from observed conduct such as pushing back, wandering off-topic, or misunderstanding a rule. Then you discard the original transcript and keep only the spec, which is what makes the artifact privacy-safe: it's a description of a customer rather than a copy of one. Whether they check the generated spec back against the original for fidelity is unstated, and that's the thing I'd most want to know.

“simulate full multi-turn conversations between each virtual customer and the AI agent”

1. How to sandbox to ensure no cross-context contamination?
2. Different models or same? Does it matter?

[Inference] (a) The contamination risks are distinct and each needs its own control. Within a single run, the visitor model must not see the agent's system prompt, tools, or internal state — only its own spec plus the dialogue so far — which means two separate processes with separate contexts, not two personas sharing one context. Across runs, you need no shared memory, fresh conversation identifiers, and a reset of the mock backend between runs, because otherwise run N's writes, a payment recorded or a ticket opened, silently change run N+1's world state. The agent's own tools have to hit mocks or a seeded test database rather than production, and this is the one that catches people out, since a tool that looks read-only often still writes an audit record. Finally the grader shouldn't know which configuration produced a transcript, or you introduce scoring bias precisely when you're diffing prompts against each other.

(b) Different models, and yes it matters. Using the same model on both sides is the classic evaluation trap: the visitor and the agent share blind spots and failure modes, so the agent scores better than it deserves and the visitor never generates the confusions a different model — or a real person — would produce. Running a different family for the visitor, and ideally a third for the judge, buys you independence. The cost is that your simulated population now inherits whatever quirks the visitor model has, which is exactly why the pool still needs to be grounded in real transcripts.

“The result of each simulation run is a scored transcript with precise reasons for any failures and artifacts users can diff across prompts, tools, or models.”

Same scoring methodology as online and human agents?

[Source] Partly, and the post is explicit about the shared part. It says "a lot of AI agent requirements map exactly to behaviors and are the same for human and AI agents," and that customers running both "can often share the same behavior definitions between the two." That's about Opera behavior definitions applied in production.

[Inference] But simulation scoring has something production scoring structurally cannot have, which is a known expected outcome — the post's "we know 'what should happen' in each scenario." So simulation can score outcome correctness by direct comparison, while production can only score behavior adherence by judgment, because nobody knows the counterfactual for a live call. My read is a two-layer scheme: shared behavior evaluators, the Opera definitions, applied everywhere, plus an outcome-comparison layer that exists only offline. That asymmetry is worth holding onto, because a high simulation pass rate and good production behavior scores are not measuring the same thing and can move independently.

“Because the simulator can spin up hundreds of variations of a scenario, it’s great for catching edge cases that never surfaced in a low-volume manual test.”

How to encourage diversity and exploration to uncover more edge cases?

[Inference] Naive sampling from one model collapses toward the mode, and you end up with hundreds of near-identical polite customers. The things that actually work: factorize and grid rather than sample, meaning enumerate the axes explicitly — authentication state, data completeness, emotional register, language proficiency, interruption behavior — and cover the cross-product instead of asking a model for a hundred varied customers. Seed from the tail of real data by sampling source transcripts inversely to cluster frequency, so rare intents are over-represented in the visitor pool. Generate adversarially by conditioning the visitor on a known failure mode, such as insisting on a policy exception, giving the account number with one digit wrong, or changing goal mid-conversation. Apply novelty pressure by embedding generated scenarios and rejecting any that sit too close to an existing pool member, which is cheap and surprisingly effective. And mine production for what you missed, which is what they describe doing post-launch and is the only genuine source of unknown unknowns.

The measurable version of all this is coverage over the state machine's transitions: which edges has the pool never exercised?

“after go-live, we also keep expanding the pool with new conversation patterns as real users interact with the agent”

Automatically or manually? How?

[Source] Unstated. The full sentence continues "so we can continually run regression tests."

[Inference] The mining step is almost certainly automatic, since it's the same visitor-generation pipeline from question 18 pointed at production transcripts instead of historical ones. The interesting decision is candidate selection, and the natural signals are conversations where the agent escalated, where the customer repeated themselves or expressed frustration, where an evaluator flagged a behavior violation, and where the conversation reason falls outside the existing taxonomy.

[Search] That last one is exactly what their Trends feature detects — their Insights post describes Real-Time Trends surfacing "emerging phrases, unusual spikes, and cross-cutting patterns," which is a natural feed into this.

[Inference] Promotion into the regression suite should be manual, or at minimum reviewed. An automatically added test encodes whatever the agent did as the expected behavior unless a human supplies the expected outcome, and the post's own emphasis on knowing "what should happen" implies someone has to.

“encapsulates everything the agent needs: its prompts, decision logic”

What is “decision logic”?

[Source] The post doesn't define it. It lists the term alongside prompts, tool integrations, and guardrails, and elsewhere mentions adjusting a "workflow stage" as a thing you can do to a config.

[Search] Their decentralized-architecture post is more concrete, describing "deterministic state management" that keeps "track of exactly where a customer is in a process, triggering the right actions at the right moment," combined with dynamic prompting that keeps "the AI focused on the immediate step."

[Inference] So decision logic is the non-LLM part of control flow — the state machine from the scoping phase, rendered as configuration. Which stage the conversation is in, which transitions are legal from here, which tool fires on entering a state, when control hands off to another sub-agent, and when the conversation escalates. The language model handles what gets said within a state; the decision logic constrains which states are reachable at all. That's the hybrid architecture they advertise, and it's also why a config bundle is diffable in a way a single prompt isn't: state transitions are structured data, prompt prose is not.

“Under the hood, the agent’s reasoning and capabilities are extended by Cresta’s AI Agent Framework which is a declarative framework that lets developers register backend functions”

What is “a declarative framework”?

[Inference] Declarative means you describe what something is and the framework works out how to execute it, as opposed to imperative, where you write the control flow yourself. Concretely here, a developer writes an ordinary function and annotates it with a schema — a name, a description, typed parameters, a return type — and the framework handles everything downstream: exposing it to the model as a callable tool, validating the model's arguments against the schema, invoking the function, and feeding the result back into the conversation. The developer never writes the "if the model requested this tool, parse its arguments and dispatch" plumbing. Python decorators are the usual surface syntax for this. The real payoff is that the tool's declaration and its implementation are the same artifact and therefore can't drift apart — which matters, because "schema mismatches" is one of the failure types the post names in its optimization loop.

“The framework is optimized for ultra-low latency”

How?

[Source] The post asserts it without method, tying it only to "preserving a smooth real-time conversation."

[Search] Cresta has a whole post on this. Their stated budget is that pauses beyond about 300 milliseconds "can feel unnatural" and anything past roughly 1.5 seconds "can rapidly degrade the experience." Component budgets they cite: audio preprocessing 25 to 50 ms, speech recognition 200 to 300 ms, turn detection at least 600 ms, language model time-to-first-token anywhere from 250 ms to over a second, and speech synthesis first byte 100 to 500 ms. The named techniques are streaming APIs with no DNS lookup in the critical path, reused connections to the model, WebRTC transport which they say "can reduce latency by up to 300ms," guardrail model calls issued concurrently with the main call rather than after it, speculative triggering meaning "starting the LLM call before the user fully stops speaking," hedging meaning "launching multiple LLM calls in parallel and using whichever returns first," and smaller models in the live loop since "reasoning models generally can't be used within the live response loop."

[Search] On tool calls specifically, which is what the framework sentence is really about, their sharpest observation is that if you look something up before the model can say anything at all, "the 1 LLM first-token latency becomes 1 LLM latency + 1 LLM first-token latency." Their mitigations are to fire predictable lookups concurrently at the start of the conversation, emit filler or wait messages for calls under about ten seconds, and run anything longer asynchronously.

[Inference] That last cluster is the actual content behind "optimized for ultra-low latency." The framework's job is to make a registered function's round trip overlap with generation rather than block it.

“Cresta’s framework supports this composition, so each skill can be built and tested independently”

What does skill mean here?

[Source] The post uses skill more or less interchangeably with specialized sub-agent — the preceding sentence is about "specialized sub-agents for distinct tasks." It also uses "micro-skills → agents" back in the scoping section, describing task decomposition that informs multi-agent architecture.

[Search] Their decentralized post names the actual units: "authentication, policy verification, troubleshooting workflows, payment handling, and more."

[Inference] So a skill here is a bounded task with its own prompt, its own tools, and its own entry and exit conditions — one coherent chunk of the state machine, packaged so it can be exercised in isolation. Worth noting this is not the Anthropic sense of "skill" as a loadable instruction bundle; it's much closer to a microservice with a prompt attached. A good test for whether something deserves to be its own skill is whether you can write a pass/fail condition for it without referring to the rest of the conversation.

“then unified in an orchestrated flow”

By an orchestrator agent?

[Source] The post says "orchestrated flow" and leaves the mechanism unspecified.

[Search] Their architecture post argues explicitly against a central orchestrator agent. It says a central orchestrator "can also create a single point of failure," that errors "accumulate across long conversations," and that centralized designs "introduce extra steps, since every decision and turn reroutes through the orchestrator." Their alternative is "a network of specialized subagents" that "handle their areas of expertise directly and in parallel," coordinated by deterministic state management.

[Inference] So "orchestrated" here means orchestrated by configuration, not by a supervising model. The state machine decides which skill is active and when control transfers, and there's no LLM sitting above the sub-agents making a routing decision every turn. That's a substantive architectural claim and a defensible one for voice, since it removes a model call from the critical path and makes handoffs deterministic and independently testable. The tradeoff is rigidity: anything the state machine didn't anticipate has no path through the graph and falls through to the escalation edge.

“The optimization loop is simple: Tweak the agent config or remote functions, run the test set, and fix whatever shows up in red (bad routes, schema mismatches, guardrail violation etc)”

Overfitting concerns?

[Source] Not addressed. The post presents the loop approvingly and notes that every run is archived for comparison against the last green one.

[Inference] Yes, and I'd call this the sharpest methodological gap in the post. Fixing whatever shows up red and then rerunning the same test set is gradient descent on the test set with a human as the optimizer. Four specific risks follow. You get test-set memorization, where prompt edits patch individual failing scenarios rather than the underlying behavior, which shows up as a pass rate that climbs while production quality doesn't move. Nothing in the description mentions a held-out split the config is never tuned against, which is the minimum defense. With hundreds of stochastic scenarios and repeated runs you get multiple-comparison drift, where some red-to-green flips are noise and chasing them adds prompt text that constrains nothing real. And prompt bloat is the visible symptom of all of it: each patch appends a rule, the prompt grows, instruction-following degrades, earlier rules get quietly ignored, and the fix for failure N breaks passing case M.

What I'd expect in a mature version: a frozen holdout set, freshly generated scenarios each cycle, several seeds per scenario with a significance test on the difference rather than eyeballing red versus green, and a regression gate that reruns previously-passing cases. Their post-launch mining partly addresses this by continuously refreshing the pool, which is the right instinct — but it only works if new scenarios are being added faster than they're being tuned against.

“If it clears the requirements (pass rate, no criticals, tool-accuracy targets)”

1. What is pass rate?
2. What are tool-accuracy targets and why do we care about them?

[Inference] (a) Pass rate is the fraction of simulated scenarios where the agent reached the expected outcome, with the post's "we know 'what should happen'" providing the comparison basis. It's a conversation-level metric, which their companion post distinguishes from turn-level checks. Two things need pinning down before a pass rate means anything across versions: what counts as a pass, meaning exact outcome match versus evaluator-judged acceptability, and how the scenario pool is weighted, since a pass rate computed over a pool that over-samples easy intents isn't comparable to one computed over a harder pool.

(b) Tool accuracy is whether the agent called the right backend function, at the right moment, with correctly-formed arguments. It decomposes into at least four failure modes worth tracking separately: the wrong tool was selected, the right tool was called at the wrong point in the flow, the arguments were malformed or hallucinated — the "schema mismatches" the post names — or no tool was called when one was required.

It earns its own gate for three reasons. It's where the irreversible damage lives, since a clumsy sentence is recoverable and a refund issued to the wrong account is not. It's the one dimension where the agent touches systems of record, so its errors escape the conversation and persist. And unlike conversational quality it's cheaply and objectively checkable, because you know what the correct call was, which makes it a good hard gate rather than a soft score.

“Once an AI agent is live and serving production traffic, we monitor its performance”

Who is “we” here? Cresta or the customer?

[Source] Ambiguous, and the referent shifts within the section. The sentence continues "much like we monitor human agent conversations," and human-agent monitoring is done by the customer's quality team using Cresta's tools. But the following sentences say "we monitor all the metrics we care about tracking," and then describe "making it easy for our customers to align the system," which puts customers in the third person.

[Inference] Most likely both, with the weight on the customer. Everything described — Dashboard Builder, Opera, AI Analyst — is a self-serve product the customer operates, and the entire framing of Opera is that customers define their own behaviors. Cresta presumably also watches closely during deployment and ramp. The slippage in "we" is worth noticing because it conceals an operating-model question: is post-launch tuning a service Cresta performs, or a product the customer runs? The post reads as the latter, sold with the former during onboarding.

“This includes metrics common to AI agents, or specific to certain types of AI agents. For example: Interruptions”

What are interruptions?

[Inference] An interruption is the customer speaking while the agent is still speaking — barge-in. It gets tracked because it's a cheap, automatically detectable proxy for several distinct problems at once: the agent is talking too long, the turn-detection threshold is mis-tuned so the agent starts before the customer has finished, the response was wrong and the customer is cutting in to correct it, or the customer is simply frustrated. A rising interruption rate is an early warning that requires nobody to read a transcript. Its weakness is exactly that ambiguity — it won't tell you which of those four is happening, so it's a trigger for investigation rather than a diagnosis.

“Trends and Anomalies page automatically detects shifts in the distributions of conversation reasons”

Is this mechanistically different from Insights?

[Search] Their Insights post draws the distinction directly. Topic Discovery provides "durable structure: a trusted view of what customer conversations are about," while Real-Time Trends provides "the real-time signal: visibility into what is changing right now, including events no taxonomy was built to catch."

[Inference] So they're layered rather than alternative, and the mechanical difference is the unit of work. Topic Discovery is a batch clustering job that induces a taxonomy: expensive, run periodically, output is a stable label set. Trends and Anomalies is a streaming statistical job over labels that have already been assigned: cheap, run continuously, output is a change signal on distributions you already defined. The phrase about "events no taxonomy was built to catch" implies a second component that isn't just distribution monitoring — emerging-phrase detection over unlabeled text, which catches the thing that doesn't have a bucket yet. That's the piece that genuinely overlaps with clustering, and is presumably a lighter-weight version of it run over a recent window.

“Show them a specific conversation turn where the agent did something questionable”

How do you know in advance if an agent did something questionable without showing it to a human?

[Source] The post doesn't address the selection problem, which is the load-bearing gap in its Lessons Learned section. Their companion evaluator post doesn't either — it covers how to calibrate evaluators, including a preference for binary over numeric scales because "numerical scales are inherently more subjective than binary classifiers," but not how candidates get chosen for review in the first place.

[Inference] This is the classic active-learning problem, and the key realization is that you don't need to know the right answer to select well. You need a cheap signal that correlates with being wrong. The usable ones, roughly in order of yield: evaluator disagreement, where you run two or more judges with different prompts or models and route every disagreement to a human; evaluator uncertainty, where the judge's score sits near its decision boundary or its self-consistency across sampled runs is low; hard rule violations, which are detectable deterministically and always worth surfacing — a promise made, a verification step skipped, a tool called before authentication; outcome signals such as escalations, customer repetition, negative sentiment, or a callback within 24 hours, which need no judgment at all because the conversation tells you it went badly; and novelty, meaning turns far from anything in the labeled set, where the evaluator is least trustworthy.

The honest framing is that you can't know a turn is questionable, only that it's uncertain or anomalous. What that buys you is throughput: it converts "read a thousand transcripts" into "answer fifty yes-or-no questions," which is precisely the fatigue problem the post's lesson is about. The failure mode to watch is that uncertainty sampling systematically misses confident-but-wrong turns, so a small random audit sample has to run alongside it.

“Every clarification from a human should either adjust the agent or its evaluation guidelines going forward”

Adjust in what way? Prompt?

[Source] Verbatim, and the sentence continues "so we don't ask the same question twice." The mechanism is not specified.

[Inference] The sentence names two destinations, and they take different kinds of edit. Adjusting the agent means changing the config bundle, and the post's own framing says a config is much more than a prompt — so the right edit depends on the failure. A wrong tool call is a schema or tool-description fix, not a prompt fix. A missing empathy line is a prompt fix. An illegal path through the conversation is a state-machine fix. A hard compliance rule shouldn't go in the prompt at all; it belongs in deterministic control flow. Adjusting the evaluation guidelines is the other half and it changes the judge rather than the agent: when a human says "no, that response was actually fine," the agent was right and the evaluator was wrong, so the rubric gets amended.

[Search] That second half is what their evaluator post is about — aligning stakeholders on what a behavior like "Agent Assumes Payment" means in a given context, where "each misalignment creates an opportunity to refine the guideline."

[Inference] The requirement hiding inside "so we don't ask the same question twice" is that each clarification has to be stored as a durable artifact with provenance — a rubric clause or a config line — rather than applied once and forgotten. Otherwise the same ambiguity resurfaces with the next reviewer.

“The key is to have a process that rapidly discovers and fixes failures when they occur.”

What part of their pipeline actually provides rapid feedback?

[Source] The post answers this in the very next sentence: "That's why the simulated testing and the post-launch analytics are so vital." No turnaround times appear anywhere in the post.

[Inference] Discovery and fixing have different bottlenecks and it's worth separating them. Discovery is the genuinely fast part, and it's the post-launch layer that delivers it: Trends and Anomalies flags distribution shifts without anyone asking, the Opera behavior evaluators score every conversation rather than a sample, and AI Analyst answers an ad-hoc question in minutes instead of requiring an analyst to pull and read transcripts.

Fixing is where the clock slows. The loop is edit config, run the simulation suite, compare against the last green run, lock the version, push. Simulation is the part that parallelizes, but the post never says how long a full suite takes, and that number is what actually sets cycle time.

[Inference] What's conspicuously absent is anything between "detected in production" and "fixed in the next version" — no hot-fix path, no feature flag, no way to disable one misbehaving skill while the real fix is built. Config versioning and rollback are the closest thing mentioned, but rollback reverts everything rather than the one broken flow. For a voice agent carrying live traffic, that gap matters more than the length of a test run.

“We learned to prioritize requirements by criticality.”

How does this prioritization actually manifest?

[Inference] It should show up in four concrete places. In the release gate, where the criteria quoted in Q29 are "pass rate, no criticals, tool-accuracy targets" — "no criticals" is a hard blocker while pass rate is a tunable threshold, so criticality manifests as a different kind of gate, not a heavier weight. In where the behavior is enforced, since critical requirements shouldn't live in prompt text at all: you cannot reach the account-change state without passing through verification, and that's a graph property rather than an instruction. In evaluation scope, where their companion post's turn-level checks run on every conversation for critical behaviors while style preferences get sampled conversation-level scoring. And in the agent's own trade-off instructions, which is what the post means by guiding trade-offs — telling it explicitly that if brevity conflicts with a required disclosure, the disclosure wins.

[Inference] The open question the post leaves is who assigns criticality and whether the scale is graded or binary. Binary is usually enough and avoids a long argument about tiers.

In general, how would you design this system to ensure that callers aren’t able to use the generic LLM capabilities and are limited to its customer support role? What guardrails?

[Inference] Layer them, because no single one holds, and note which layers are constraints versus requests.

The strongest guardrail is architectural, and their design already provides it. If the agent is a set of narrow skills gated by a deterministic state machine (Q27), there is no state in the graph whose job is "answer general questions," so an off-role request has nowhere to go. A monolithic prompt-driven agent has no such structural limit.

The second is capability limitation: the agent can only do what its registered functions do. If no function issues a refund above a threshold, no amount of persuasion produces one. This is the layer that cannot be talked around.

The third is a small, fast input classifier ahead of generation, labelling each incoming turn in-scope, out-of-scope, or adversarial and routing the last two to a fixed response rather than to the model. The fourth is output filtering — the concurrent guardrail model call their latency post describes, running in parallel with the main call so it costs no response time, checking for off-topic content, unauthorized commitments, and data leakage. Compliance disclosures and verification prompts should be templated strings rather than generated text, since there is then nothing to manipulate.

[Inference] The framing that matters: an instruction in a prompt is a request, not a constraint. Anything you genuinely cannot allow should be enforced by what the system is able to do.

How would you improve this system if you were building it today?

[Inference] Five changes, ordered by how much they would move real quality.

First, fix the evaluation loop, because everything else is measured through it. Freeze a holdout set the config is never tuned against, generate fresh scenarios each cycle, and run several seeds per scenario with a significance test rather than eyeballing red versus green. Without this the pass rate is a vanity metric (Q28).

Second, decompose the simulated visitor. Store persona, world state, goal, and expected outcome as four independent artifacts and sample the cross-product instead of hand-building bundled objects (Q17). This is what makes coverage measurable — you can then report which state-machine transitions the scenario pool has never exercised.

Third, build the human-review sampler the post assumes but never describes (Q33): disagreement between two judges, plus uncertainty, plus a small random audit sample to catch the confident-but-wrong turns that uncertainty sampling misses.

Fourth, split automation readiness into two scores (Q10) — build feasibility and automation risk — and let the risk score act as a veto rather than averaging into a single number.

Fifth, add a surgical rollback path. Being able to disable one skill behind a flag, rather than reverting an entire config version, is what turns "rapidly fixes failures" (Q35) from an aspiration into an operational property.

[Inference] The one thing I would keep unchanged is the decentralized state-machine architecture. It is the decision that makes every other part of the system testable.

“compliant, trusted AI agent”

What are examples of standard compliance requirements for agents?

[INFERENCE] The contact center requirements that actually get encoded as evaluator criteria, grouped by the regime that imposes them:

Debt collection. The FDCPA mini-Miranda ("this is an attempt to collect a debt"), restrictions on contact hours, and no disclosure of the debt to third parties. Regulation F adds call-frequency limits.

Payment handling. PCI DSS forbids storing or logging full card numbers and security codes, which in practice means pausing recording and transcription during card capture.

Health. HIPAA's minimum-necessary rule, plus identity verification before any protected health information is discussed.

Contact and recording. TCPA consent for automated outbound calls, and two-party recording consent in states that require it.

AI-specific. Disclosure that the customer is speaking to a bot, required under California's bot disclosure law and the EU AI Act's transparency provisions, and a working path to reach a human.

The first four predate agents. Only the last is new.

“Enterprises that equate high evaluation scores on OOTB metrics”

What are standard OOTB evaluator metrics?

[INFERENCE] What ships in the common frameworks — RAGAS, DeepEval, Arize Phoenix, LangSmith, and OpenAI Evals — sorted by what each actually measures:

Grounding. Faithfulness, or whether the answer is supported by retrieved context, plus context precision and recall. These are the RAGAS core and the most mature of the group.

Task correctness. Answer similarity against a reference, and task completion.

Safety. Toxicity, PII leakage, bias, and jailbreak or prompt-injection detection.

Conversational quality. Coherence, helpfulness, empathy, tone, verbosity. This is the subjective cluster and the one the post's argument targets.

Agent-specific. Tool-call correctness and trajectory adherence — the newest category and the least standardized.

The post's critique lands hardest on the fourth group, where the construct itself is contested. It applies much less to grounding and tool-call checks, which have something closer to a ground truth.

“Consider the behavior, “Agent Assumes Payment.””

What are other examples of “behavior[s]”?

[INFERENCE] A behavior here is a named, observable conversational act specific enough that a yes-or-no answer is possible. That constraint follows from the post's second principle, and it's what separates a behavior from a quality score. Examples by where they occur:

Opening. Agent verifies identity before discussing the account. Agent states the company and purpose of the call.

Diagnosis. Agent asks a clarifying question before proposing a solution. Agent acknowledges a stated frustration before moving on.

Resolution. Agent commits to a specific next step and timeline. Agent quotes policy without checking the account. Agent over-promises a resolution date.

Compliance. Agent reads the required disclosure before collecting payment details.

Closing. Agent confirms the issue is resolved. Agent offers an escalation path when it isn't.

These read like contact center QA scorecard items, which is almost certainly where the taxonomy comes from — Cresta's origin is human agent quality management.

“ask different stakeholders to label each as a positive or negative example of the behavior”

Are they labeling whether the agent exhibited the behavior or whether it’s good or bad that it did?

[SOURCE] The post is genuinely ambiguous, and you can watch it slide between the two readings. "Positive or negative example of the behavior" is classification language — did this instance instantiate the label. But a few sentences later: "whether it's 'good' or 'bad' depends entirely on the context." That's valence, a different question.

[INFERENCE] The coherent reading is that the labeling task is detection, and the collections-versus-sales contrast is diagnostic of a problem rather than an illustration of the method. In both scenarios the agent performs the same act. Stakeholders resist labeling the sales utterance as "Agent Assumes Payment" because it doesn't feel wrong — their valence judgment is contaminating their detection judgment.

That contamination is worth naming, because it means the disagreement the post attributes to unclear guidelines is partly a task-design flaw. The fix is to split the label into two fields: did the behavior occur, and is it desirable in this context. The first is a classifier; the second is policy. Collapsing them guarantees the inter-annotator disagreement the post then sets out to fix.

“Even among business experts familiar with the space, judgments often differ.”

How to handle disagreements?

[INFERENCE] Four steps, in order:

Measure before resolving. Compute inter-annotator agreement — Cohen's kappa for two annotators, Fleiss' kappa or Krippendorff's alpha for more. The post mentions kappa later for its own evaluators. The number tells you whether the guideline is ambiguous or a particular annotator is an outlier, which have different fixes.

Adjudicate in discussion, not by majority vote. Majority vote produces a label while hiding the ambiguity that caused the split, so the guideline never improves.

Write the resolution down as a rule plus a worked example. The examples transfer better than the abstract rule, and they become the judge's few-shot set later.

Re-annotate a fresh sample to confirm agreement actually rose.

Accept a floor. Some disagreement is irreducible, and there you decide by policy — default to the conservative reading on compliance items — rather than pretending consensus exists.

How to know which/how many examples need multiple reviewers vs single expert?

[SOURCE] Not directly answered, but the post supplies the governing distinction: multiple-annotator alignment is "a key requirement" for models "designed to work across use cases and industries," while aligning to a single expert produces a preference-aligned model that is "easier to define" but has "a more specific scope."

[INFERENCE] That settles the model-level question — cross-customer evaluators need multiple annotators, single-organization evaluators don't. The volume question is separate and the post is silent on it.

You don't multiply-label everything. The normal approach is to double- or triple-label a random subset, often ten to twenty percent, purely to estimate agreement, and single-label the remainder once agreement is acceptable. The subset is a measurement instrument, not the training data.

Beyond that random sample, route specific items to multiple reviewers: those near the decision boundary, those where the first annotator flagged low confidence, and anything in a compliance-critical category where a wrong label is expensive. The logic is stratified sampling — spend extra review where variance is high or the cost of error is high, and accept one pass everywhere else.

I’m not fully convinced that going from a small number of numerical scores to a large number of booleans would solve the inconsistency problem. It feels like we’d have the same amount of room for variation in either case unless we narrow the criteria.

[INFERENCE] Your objection is largely right, and the phrase the post quotes gives it away: the work is being done by precise, not by boolean. Decomposing a ten-point quality score into twenty vague booleans — "was the agent professional" — multiplies the ambiguity rather than removing it. Narrowing each criterion to a concrete checkable fact is the actual mechanism, and that's a change in the construct, not in the response format.

Binary does contribute something real and separable, though. A ten-point scale requires agreement on nine decision boundaries, which the post notes ("one for each decision boundary"). Even with a perfectly clear construct, annotators who agree an example is "good" will still split between seven and eight. Binarizing eliminates that class of noise entirely.

So there are two effects and the post conflates them. Narrowing the criteria is the large one; removing boundary ambiguity is the smaller but genuine one. Neither works without the other.

For Precise Boolean rubrics, what happens if certain criteria aren’t applicable? Like if someone asks an agent for advice, I don’t think grading its factuality would make sense.

[SOURCE] Not addressed. The post presents rubrics as uniformly applied and never mentions inapplicable criteria.

[INFERENCE] You need a third value, and conflating "not applicable" with "no" is a real and common defect. It depresses scores on conversations where the criterion never arose, which makes your aggregate metric a function of conversation mix rather than agent quality — two agents of identical quality score differently because they handled different intents.

Two ways to handle it, which differ in where the logic lives:

Gating. An applicability predicate decides whether the criterion is scored at all. "Was a factual claim made?" gates the factuality check. This is cleaner, because applicability is usually a much easier boolean than the criterion itself.

Three-valued labels. Score yes, no, or N/A, and aggregate as passed over applicable rather than passed over total.

On your advice example specifically, the sharper move is usually to narrow rather than exclude. Advice still rests on checkable premises — "your plan covers this procedure" — so scope factuality to verifiable claims instead of marking the whole conversation inapplicable.

“The only sustainable way to achieve scalable, fully-aligned LLM judges is through continuous human calibration.”

What is the continuous aspect of this?

[INFERENCE] Continuous because three things drift underneath a judge that was calibrated once:

The judge model changes. A provider version upgrade shifts judge behavior, and your agreement numbers were measured against the previous version.

The agent changes. As it improves, the conversations reaching the judge shift toward rarer and harder cases — precisely the region where it was never calibrated.

The standard changes. New products, regulations, and policies redefine what counts as good.

Operationally that means a standing sample of judge outputs re-labeled by humans on a fixed cadence, agreement tracked as a monitored metric with alerting when it falls, mandatory re-validation before any judge model change ships, and newly discovered edge cases folded back into the guideline.

The framing worth carrying: inter-annotator agreement is a production metric, not a one-time acceptance test.

“By injecting human preferences”

What does this mean?

[INFERENCE] Given everything else the post says, this almost certainly means prompt-level calibration: human labels and the clarified written criteria they produce go into the judge's rubric and its few-shot examples. Cresta emphasizes that guidelines are customer-visible and customer-adjustable, which only makes sense if the guideline is text in a prompt rather than weights in a model.

Worth flagging the terminology, because it's borrowed. "Human preferences" in machine learning normally means pairwise comparison data used for RLHF or DPO — you collect A-versus-B judgments, fit a reward model, and optimize against it. Nothing in the post mentions training, reward models, or preference pairs.

The two have very different cost and reproducibility profiles, so the loose usage matters. Rubric calibration is cheap, inspectable, and editable. Preference optimization is none of those. If this comes up with them, it's worth asking which they actually mean.

“Cresta’s guidelines are exposed to customers, so you can understand exactly what each LLM judge is doing, and even adjust the criteria to better fit your organizations’ standards”

In what ways could it make sense to adjust the criteria?

[INFERENCE] Adjustments that legitimately make sense, by what's being changed:

Valence. Keep the detection rule, flip whether the behavior is desirable. This is the collections-versus-sales case exactly.

Strictness. Where the threshold sits — how much hedging still counts as assuming payment.

Applicability. Which conversation types the criterion gates on.

Required elements. Adding a jurisdiction-specific disclosure to a compliant-closing criterion.

Vocabulary. Mapping criteria onto your existing QA scorecard names so results reconcile with the quality program you already run.

One risk the post doesn't raise: adjustable criteria break comparability. Once two customers edit the same evaluator differently, Cresta's "proven to generalize" claim no longer covers either edited version, and cross-customer benchmarks stop meaning anything. Any adjustment should trigger re-validation against human labels, or you've silently decalibrated a judge whose credibility rested on the original guideline.

“OOTB evaluators make big promises about objectivity and scale, but in practice, they deliver metrics detached from meaning.”

This seems plausible but what would be a rigorous quantitative way to measure this?

[INFERENCE] Three tests, distinguished by what each compares against:

Agreement with domain experts. Run the evaluator and your experts on the same conversations and compute kappa or Krippendorff's alpha. Benchmark against human-to-human agreement on that same set, not against one hundred percent — the human ceiling is the only fair target.

Predictive validity. Does the evaluator's score correlate with outcomes that already matter — repeat contact rate, escalation rate, CSAT, compliance audit failures? This is the most direct operationalization of "detached from meaning," and the strongest of the three, because it needs no labels and no one's opinion.

Domain sensitivity. Take matched conversation pairs where experts judge the same behavior differently by context — the collections and sales examples. If the evaluator assigns both the same score, it is demonstrably blind to the thing that determines correctness.

Run all three. The second is where a generic evaluator most often fails without anyone noticing.

“where our iteration velocity started to plummet”

What aspect of their testing process was especially slow and would cause this?

[SOURCE] Partly answered. The anti-pattern section names it: they "over-indexed on brittle, turn-specific assertions and fine-tuning before we had enough test volume to see impact × frequency." The trigger was launching complex billing and survey use cases.

[INFERENCE] The mechanism the post leaves implicit is that the expensive step is human, not compute. A turn-specific assertion hard-codes what the agent should say or do at a particular moment, so every prompt or flow change invalidates a batch of them and someone has to rewrite the expectations by hand. Maintenance cost scales with turns per flow times tests per turn, and billing and survey flows are long — which is exactly why those launches were the breaking point.

Two further costs compound it. Brittle assertions produce false failures on valid rewordings, and each one consumes triage time before you learn nothing was wrong. And because the assertions were written before they knew which behaviors were frequent or consequential, much of that maintenance was spent on behaviors that didn't matter.

So the slow part was rewriting and triaging tests between iterations, not running them.

“able to run 15x more tests with a far greater degree of diversity, leading to 35% faster release cycles”

Why does 15x more tests correspond to only a 35% faster release cycle?

[INFERENCE] There's no reason to expect proportionality, because the two numbers measure different things. 15x is test volume — an input. 35% is cycle time — an output with many other components.

The ceiling is structural. If testing occupies half your release cycle, eliminating testing time entirely caps you at 50% faster. A 35% improvement is consistent with testing having been a large but not dominant share of cycle time. Design, implementation, review, and customer acceptance don't speed up because you ran more tests.

More importantly, the 15x isn't what produced the speedup. Running more tests is, on its own, slower. The speedup came from removing the manual maintenance bottleneck in Q1; the extra volume is what that removal made affordable, and it's what bought the quality gain instead.

Worth flagging: this is a vendor figure with no stated baseline or comparison method. Treat the direction as the claim and the magnitudes as unverified.

“improving accuracy by 20%”

What metric is this?

[SOURCE] Never defined. No metric name, baseline, denominator, or measurement window appears anywhere in the post. This is its least substantiated claim.

[INFERENCE] In this domain "accuracy" could plausibly mean any of several things, and they aren't interchangeable: goal or task completion rate (did the customer's issue get resolved), containment rate (handled without human escalation), per-turn correctness against a golden response, or judge-scored flow adherence. A 20% move means something quite different in each.

The ambiguity that matters most is absolute versus relative. Twenty percentage points (70% → 90%) and a 20% relative gain (70% → 84%) are very different results, and the phrasing supports either.

There's also no stated denominator. Accuracy measured over which conversations — all traffic, the complex billing and survey flows that triggered the rework, or the test suite itself? Improvement measured on your own expanded test suite is partly circular, since you changed what you were measuring at the same time.

Without those, the number isn't interpretable. It's worth asking about directly if this comes up in conversation with them.

“Because this can be automated”

Can’t everything they do be automated? How is dynamic testing unique in this regard?

[INFERENCE] Your challenge is correct as stated — static tests also run automatically once they exist. The post is being loose, and the real asymmetry is in a different place: test case generation, not test execution.

Splitting the pipeline cleanly:

Generation. Dynamic testing produces new conversations from a goal or persona specification, so coverage scales with compute. Static testing requires a human to notice a case, capture it, and define expected behavior, so coverage scales with headcount. This is the only stage where the two genuinely differ.

Execution. Both are fully automated. No distinction.

Grading. Both use the same evaluators — deterministic checks and LLM judges. No distinction.

Calibration and review. Human, for both.

So the defensible version of the claim is that dynamic testing is the only method whose coverage grows without proportional human effort, which is why it can serve as the volume driver. The post compressed that into "can be automated," which overstates the contrast.

“from early design exploration through post-launch monitoring though post launch”

What’s the difference between post-launch monitoring and post-launch?

[INFERENCE] So the substantive claim is about cadence, not about two concepts: run dynamic testing across the entire lifecycle, but expect its intensity to drop after launch and spike again around major updates.

The distinction actually worth drawing — which the post does make, later, in its "Bonus Points" section — is between offline testing and real-world monitoring. Offline testing runs simulated or replayed conversations against known expectations. Real-world monitoring evaluates live production conversations where there is no ground truth. The post says the better your production monitoring gets, the less you need high-volume offline testing.

That's the meaningful post-launch split, and it's a different axis from the one the garbled sentence appears to be drawing.

“Running a large number of simulated customer conversations”

How to determine right number?

[INFERENCE] It's a statistical power question, not a fixed number, and it decomposes into two multiplied quantities: runs per scenario, and number of scenarios.

Runs per scenario. You're estimating a pass rate, so precision scales as one over the square root of the run count. Distinguishing a 90% pass rate from a 95% one with confidence takes on the order of several hundred runs per scenario; confirming a flow is roughly working takes tens. Set this from the smallest difference you actually need to detect. Combined with the prior post's pass^k framing, you also need enough reruns to measure reliability rather than capability.

Number of scenarios. Driven by how many distinct intents and flows you support, weighted by the impact × frequency criterion the post uses elsewhere.

A practical stopping rule beats a formula here: keep adding runs until new runs stop surfacing new failure modes and stop moving your pass-rate estimates. That saturation point is observable, and it's where marginal cost exceeds marginal information.

“replay them against updated versions of the agent to confirm nothing regresses”

Is there any reason to expect regressions? What circumstances make it more likely?

[INFERENCE] Yes, and more than in conventional software, because LLM agents lack module boundaries. A prompt is one global object — editing an instruction for the refund flow changes behavior on the billing flow too, since both are conditioned on the whole prompt. There is no compiler or type system to contain the blast radius, and no way to know what you affected except by testing it.

Other sources, roughly in order of how much they matter: model version upgrades, which change behavior everywhere at once; context pressure, where adding instructions degrades adherence to existing ones; tool or schema changes; and retrieval index updates that change what the agent sees.

Circumstances raising the likelihood: large prompt edits, edits to shared or global instructions rather than flow-specific ones, a model swap, and any flow that was only marginally passing before — those flip on small perturbations.

One caution: a single failed replay may be resampling noise rather than a real regression. That's what pass^k is for.

“Use it selectively early on, only for issues caught during dynamic testing or key “golden” flows.”

Why?

[SOURCE] The post supplies the reasoning across two sections. Turn-level assertions are "brittle when used broadly," and the anti-pattern was building them "before we had enough test volume to see impact × frequency."

[INFERENCE] Two distinct reasons, and it's worth separating them.

Shelf life. Early on the agent's design is still moving. A snapshot freezes behavior at a moment in context, so snapshots written during active design encode behavior that's about to be deliberately redesigned. You pay maintenance cost on tests with short useful lives, and worse, they fail for the right reason — you changed something on purpose — which trains the team to ignore failures.

Selection. You don't yet know which behaviors are worth locking down. "Impact × frequency" is the selection criterion, and neither term is observable until you have volume. Dynamic testing is the discovery mechanism that produces that information.

So the sequencing is deliberate: explore broadly with dynamic testing to find out what matters and what breaks, then spend the expensive per-case curation only on what you've learned is consequential and stable.

“Continue building on test cases as a regression set”

How to build?

[SOURCE] Partly answered. The Bonus Points section supplies the intake source: effective real-world monitoring "allows you to discover additional edge cases and errors to add to your existing static test sets."

[INFERENCE] The pipeline that implies: production monitoring flags a suspected failure, a human confirms it's a genuine defect rather than a judge error, the conversation state at the failing turn is captured, expected behavior is defined, and it enters the suite as a snapshot.

Four things the post omits that determine whether the suite stays useful:

Deduplicate on intake, or one recurring bug fills the suite with near-identical cases and skews what you're measuring.

Tag by flow and severity, so you can run compliance-critical subsets on every change and the full suite less often.

Prune deliberately. When behavior is intentionally redesigned, the corresponding snapshots must be retired or updated. Skip this and the suite accumulates false failures until people stop trusting it — the failure mode that kills regression suites.

Record why each case exists, since a snapshot with no rationale can't be safely retired later.

“This can also include adherence to an overall flow.”

I’m assuming this is being evaluated by an LLM. If so, couldn’t we just put the desired flow in the agent instructions as well? In which case, I’d assume it’s very unlikely for it to deviate from that trajectory.

[INFERENCE] The premise — instructed therefore compliant — is where this breaks, for four reasons.

Instruction adherence degrades with load. A flow spec competes with every other instruction in the prompt, and adherence falls as instruction count and context length grow.

Customers force deviation. Side quests and mid-flow topic changes mean the ideal trajectory often can't be walked straight, and the spec underspecifies how to leave and re-enter it.

Instructions conflict. "Be responsive and helpful" pulls against "complete these steps in order," and the resolution isn't deterministic.

Non-determinism. Even a well-followed flow fails some fraction of runs.

There's also a design reason to keep them separate: the instruction states intent, the evaluator measures realized behavior. An agent that can't follow the flow is equally unable to report reliably that it did, so the check has to be external.

“This approach is resilient to prompt changes”

What does this mean?

[INFERENCE] It means the evaluation criterion doesn't depend on the agent's specific wording. A conversation-level check asks whether the refund was issued and the steps happened in the right order. Reword the prompt, and the agent phrases things differently but the answer to that question is unchanged. A turn-level check asserting specific confirmation wording fails the moment the wording changes, even though nothing is actually wrong.

So "resilient" means a low false-failure rate under changes that don't affect correctness — which converts directly into lower test maintenance cost, the bottleneck from Q1.

The trade-off is sensitivity, and the post is right to keep both scopes rather than picking one. A conversation-level check won't catch a missing disclosure at turn four if the outcome still succeeded. Resilience and precision trade against each other here, which is why compliance work stays at turn level despite the maintenance cost.

“takes the right actions at specific steps of an interaction”

How is this different from the static/snapshot testing?

[INFERENCE] So they compose into a two-by-two, and the reason they feel like the same thing is that one cell dominates in practice.

Static plus turn-level is the compliance combination the post recommends — replay a recorded state, assert the disclosure appeared. This pairing is common enough that the two concepts get conflated.

Dynamic plus turn-level is real and useful: run simulated conversations, then check whether the confirmation followed payment capture in each. You get turn-level precision across generated rather than curated inputs.

Static plus conversation-level is also real: replay a recorded conversation and judge whether the whole flow succeeded.

The clean statement: static versus dynamic determines where the customer turns come from. Turn versus conversation determines what you assert on. The first is about inputs, the second about measurement.

“now real world monitoring becomes critical”

What would that consist of?

[INFERENCE] The structural difference the post doesn't name: offline you have ground truth, in production you don't. Everything below follows from that.

Sampling. Judging every conversation with an LLM is cost-prohibitive at production volume. Stratified sampling weighted toward high-risk flows, with full coverage only where checks are cheap and deterministic.

Operational signals that need no judge. Containment rate, escalation-to-human rate, conversation length, repeat-contact rate within 24 hours, and where customers abandon. These are cheap, continuous, and often detect a regression before any judge does.

Drift detection. Alert on movement in those rates rather than on absolute thresholds, since you have no correct answer to compare against.

Feedback loop. Route confirmed failures into the static suite through the Q9 pipeline.

The judge scores tell you about quality; the operational signals tell you fastest that something broke.

“Large Language Models (LLMs) introduce non-determinism”

What are the different sources of non-determinism in LLMs?

Two groups.

Deliberate sampling — temperature, top-p, top-k, seed. This you asked for and can switch off.

Non-determinism that survives greedy decoding. Floating-point addition isn't associative, and GPU kernels reduce in whatever order is fastest. That order depends on batch shape, and inference servers batch concurrent requests together, so your output can depend on who else hit the endpoint at the same moment. Mixture-of-experts routing, prefix caching, and kernel version drift add to it. Once any of this flips a near-tied argmax, the divergence compounds through the rest of the generation.

For agents, the model isn't the only stochastic part. Tool responses, retrieval indices, and account state all drift. A perfectly deterministic model would still give non-reproducible agent runs.

“meaning the same input can generate an

infinite number of responses”

Is this true even with nucleus sampling?

False as stated, and nucleus sampling makes it more false, not less.

The output space is finite in every case: a model with vocabulary V and maximum output length L can emit at most V^L sequences. For a 100,000-token vocabulary over a few thousand tokens that is astronomically large, but it is a number. Nothing about sampling makes it infinite.

Nucleus sampling then shrinks it. Top-p truncates to the smallest set of tokens whose cumulative probability reaches p and renormalizes, assigning exactly zero to everything in the tail. So the set of reachable outputs is strictly smaller than under unrestricted sampling. Top-k does the same more bluntly.

Your instinct points the right way — top-p is a counterexample to the framing.

The post's argument doesn't depend on the word, though. It needs only "too many valid outputs to pre-script," which holds under aggressive top-p. The imprecision is rhetorical rather than load-bearing, but worth catching: genuine infinitude would imply things about coverage that a large finite space does not.

“In testing, this can create the illusion of success. For example, an AI agent might tell a customer their account issue is resolved simply because the customer insists it is, even though the agent never confirmed the fix in the system. In a goal-driven simulation, this may appear as a 'pass' even though the task was never actually completed.”

Why would this appear as a pass? Shouldn't the evaluator also have access to the system to check if the issue was resolved?

[INFERENCE] Your objection is right, and the post concedes it two sections later: its own remedy is deterministic evaluation using "logic checks like verifying successful API calls, accurate data retrieval." That is exactly the system access you're describing.

So this isn't inherent to goal-driven evaluation. It's inherent to transcript-only evaluation, and it arises when any of three things hold: the harness has no real backend and tools return canned success; the evaluator is an LLM reading the conversation rather than querying state; or "goal met" has been operationalized as the agent's own closing assertion, making the agent its own judge.

[INFERENCE] The principle: never let the evaluated system supply the evidence for its own evaluation. Assert on backend side effects, not transcript claims. Where a real backend is impractical, give the mock inspectable state so you can still assert on effects.

The post would read better framing this as a design error it has since fixed.

“For example, if an AI agent fails to help a customer reset an alarm, it can be unclear whether it forgot to ask them to reset their WiFi or failed to prompt them to check the battery. And when several breakdowns occur at once, it becomes difficult to understand the full scope of what went wrong. You know it failed, but not where or why.”

In this example, an effective AI agent would have conducted request-specific tasks. Since the set of request-specific tasks would naturally vary for different requests, how can you construct a comprehensive and granular evaluation suite that works for a diverse set of requests?

[INFERENCE] Separate what generalizes from what doesn't, so authoring cost scales with intents rather than conversations.

Generic evaluators, written once. Hallucination, tone, compliance disclosure, whether the agent verified before acting, whether it looped. These apply regardless of what the customer wanted.

Per-intent checkpoint rubrics. For each intent, define the checkpoints correct handling must hit. The alarm example becomes a checklist: check power, check battery, check WiFi, confirm the reset took. Store these as data on the test case, not as evaluator code, so adding an intent is a data change.

Derive rubrics from production. Cluster historical conversations by intent and extract the action sequences characterizing successful resolutions. This is the only part that scales to a long tail — and where Cresta's combined human-agent and AI-agent corpus would actually pay off.

Granularity then falls out: a failure reports which checkpoint was missed.

“Even with sophisticated modeling, simulated testing environments often fail to fully represent real-world customer behavior. LLMs—and even human testers—don't always act like real customers, making it difficult to ensure that test scenarios mirror the conversations agents face in production.”

What does this have to do with goal-driven evaluation specifically?

[INFERENCE] It doesn't belong there, and flagging it is correct. This limits simulation as a testing method, not goal-driven evaluation as an evaluation philosophy. The post itself makes the distinction a paragraph earlier, writing that "simulation became a core testing method" in order to enable goal-driven evaluation — a dependency, not an identity.

The post's own taxonomy separates testing methods (static replay, snapshot, simulated customer) from evaluation approaches (deterministic checks, LLM judges, human review). Goal-driven evaluation is an evaluation approach; simulator realism is a property of one testing method. Run goal-driven evaluation on replayed real transcripts and the realism problem disappears, because the customer turns are real.

[INFERENCE] The defensible version: goal-driven evaluation is most valuable when exploring counterfactual agent behavior, which needs a live counterparty, which in practice means a simulator — so in its highest-value application it inherits simulator bias. Real constraint. Flattening it into "limitation of goal-driven evaluation" hides that the fix lives in the simulator.

“verify behavior remains stable across agent updates”

What behavior is being looked at here and how is it assessed?

[SOURCE] The post specifies the input (real historical or test conversations, held fixed) and the purpose (stability across updates), but names neither the behaviors observed nor the comparison mechanism.

[INFERENCE] Holding the conversation fixed is what makes this tractable: replay the customer turns as a constant and regenerate the agent's side, so any difference is attributable to the agent change rather than to conversational drift.

Observable behaviors, in descending order of how reliably you can assert on them: tool calls (which tool, what arguments, what order), knowledge retrieval (which article or record), policy actions (disclosure read, identity verified), and finally free-text response content.

Assessment has to be mixed, which the post leaves implicit. Structured behaviors admit exact comparison — tool name, argument values, and article ID either match the prior version or they don't. Free text cannot, because exact match would flag every valid rewording as a regression, the precise failure the post's opening section criticizes. So free text needs an LLM judge or a semantic similarity threshold against the prior response.

“Captures a moment in context (e.g., when

the AI agent gave an incorrect or best-practice response) for regression testing”

Why do they call it regression testing? What value does this add compared to the full evaluation suite?

[INFERENCE] The term comes from software engineering, where a regression is a previously-working behavior breaking again. The implied workflow: a bad response is observed in production, fixed through a prompt or flow change, and that exact moment is frozen as a test so the fix can't silently revert when something unrelated changes.

Value over the full suite comes in three parts.

Cost and latency. A snapshot is one turn with fixed context; a simulated conversation is many turns with a second model in the loop. You can run thousands of snapshots per prompt change. You cannot do that with simulations.

Diagnostic precision. A snapshot failure localizes to a known behavior. A simulation failure says the conversation went wrong somewhere — the exact complaint the post raises about diagnostic value.

Institutional memory. The suite accumulates a record of every mistake the system has made.

The limitation is symmetric: snapshots only cover failures you've already seen. Hence the pairing with simulation.

“Uses an LLM-based customer that follows a goal or instruction (e.g., booking a flight) under varied conditions, exposing real-world unpredictability and helping identify how the agent handles edge cases”

How do you actually get the simulation to explore more diverse customer trajectories?

[INFERENCE] The levers, weakest to strongest:

Sampling parameters. Raising temperature on the user model is the obvious move and the weakest — it produces paraphrases, not new trajectories.

Persona and goal sampling. Sample over personas (terse, confused, hostile, changes their mind mid-conversation), goals, and hidden constraints the agent must surface by asking. This is the first real gain, because trajectory diversity comes from intent structure, not word choice.

Seeding from production. Draw opening turns, grievance types, and account states from real transcripts, so you test against the distribution you'll actually see.

Environment perturbation. Inject tool failures, timeouts, missing records, stale data, ambiguous account states. Often the highest-yield source, because it forces recovery paths no user-side variation reaches.

Adversarial or curriculum search. Reward the simulator for finding failures and resample around them. The only approach that systematically finds what you didn't think to look for.

The last two account for most of the real coverage.

“Uses logic checks like verifying successful API calls”

How do you verify successful API calls?

[INFERENCE] This collapses three different assertions, and they catch different failures.

Call level — was the right call made? Assert on the structured tool-call record the agent framework emits: tool name, argument values, ordering, and absence of calls that shouldn't have happened. Assert against the framework's record, not the prose transcript, because an agent can narrate a call it never made.

Response level — did it succeed? Status code, absence of an error payload, and schema validation of the body. Schema validation matters more than it sounds: a 200 with a malformed or empty body is a common silent failure.

Effect level — did the world change? Query the sandbox backend afterward and assert on state: the ticket exists with the right fields, the address is updated. This is the layer that catches the Q3 sycophancy case, because it trusts neither the agent's claim nor the API's response.

Most teams implement the first two. The third needs a sandboxed backend with inspectable, resettable state — real engineering investment, not a framework feature.

“Expert-aligned LLM judge evaluation: Employs calibrated LLM evaluators to assess nuanced behaviors, such as flow adherence, response relevance, or factuality”

Calibrated how exactly?

[INFERENCE] What calibration means in practice:

Build the reference set. Multiple experts independently score the same conversations, then disagreements resolve into a consensus label. Multiplicity matters — one annotator gives you no way to know what agreement is achievable.

Measure agreement. Cohen's kappa or Krippendorff's alpha for categorical judgments; correlation or mean absolute error for scalars. Raw accuracy misleads on skewed label distributions, which compliance checks usually have.

Iterate. Adjust rubric wording, few-shot anchor examples, and thresholds until agreement clears the bar. Anchors do most of the work.

Re-validate on change. Judge behavior shifts with the underlying model version, so this is a standing obligation.

“Manual human review: Involves experts validating edge cases, refining evaluator calibration, and ensuring that LLM-based assessments align with enterprise quality standards”

What is an edge case in this context?

[INFERENCE] Two meanings are in circulation and the post blurs them.

Edge case as rare input — unusual account states, rare intents, unusual customer behavior. The conventional software meaning.

Edge case as contested judgment — ambiguous intent, conflicting policies, conversations where the agent was technically correct but the experience was poor, or cases where an LLM judge and a deterministic check disagree.

The second is what the quoted sentence needs, because it concerns where to spend expert human attention. The operationally useful definition: a conversation where your automated evaluators are unreliable — low judge confidence, evaluator disagreement, or a score near a decision threshold. Those are the cases where human review changes an outcome. Rare-but-unambiguous inputs don't need a human; the evaluator handles them.

This definition also makes human review schedulable rather than open-ended. You route the low-confidence tail to experts, and the volume becomes something you can budget.

How do you determine whether it’s “align[ed] with enterprise quality standards”? What metric do you measure and what threshold do you need to meet?

[INFERENCE] Metric. The same machinery as Q10: agreement between the automated evaluator and expert consensus on a labeled sample. But choose it per evaluator rather than globally, because error costs are asymmetric and differ by type. For a compliance check, recall dominates — a missed disclosure violation is regulatory exposure, a false alarm costs minutes of review. For a subjective quality score, correlation or kappa against consensus. For a factuality judge, precision and recall reported separately rather than blended into an F1 that hides which is failing.

[INFERENCE] Threshold. No universal number, and claiming one would be the wrong interview answer. The principled bar is judge-to-human agreement approaching human-to-human agreement on the same data. If two experts agree 80% of the time, demanding 95% from the judge is incoherent — you'd be asking it to outperform its own ground truth. So measure inter-annotator agreement first and treat it as the ceiling.

For high-stakes evaluators, the usual pattern is a recall floor with accepted lower precision, plus routing everything flagged to human review.

“By combining these layers”

How do you combine turn-based and conversation-based layers? Single score (potentially weighed), multiple scores or what?

[INFERENCE] A single weighted score is the intuitive answer and the wrong one, for two reasons. It lets a strong conversation-level outcome mask a turn-level compliance violation, which is backwards for a regulated contact center. And the weights are unjustifiable — there's no principled exchange rate between "read the disclosure" and "resolved the issue."

What works:

Gating. Hard binary gates on compliance and correctness: any violation fails the run outright, regardless of outcome quality. Graded quality scores are computed only among runs that clear the gates. Some failures are not compensable, and this encodes that.

Scorecard, not scalar. Report per-evaluator pass rates as a vector. Any aggregation happens at the reporting layer and never destroys the components.

Structured roll-up. Turn-level results aggregate through explicit operators — any violation, all required steps present, count of loops — chosen per evaluator. This preserves the drill-down path the post's own diagnostic complaint demands.

“must be assessed for both correctness and quality”

Difference between correctness and quality?

[INFERENCE] Correctness is whether the right thing happened, objectively verifiable: the right API called with the right arguments, the right record retrieved, the issue actually resolved in the system, the compliance step performed. Ground truth you can query. Correctness is measured.

Quality is how it happened, and it requires judgment: clarity, efficiency, tone, whether the customer was made to repeat themselves, whether a real person would come away satisfied. No system state to query. Quality is graded, by LLM judges and human review.

[INFERENCE] The post's own two examples fail in opposite directions and are worth keeping as the canonical illustration:

• The flight-booking agent caught in a loop of repeated confirmations is correct but low quality — the booking exists, the experience was poor.
• The sycophantic agent declaring an issue resolved because the customer insisted is superficially high quality but incorrect — it reads smoothly and nothing was fixed.

The second is more dangerous, and it's what a quality-only stack systematically misses. That asymmetry is the argument for keeping both.

“capture both erroneous and "golden" interactions”

Are ‘erroneous’ and ‘golden’ binary or scores?

[INFERENCE] The tension resolves once you see the words doing two different jobs.

As curation labels, binary. Deciding whether a conversation enters the regression suite as known-bad or reference-good is a yes/no promotion decision made by a human. A 0.7 on that axis would mean nothing operationally.

As an evaluation criterion, graded. A Golden Response evaluator compares actual output against the stored reference. Binary exact match would fail on every valid rewording — the precise failure the post opens by criticizing. So it produces a similarity score against a threshold.

So: binary going in, scored evaluating against.

[INFERENCE] A binary curation label also loses information. "Erroneous" flattens a compliance violation and a clunky phrasing into one bucket, which matters if you later want to prioritize the suite or weight failures by cost. A severity field is cheap at curation time and expensive to reconstruct afterward.

“But the majority of people who churn out of SNAP do so from failure to successfully navigate the web of actions required to stay enrolled once eligible”

What is this web of actions?

[SEARCH: 7 CFR 273.12 and 273.14] The recurring obligations are four distinct things, each with its own deadline and its own notice:

1. Change reporting. Under simplified reporting, which most states use, the household must report when gross monthly income crosses the limit for its household size. Under full change reporting, the household must also report unearned income changes over $100, any addition or loss of a household member, residence changes affecting shelter costs, and work hours falling below 20 per week. The deadline is 10 days.
2. The periodic or interim report. A mid-certification form, typically around month six of a twelve-month period. Failing to return it terminates participation.
3. Recertification. A notice of expiration, a new application, an interview at least once every 12 months, and verification documents, generally due at least 10 days before the certification period ends.
4. Work-requirement documentation for able-bodied adults without dependents, which HR 1 extends up to age 65.

[INFERENCE] The word "web" is doing real work here. These are not one process with four steps; they are four independently timed processes whose triggers overlap, and a single life event can put a household into more than one of them at once.

“We conducted a compensated survey with 54 SNAP recipients across 23 states using our voice-based research assistant”

What could the backend of this voice assistant look like?

[INFERENCE] The conventional stack for this is either a pipeline — telephony or in-app WebRTC audio, streaming speech recognition, a language model that holds the interview guide as a loose state machine, then text-to-speech — or a single speech-to-speech model, which cuts the turn latency that makes pipelines feel stilted. The interview guide matters more than the model choice: it has to specify which topics must be covered, when to follow up, and hard guardrails against giving eligibility advice, since a research interview that accidentally advises someone about their case is a liability. After the call you would expect transcription, personally identifiable information redaction, and model-assisted coding of transcripts into themes.

[SEARCH] Several commercial platforms do exactly this kind of AI-moderated interviewing — Listen Labs, Outset, and Strella are the established ones as of 2026. The two metrics the post reports, completion rate and average conversation length, are the metrics such platforms surface natively, though the post does not say whether Propel bought or built.

“Could voice be an effective interface for helping someone report an address change?”

Is the hypothesis that people (a) find it easier to communicate over voice or (b) the address change is more likely to come up over the course of a full conversation as opposed to being proactively volunteered?

[INFERENCE] The evidence in the post supports (b) and points at a third mechanism that I think is the author's actual claim.

Support for (b) is direct: "In several conversations, interviewees reported case changes that had not yet been reported to their state SNAP program." Those disclosures emerged during conversation rather than being proactively volunteered. The strongest single instance is the participant who had given the same address for four years and corrected herself partway through — "I don't live there anymore actually. I'm homeless now." A form would have collected the stale address and recorded it as accurate.

The third mechanism, which the post emphasizes more than either of yours, is translation. Voice lets someone narrate an event in ordinary language and have the system decide which administrative transactions it maps to, instead of requiring the person to pick the correct transaction first. That is about the mapping burden, not about typing ease or recall.

Option (a) is plausible and consistent with the six-minute average completion, but the post offers no evidence for it.

“Although SNAP programs are federally required to deliver notices by mail”

Only by mail or do they do email too?

[SEARCH: FNS guidance on electronic notice options] The claim as written is imprecise. Since November 2017, states may send SNAP notices electronically without a waiver, subject to conditions: the household must consent, the state must provide security protections, and the household can opt back to paper at any time. The permitted combinations are email plus a secure online account, or email plus text. Text alone still requires a waiver, because it gives no delivery confirmation. If an email address becomes inactive, the state must automatically resume paper notices, and paper must remain available on request.

[INFERENCE] The accurate framing is that paper is the default and the mandated fallback, not the only lawful channel. That does not weaken the post's argument so much as relocate it: electronic delivery requires an affirmative opt-in, which is itself one more procedural action that the households most likely to move are least likely to have completed. Brittleness survives the correction.

“Reporting an address change presumes you have an address to report.”

Are there any general mail reception centers that you can put down if you don’t have a specific address? Like a USPS location or something?

[SEARCH] Yes, and a fixed residence is not required for SNAP eligibility. Pennsylvania's SNAP policy manual, which is typical of state manuals, lists as acceptable mailing addresses: a friend or relative's address, a post office box, a social service agency, "any address where the client can get mail without threat of it being lost or stolen," and the county assistance office itself as a last resort.

The USPS option you are describing is General Delivery. Mail addressed to a person's full name, then "GENERAL DELIVERY," then city, state and ZIP, is held at a designated post office for pickup. It is held a maximum of 30 days, pickup requires photo identification matching the name on the mailpiece, and not every post office offers it — availability has to be checked through the USPS location finder.

[INFERENCE] Each of these presumes the person knows the option exists, can reach the pickup point, and holds valid photo ID. Those are the same access constraints that produced the original failure, which is likely why the post frames it as a design problem rather than an information problem.

“we might treat a recent move or other life events as a signal that case continuity is at elevated risk”

Who is ‘we’ here? Propel? If yes, what information do they have access to that would enable this?

[SOURCE] Genuinely ambiguous. The sentence opens "Instead of designing a better address-change process," which reads as a statement to the benefits-design field rather than a Propel product claim. Propel appears elsewhere in the post as the author's employer and as one example of a trusted existing channel, alongside an assistor or a state portal. The post never says what data would drive the risk signal.

[SEARCH] Propel's public product description covers EBT balance on the home screen, transaction tracking, deposit history, and card security features such as locking and blocking out-of-state or online transactions. Propel states it does not store EBT card PINs, complete card numbers, or Social Security numbers, and that it accesses accounts with user permission through state EBT systems. Case status, certification end dates, address of record, and notice history do not appear in that description.

[INFERENCE] So if "we" is Propel, the available signals are indirect: the geography of where transactions occur, a change in deposit amount, or a direct question to the user. That supports a prompt to ask, not a reliable risk score. A state agency, by contrast, already holds the address of record, the certification calendar, returned mail, and notice history, which is where this idea is actually actionable. My reading is that "we" means the field, and the sentence is a design provocation rather than a roadmap.

“we now have an opportunity to use language models that are unusually well suited to translation between highly personalized, unstructured, unique situations (and spoken/written languages) and highly structured administrative systems”

This seems plausible but what is the input layer/UX? Would Propel users have regular AI interviews to get status updates on their life?

[INFERENCE] Recurring scheduled interviews would be a poor design — it would add a new standing obligation, which is the burden the post is trying to remove, and it would collect most of its information when nothing has changed. Three better triggers, which are mutually exclusive and cover the realistic space:

1. User-initiated. The person opens the app with something to do — "I need to report that I moved" — and the conversation starts from their framing rather than a form.
2. Signal-initiated. An external cue starts a short check-in: an approaching recertification date, a deposit that changed amount, returned mail, or a long gap in transactions.
3. Document-initiated. The person photographs or forwards a notice they do not understand, and the assistant explains it and drafts the response.

The post's own requirement that the person confirm and review before submission implies a visible review step, which suggests voice as an input mode inside an app rather than a voice-only product.

“AI could make those systems easier to navigate by absorbing some of that translation work.”

Concrete example?

[INFERENCE, using the reporting rules from Question 1] That one sentence contains three separately reportable items:

1. A household composition change, since the household she is now part of is not the household on file.
2. A shelter cost change, which affects the excess shelter deduction and therefore the benefit amount.
3. An earned income change, which under simplified reporting is reportable only if gross monthly income now exceeds the limit for the household size — and the household size just changed, so the threshold itself moved.

A system absorbing the translation work would recognize all three from the sentence, compute the new income threshold for the new household size, ask only for the hours and wage rate needed to test it, tell her which items actually require reporting and that the deadline is 10 days, name the documents she will need (a lease or rent receipt, recent pay stubs), pre-fill the portal submission, and show her the structured result before anything is sent.

Today, every step of that reasoning is the recipient's unpaid work, and getting it wrong looks identical to not caring.

“It could also make verification, monitoring, and information requests cheaper for institutions, enabling more administrative complexity rather than less.”

Concrete example?

[SEARCH: HR 1 SNAP provisions] HR 1 makes the incentive concrete. Beginning in FY2028, states pay a share of benefit costs scaled to their payment error rate: 5% below a 6% error rate, rising through 15% and 20% to 25% above a 10% error rate. Administrative cost sharing shifts from an even split to 75% state. Work requirements move to ages 17 through 65, the ABAWD exemption age rises from 55 to 65, parents of children aged 7 and above lose their exemption, and exemptions for veterans, people experiencing homelessness, and former foster youth sunset on October 1, 2030.

[INFERENCE] Put those together. A state now has a direct financial reason to drive its error rate down and a much larger population whose work hours must be verified. If an agent can call employers, parse pay stubs, cross-check payroll databases, and generate information requests at near-zero marginal cost, monthly verification becomes affordable where annual verification was the practical ceiling. Every automated request is still a deadline a recipient must meet with documents they may not have. Agency cost falls; the number of procedural failure points rises. One commenter on the post makes the same argument by analogy to automated claims denial in private health insurance — the technology is neutral about which direction it is pointed.

“A person described what happened in ordinary language and the system identified relevant facts”

Does the system require additional context beyond its pretraining/world knowledge? I’m assuming all relevant guidelines are public and digitized, right?

[SEARCH and INFERENCE] Your assumption is mostly right on availability and wrong on sufficiency. Four separate context needs:

1. Federal rules are public and digitized. 7 CFR Part 273 is on eCFR, and FNS guidance documents are on the USDA guidance portal. Most state policy manuals are also on the open web — Pennsylvania's, which I used for Question 5, is an ordinary public web page.
2. But state variation is the whole problem. SNAP is state-administered with wide option-level variation: simplified versus full change reporting, certification length, interview waivers, standard utility allowances, ABAWD waiver areas. A model answering from a pre-trained average will be confidently wrong for a specific state, and HR 1 implementation is changing these rules right now, which makes the pre-training cutoff a live hazard rather than a theoretical one.
3. Some required context is not in any manual. The specific portal's form fields and submission mechanics, what the county office accepts as document intake, and current processing timelines are operational facts, not policy text.
4. Case-specific state is agency data. The certification end date, what is already on file, which notices were sent and when. No amount of document retrieval provides this; it requires an integration with the state system of record.

So: retrieval over versioned federal and state policy with effective dates attached, plus a read integration with the agency. Pre-training alone produces fluent, plausible, state-inaccurate answers, which is precisely the failure mode a high-stakes benefits system cannot absorb.

What are CSAT and resolution rate? How are they measured? Are the numbers available immediately or is there any latency?

From Decagon's own glossary (separate source): CSAT is the share of surveyed customers giving a top rating — their example is 60 of 100 customers rating 4 to 5 out of 5, giving 60% — collected by a short survey delivered "immediately after a meaningful customer touchpoint." Resolution rate is resolved issues divided by total issues times 100, where resolved means the customer needed no further follow-up.

Inferred from background knowledge: the latency question is the important one, and the two metrics behave differently. CSAT arrives within minutes to days, but only from people who respond. Support survey response rates typically sit well below half of contacts, so a per-variant CSAT estimate needs days to weeks of traffic before its confidence interval is narrow enough to act on. Resolution rate is worse, because "no further follow-up" requires waiting out a re-contact window of roughly one to seven days. Until that window closes the measurement is censored, and early readings look systematically better than the settled value. Both are therefore lagging indicators, which is exactly why teams pair them with immediate proxies such as escalation rate, containment, and judge scores computed on live traffic.

“We use models from OpenAI, Anthropic, Gemini, and other providers, alongside our own fine-tuned versions of open-source and commercial models.”

Why so many different models?

Inferred: five reasons, roughly in order of how much they matter.

First, an agent is not one model call. Intent classification, retrieval reranking, guardrail checks, summarization, and final response generation sit at very different points on the cost-latency-quality curve. A small fine-tuned classifier can do intent routing at a small fraction of the cost and latency of a frontier model, and there is no quality reason to use the expensive model there.

Second, enterprise deployments differ in tone, domain vocabulary, and compliance posture, which is what fine-tunes buy you.

Third, their evaluation apparatus makes heterogeneity self-generating. The post says an experiment variant "might involve anything from a new system prompt to an entirely new language model," so if you continuously run model bake-offs per workflow, you end up with a different winner in different places.

Fourth, vendor concentration risk: capacity limits, price changes, deprecations, outages, and data-residency requirements all argue for being able to swap providers.

Fifth, judge independence. Generating with one model family and judging with another avoids the self-preference bias where a model scores its own outputs too highly.

“We rigorously assess every component of the agent, from response generation to retrieval to safety.”

How do you assess safety?

Inferred from standard practice: safety evaluation for a customer support agent breaks into four fairly distinct things.

The first is an adversarial test suite held fixed across releases, covering prompt injection, jailbreaks, requests that would leak another customer's data or PII, out-of-scope advice such as legal, medical, or financial guidance, and brand-unsafe statements. These are scored pass or fail against a written policy, and you measure both directions of refusal: the agent should refuse genuinely unsafe requests and should not refuse benign ones.

The second, and for a support agent the most consequential, is groundedness. The realistic safety failure here is not toxicity but a confident, wrong policy statement, such as telling a customer they qualify for a refund when they do not. You test this by checking that every factual claim in the response is supported by the retrieved context, using either entailment models or an LLM judge given the source material.

The third is action safety. Once the agent can call tools, the question becomes whether it ever took an irreversible action such as issuing a refund or cancelling an order without the required verification.

The fourth is online: guardrail classifiers running inline at inference time, plus continuous sampling of live conversations for review, which is what a Watchtower-style always-on QA product does.

“Every variant must demonstrate consistent performance offline before it is eligible for online testing”

What does consistent mean in this context?

Inferred: two readings are plausible and both are probably intended. The first is consistency across strata, meaning no regression on any sampled slice rather than just a better global average. This reading is supported by the post's emphasis that sampling is stratified across issue types and customer segments; the whole point of stratifying is so that a variant which lifts the aggregate while breaking one customer's billing flow gets caught. The second is consistency across criteria and across repeated runs: no material regression on relevance, correctness, naturalness, or empathy individually, and stable results when the judge is re-run, since LLM judges are stochastic and a single scoring pass has real variance.

In practice this usually gets implemented as a non-inferiority test per slice — the variant must score at least as well as control minus a small tolerance on every gate metric — combined with a genuine win on whatever metric the experiment was designed to move.

“Using an LLM-as-judge system, we evaluate structured triplets consisting of a user query, the context provided to the model, and the model's generated response.”

Only initial user query or intermediate queries as well?

Inferred: almost certainly any turn, not just the first. Two things point that way. The middle element, "the context provided to the model," only has meaning per generation call — it is the assembled prompt for that specific turn, containing retrieved knowledge base passages, conversation history so far, and customer account data. And sampling only opening turns would miss where support conversations actually break down, which is in clarification turns, turns following a tool call, and the decision to escalate.

The limitation worth naming: grading triplets is a static evaluation. Each response is scored conditioned on a history that a human actually produced, so it cannot capture errors that compound across a conversation, where an early turn steers the dialogue somewhere the grader never observes and every later response looks locally reasonable given a history the agent itself would never have reached. The online A/B test is the only part of their described system that catches this, which means failures of that kind are discovered on real customer traffic rather than offline.

“These triplets are drawn from real-world interactions, ensuring that our evaluation process reflects authentic user needs and scenarios”

Sample size and sampling process? How to ensure representativeness/diversity?

Stated in the post: they "sample across key dimensions, such as issue types and customer segments, to ensure balanced testing." No sample size, no selection mechanism, no mention of reweighting.

Inferred: there is a real tension hiding in the word "balanced." Balanced sampling and representative sampling are different objectives. Balancing oversamples rare strata so that each slice has enough data to say something about it, which necessarily distorts the mix away from production. That means the unweighted aggregate score is not an estimate of production quality. To recover a number that describes live performance you have to weight each stratum back by its true traffic share. The post says nothing about doing this, and it is the kind of detail that quietly makes an offline number uninterpretable.

On sizing, the quantity that matters is observations per slice, not the total. As a rough anchor: detecting a 0.1-point shift on a 1-to-5 scale with a standard deviation near 1 needs on the order of 1,600 observations per arm, and resolving a 3-point difference in a pass rate sitting around 90% needs roughly 1,800 per arm. Both numbers drop substantially with a paired design, where the same triplet is scored under both the control and the variant. Since the context is fixed in their setup, pairing is the obvious choice and I would expect them to use it.

“Each response is scored against several key criteria as part of a scalable, continuous, and stratified evaluation process”

Any weighting of criteria? Binary or scores?

Inferred: the criteria are almost certainly scored separately rather than blended, because they are not substitutable. Correctness is a floor and empathy is a preference. If you average them into one number, a warm, fluent, confidently wrong answer can outscore a blunt correct one, which is the single failure an enterprise support vendor cannot ship. The design I would expect, and would argue for, is correctness and relevance as hard gates with pass thresholds, with naturalness and empathy as optimization targets that can only be traded against each other inside the set of responses that already passed.

On the scale: ordinal scores, most likely a 1-to-5 Likert or a three-point pass, borderline, fail scheme, rather than pure binary. Binary throws away information about near-misses, and scales beyond about five points exceed what a judge can apply consistently. Worth knowing that LLM judges are meaningfully more reliable at relative comparisons than absolute ones, so pairwise scoring of variant against control on the same triplet, with position randomized, is the more trustworthy measurement. It does not give you an absolute threshold to gate on, which is probably why an absolute scoring scheme is used here.

If weights do exist, the principled way to set them is to fit them against observed CSAT, so the composite is the thing that actually predicts the business metric rather than a number chosen by intuition.

“We also audit a subset of triplets with human labellers”

What does this mean? What’s being audited?

Inferred: the object under audit is the judge, not the agent. The human label is treated as the reference and the judge's score as a prediction, so the output of the exercise is judge-quality measurement. That means agreement with humans, reported both as raw agreement and as a chance-corrected statistic such as Cohen's or Krippendorff's kappa, and reported per criterion and per slice rather than pooled. Judges are typically strong on relevance and weak on correctness, because correctness requires domain knowledge the judge does not have.

The second function is calibration. If the judge says 92% correct and humans say 85%, you have learned the offset and can either correct the threshold or repair the rubric.

The third is drift detection, and it is the one teams usually discover too late. When the judge model is upgraded, the score scale shifts silently and historical thresholds stop meaning what they meant. A recurring human-audited set is how you notice that the measuring instrument moved rather than the agent.

One thing worth measuring alongside all of this is human-to-human agreement, since it is the ceiling. A judge that agrees with humans as well as humans agree with each other has nothing left to gain.

What model for LLM-as-judge?

Inferred: rather than guess theirs, here are the constraints that determine the choice. It should be a strong model from a family other than the one generating the responses, because judges exhibit self-preference and score their own outputs higher, a well-documented effect in exactly the literature the post cites. It must be pinned to a specific version, because a silent provider-side upgrade moves your metric without the agent changing, and every historical comparison breaks.

Cost is the other constraint. Judging every response with a frontier model is expensive at their volume, so the common pattern is a smaller distilled judge, fine-tuned on frontier-judge plus human labels, doing the bulk scoring, with the frontier model scoring a sample to keep the cheap judge calibrated.

For correctness specifically, a judge without the source material can only assess plausibility, not accuracy. Their triplet format solves this by construction, since the retrieved context is part of what the judge sees.

What prompt/context given to LLM-as-judge?

Inferred: a well-built judge prompt contains six things.

1. It states the role and presents the three triplet elements under clear delimiters, with an instruction to evaluate only the response.
2. It gives one rubric per criterion with each score level anchored by a concrete worked example rather than an adjective, because "3 means partially addresses the question" is not reproducible while a demonstrated example is.
3. It states explicitly that correctness is judged against the provided context, and that a claim unsupported by that context counts as incorrect even when plausible. Without this the judge grades against its own world knowledge and hallucinations pass.
4. It asks for brief reasoning before the score, then structured output such as JSON so results parse reliably.
5. It controls for known biases: randomize ordering in any pairwise comparison, never reveal which variant produced a response, and discourage rewarding length, since judges systematically favor longer answers.
6. It few-shots from the human-audited examples, which has the useful side effect of tying the rubric directly to the human ground truth.

“In parallel, we evaluate responses against a ground truth evaluation set: a curated collection of user queries with ideal responses”

How concretely to evaluate generated responses against ideal responses?

Inferred: the phrase "factuality and intent coverage" describes two different measurements, and they need different machinery.

Intent coverage asks whether the response addressed everything the ideal response addressed. The robust way to measure this is to decompose the reference answer into the discrete facts or steps it contains, then check each one for presence in the generated response. That gives you recall over required elements, paired with a precision check for content the response asserts that the reference does not support.

Factuality is per-claim entailment: extract the claims in the generated response and classify each as supported, contradicted, or unsupported relative to the reference and the source documents.

What not to use is equally important. Exact match, BLEU, and ROUGE punish valid rephrasings and reward parroting, and support answers have many correct phrasings. Raw embedding similarity fails precisely where the stakes are highest, since "you will be refunded" and "you will not be refunded" sit almost on top of each other in embedding space.

One operational issue the post does not raise: ground truth sets decay. When a return window changes from 30 days to 60, the ideal response silently becomes wrong and the benchmark starts penalizing correct answers. The set needs an owner and a refresh cadence tied to policy changes.

“we apply similar rigor across the full spectrum of our AI workflows to ensure each component performs reliably”

How are the other components evaluated?

Inferred, component by component:

• Retrieval is measured with recall and precision at k and ranking metrics such as nDCG against labeled query-to-document relevance, because generation cannot be correct if the right document was never retrieved. The separate and more useful measurement is context sufficiency: whether the retrieved set contains enough to answer at all. That split matters because it tells you whether a wrong answer is a retrieval failure or a generation failure, and those go to different owners.
• Intent classification and routing get per-intent precision and recall with a confusion matrix. The expensive errors are misroutes on high-severity intents such as fraud or cancellations.
• Tool and API calls are scored on recorded traces for whether the right tool was chosen, the arguments were correct, and errors or empty results were handled.
• Multi-step reasoning and policy adherence are tested with scripted scenarios that have required steps in a required order, such as verify identity, then check eligibility, then act.
• Escalation is a classifier decision with its own precision and recall. Both error types cost something: unnecessary handoffs cost money, missed handoffs cost trust.
• End to end, the method that covers what triplets cannot is simulated multi-turn conversations driven by a user-simulator model and scored on task completion.

“We manage rollouts by gradually increasing traffic to the variant group as performance improves”

How concretely to determine when to increase traffic?

Inferred: a defensible ramp has four parts.

There is a fixed ladder of traffic shares, something like 1%, then 5%, 25%, 50%, with a hold at each rung long enough for the lagging metrics to settle, meaning at least one full re-contact window and ideally a full week to average over day-of-week effects.

There are two classes of metric with different decision rules. Guardrails such as escalation rate, safety violations, error rate, and latency are evaluated as non-inferiority tests with automatic rollback when breached, and they move fast enough to read at low traffic. Success metrics such as CSAT and resolution rate are slow and usually only become decisive at higher traffic shares, so they gate the last steps rather than the first.

The statistical point that most teams get wrong: if you look at the data repeatedly and can stop early, ordinary fixed-horizon p-values are invalid and your false positive rate is much higher than you think. The fix is sequential testing — always-valid p-values, mixture sequential probability ratio tests, or group-sequential boundaries — so that continuous monitoring is legitimate rather than decorative.

Finally, promotion should require that no guardrail was breached, not merely that the average improved. An average can absorb a severe regression in one segment, and for a multi-tenant vendor that regression is concentrated on a single customer.

“Customers can also opt out of experimentation, giving them full control over model exposure during evaluation.”

Validity concerns because of selection bias?

Inferred: yes, but it is worth being precise about which kind of validity is threatened, because the two answers are different.

Internal validity survives. Opt-out happens at the customer level and before randomization, and it does not depend on which arm anyone lands in. Randomization within the remaining population is therefore still clean, and the estimate is unbiased for the population that participates.

External validity is what suffers. What you measure is a local effect on opted-in customers, and opt-out is emphatically not random. The customers most likely to opt out are the largest, the most regulated, the most risk-averse, and often the ones with the most complex workflows, which is plausibly the same set where a new model performs worst. So the measured lift is probably optimistic relative to what happens after full rollout, and the bias runs in the direction that flatters the variant.

There is a second problem the post's framing hides. Assignment at the customer level means the effective sample size is the number of customers, not the number of conversations. Conversations within one customer share policies, intent mix, and phrasing, so they are heavily correlated, and conversation-level confidence intervals computed naively will be far too narrow. This needs clustered standard errors or a customer-level analysis.

Mitigations worth asking about: report what share of traffic is opted in, compare the opted-in and opted-out populations on observable characteristics, reweight results toward the full traffic mix, and roll out to opted-out customers in a later, more heavily monitored stage.

“we monitor closely for noise, novelty effects, and performance drift over time”

How are each of these concretely monitored?

Inferred, taking them in order:

Noise is handled by measuring it before you need it. Run A/A tests, splitting control against itself, to learn each metric's natural variance and confirm the randomization is sound, then compute a minimum detectable effect so you do not read differences the design cannot resolve. Variance reduction using pre-period data (the CUPED technique) tightens intervals materially and is nearly free when you have historical metrics per customer. Sequential boundaries keep repeated looks from manufacturing significance.

Novelty effects are detected by plotting the treatment effect against days since exposure rather than pooling the whole period. A novelty or primacy effect appears as an effect that decays or grows over the first days and then flattens, and the rule is to hold the ramp until the curve is flat. Splitting new from returning end users separates the two mechanisms. In a business-to-business agent context it is worth noting that the novelty often sits with the human support agents observing the change rather than with end customers.

Drift is monitored by watching the control arm over time. If control itself moves, something outside the experiment changed: traffic mix, a policy update, an upstream model version, seasonality. Alongside that you monitor input distribution shift in intent mix and query embeddings, and you keep a pinned judge scoring a fixed reference set. If scores on the fixed set move, the measuring instrument drifted rather than the agent, and that distinction is the whole game.

“Insights from every phase feed back into the experiment loop: improving prompts, upgrading models, and refining overall agent behaviors.”

How exactly are these insights used?

Inferred: the version of this loop that actually compounds is error-driven rather than score-driven. A score tells you whether to ship; it never tells you what to change. So the first step is clustering the low-scoring triplets by failure mode — retrieval missed the document, retrieval succeeded but the model answered from priors, tone was wrong, escalation was missed, the tool call was malformed — and sizing each cluster. That converts an uninterpretable three-point score drop into a ranked list of fixes.

Each mode then routes to a different owner. Retrieval misses go to knowledge base content, chunking, or the reranker. Correct-context-but-wrong-answer goes to the prompt or the model. Policy violations go to guardrails. Tone goes to the prompt or a fine-tune.

The mechanism that makes the loop accumulate value is promoting confirmed failures into the ground truth set, so every real defect becomes a permanent regression test and the same bug cannot ship twice. Running the other direction, escalated and low-CSAT conversations from online tests should become the next cycle's offline triplets, which is what keeps the offline set representative as traffic shifts underneath it.

Decagon's own GEPA post is a concrete instance of this loop: prompts optimized against a labeled holdout of roughly 600 examples, reported as a 5 to 6 percent improvement, with the stated discipline being to "treat prompt engineering as software engineering." That is a separate source, not this post.

How would you improve this system if you were building it today?

1. Add conversation-level evaluation. The largest gap is that offline evaluation is entirely turn-level. Triplet grading conditions each response on a history a human produced, so it structurally cannot see errors that compound across a conversation. Add simulated multi-turn evaluation with a user-simulator model and task-success scoring, so those failures get caught before customer traffic rather than during it.
2. Treat the judge as an instrument requiring continuous calibration, not a fixed oracle. Report agreement with humans per criterion and per slice, track it over time, pin judge versions, and re-baseline explicitly whenever the judge changes. An uncalibrated judge score is an unvalidated sensor, and the entire offline gate rests on it.
3. Make correctness a gate rather than a term in an average, for the reason in question 7.
4. Calibrate offline metrics against online outcomes. This is the measurement I would most want that the post does not mention. The open question is how well an offline score delta predicts the CSAT and resolution-rate delta. You can answer it by backtesting every past experiment, comparing the offline predicted effect against the observed online effect. If that correlation is weak, the offline gate is ceremony. The exercise also yields empirically grounded thresholds instead of chosen ones.
5. Reweight the stratified sample back to the production traffic mix, so the aggregate offline number estimates something real.
6. Adopt sequential testing with pre-registered guardrails and automatic rollback, and use clustered inference given tenant-level assignment.
7. Promote cost and latency to first-class evaluation metrics alongside quality. In live support a three-second response that is marginally worse frequently beats a fifteen-second response that is marginally better, and neither appears anywhere in the post.
8. Add offline replay of logged conversations against a new variant as a cheap pre-screen before spending any traffic.
9. Build per-customer dashboards rather than aggregate ones, because a vendor's risk is concentrated per tenant and an aggregate win that hides one customer regressing is a churn risk the average conceals.

“gradient-free optimizer that uses natural language reflection rather than policy gradients to adapt prompts”

What does policy gradient mean?

In RL, the “policy” is the model that chooses outputs. For an LLM, the weights set the probability of every token. A policy gradient method repeats three steps:

1. Sample outputs from the model for many inputs.
2. Score each output with a single number, called the reward.
3. Adjust the weights in the direction that makes high-reward outputs more likely. That direction is the gradient of expected reward with respect to the weights.

GRPO (Group Relative Policy Optimization) is one such method. It samples a group of answers for each input and scores each answer against the group’s average.

GEPA is “gradient-free” because it never changes weights. It edits the prompt text, guided by an LLM’s written diagnosis of what went wrong. A single reward number says how good an output was but not why, which is a big reason policy gradient methods need so many more samples.

“it outperforms reinforcement learning methods like GRPO by up to 20%”

1. By what metric?
2. Is it consistently better across most applications or does it vary?

From the GEPA paper (web, arXiv v2, accepted to ICLR 2026):

(a) The metric is each benchmark’s own score on a 0–100 scale, such as exact match, F1 or pass rate depending on the task. The “20%” is a 19-point absolute gap on HotpotQA with Qwen3 8B (62.33 for GEPA versus 43.33 for GRPO), not a relative gain.

(b) It varies a lot. GEPA beat GRPO on five of six tasks, by 19.0, 13.66, 5.19, 2.73 and 0.69 points, and lost on AIME-2025 math by 6 points (32.00 versus 38.00). The average gain was about 6 points. GRPO was run only on Qwen3 8B, with a fixed 24,000 rollouts.

Inference: “Up to 20%” is the best case. Two of the five wins were under 3 points, which may be within noise. The biggest wins came on multi-hop retrieval tasks (HotpotQA and HoVer), where better instructions about what to search for matter most. The one loss came on competition math, where weight training can build reasoning skill that instructions alone cannot.

“35× fewer model rollouts”

What is a model rollout?

Inference (background knowledge): A rollout is one complete run of the system on one input, followed by scoring the result. For Decagon’s supervisor, one rollout means giving the model one conversation with the current prompt, letting it write its reasoning and label, and comparing that label with the ground truth. The term comes from RL, where it means playing out one full episode with the current policy. Rollouts are the natural unit of cost because each one needs at least one LLM call.

From the GEPA paper (web): Budgets are counted in rollouts. GRPO used a fixed 24,000 per task. GEPA’s total budgets ranged from 1,839 to 7,051, and the paper says GEPA reached its best test performance with 4 to 35 times fewer rollouts.

Inference: GEPA’s total budgets were only about 3 to 13 times smaller than 24,000, so the 35× figure must be counted at the point where GEPA hit its best score, not over its full spend. It is a best-case number.

“We applied GEPA to optimize prompts for a production classification task”

What is the actual task? Input, output and objective?

From the post: The model is a supervisor that “analyzes conversations” and produces structured outputs with reasoning traces. It does binary classification, each prediction must be justified by a chain of reasoning, and the data has ground-truth labels (600+ labeled examples). The post calls it “a last line of defense against hallucinated or inconsistent outputs,” but never names the two classes or the metric.

From Decagon’s guardrails post (web, July 2025): Every message Decagon’s AI agent generates is reviewed by a supervisor model before it reaches the customer. The supervisor checks whether the reply stays on topic, reflects the conversation’s intent and sticks to known facts, and it can revise the reply or escalate.

Inference: Assuming this is the same supervisor, the task most likely looks like this:

• The input is the conversation so far plus the agent’s draft reply, possibly with the knowledge or policy text the agent relied on.
• The output is a reasoning trace followed by a binary verdict, roughly “OK to send” versus “flag.”
• The objective is to maximize agreement with human labels on the holdout set, using a prompt short enough to meet the latency target.

“GEPA operates through four main steps”

Is this cyclical or one-time?

From the post: Cyclical. The section is titled “The core loop,” and the post mentions “Each reflection cycle,” “More iterations,” and a reflection model called “~10-20 times during optimization,” which implies roughly 10 to 20 passes.

From the GEPA paper (web): The loop repeats until the rollout budget is used up. Each pass works like this:

1. Pick a parent prompt from a pool of candidates.
2. Run it on a small batch of training examples and collect feedback.
3. Have the reflection model write an improved prompt.
4. Check whether the new prompt beats its parent on that same batch. Only if it does, score it on the full validation set and add it to the pool.

Parents are sampled from the prompts that achieve the top score on at least one validation example (the “Pareto frontier”), so several different strategies stay alive. At the end, GEPA returns the prompt with the best average validation score.

Inference: The post’s four steps compress a loop that has two gates: a cheap check on the batch, then an expensive check on the full validation set.

“Reflection: A "reflection model" (typically a frontier LLM) analyzes failures and successes, identifying patterns in what works and what doesn't“

All examples fed into a single prompt or multiple?

From the post: Neither all at once nor one at a time. The baseline uses “10 examples per reflection,” and the batch-size ablation tested “how many examples the reflection model should see at once.” So each reflection call sees one batch of 10, and different calls see different batches.

From GEPA’s code, the DSPy docs and the GEPA paper (web): The default reflection prompt packs the current instruction plus, for every example in the batch, its inputs, the model’s response and the feedback into a single call. DSPy’s default batch size is 3, and the paper used 3, so Decagon’s batches were more than three times the standard size. Batches come from the training split, while the validation split is only used for scoring.

Inference: The task model works differently: it is called once per example. Because the reflection model never sees validation examples, validation scores are not contaminated by what the reflector has read.

“Budget: 1.0x multiplier (~150 LLM calls)“

What role does the budget play here? Isn’t the number of calls fixed by the number of examples?

From the post: Budget was one of the seven ablations, with “Diminishing returns beyond 1× baseline.” Calls also scaled with data: about 60 calls for 20 samples and about 150 for 50.

From the GEPA paper (web): The budget caps total rollouts, and the loop keeps cycling until the cap is reached.

Inference: The examples don’t fix the call count, because GEPA is iterative. The examples set the cost of each cycle, and the budget sets how many cycles run. At baseline, the costs are roughly as follows:

• Scoring the seed prompt on the 50 validation examples costs 50 calls, once.
• Each cycle costs 21 calls: 10 to run the current prompt on a batch, 1 reflection call, and 10 to score the new prompt on that batch.
• Each new prompt that wins its batch costs 50 more calls for validation.

So 150 calls buys only about 2 to 4 cycles. That conflicts with the post’s “~10-20” reflection calls, so at least one figure is loose. The “1.0x multiplier” is probably Decagon’s scaling of a budget tied to dataset size, since both 60/20 and 150/50 work out to 3 calls per sample.

“Batch size: 10 examples per reflection”

Does this imply the GEPA process is cyclical/repeated?

From the post: The batch size alone doesn’t prove it, since a one-shot optimizer could also look at a batch. The post confirms repetition elsewhere: “Each reflection cycle sees a batch of examples and proposes refinements,” and the reflection model is called about 10 to 20 times (see #5).

Inference: Yes, and batching is what makes repetition necessary. With 50 training examples and 10 per reflection, no single call sees all the data, so several cycles are needed to learn from all of it. This resembles minibatch gradient descent, which makes many small, noisy updates instead of one update from the full dataset.

Batch size trades off two things. Larger batches let the reflector compare more failures in one call, but they make each reflection prompt longer and more expensive. Smaller batches are cheaper but give a noisier picture of what is going wrong. The post rated this knob “Low impact” and found 10 “sufficient.”

“20–100 is optimal; 500 hurts performance”

Did they test 500 examples with a length-constrained prompt? I wonder if that would fix the overfitting issues.

From the post: No such run is reported. Each experiment “modified exactly one parameter,” and the baseline had no length constraint, so the 500-sample run was unconstrained. It produced prompts 75% longer, performance 2% lower and 10× the compute, which the post attributes to “More iterations with more data.”

Inference: A length cap would likely fix the bloat but only partly fix the performance, for three reasons:

1. The 500-sample run changed two things at once, more data and more cycles, so we can’t tell which one caused the drop.
2. Some bloat is built into GEPA. Its default reflection prompt (web, GEPA code) tells the model to “Identify all niche and domain specific factual information about the task and include it in the instruction.” A cap would make the reflector squeeze niche rules into fewer characters rather than stop chasing them.
3. A 2% drop may be within noise (see #18).

The clean test is a two-by-two design: 50 versus 500 samples, each with and without the 1,500-character cap, holding the number of cycles fixed.

“The optimal range of 20-100 samples provides sufficient diversity”

How are the 20 selected in the first place? Any measures to ensure representativeness?

Inference: It was most likely a random draw, possibly balanced by label. A random 20 carries two distinct risks:

1. It can miss whole groups. If, say, 10% of conversations should be flagged, a random 20 contains 2 flagged cases on average, and about 1 draw in 8 contains none.
2. Even with every group present, which specific examples land in the set can swing the result. The +1% gap between 20 and 50 samples could come from this alone.

Better practice would address three separate goals:

1. For coverage, stratify the draw by label and by known failure type, so every failure mode appears.
2. For learning signal, over-sample cases the seed prompt gets wrong, since GEPA learns mainly from failures.
3. For reliability, repeat the run on several different random subsets and report the spread.

“clear inverted-U relationship”

They say U-shape but the numbers below it seem to suggest it’s monotonically decreasing?

From the post: The bullet list starts at 20 samples, the peak, so it only shows the downslope. The table right below it adds a 10-sample row. Relative to the 50-sample baseline, performance was −5% at 10 samples, +1% at 20, the reference at 50, “Similar” at 100 and −2% at 500. That rises and then falls, which is an inverted U. A regular U would fall and then rise.

Inference: “Clear” is generous, for three reasons:

1. The rising side rests on a single point, the 10-sample run.
2. The gaps among 20, 50 and 100 samples are 1% or less and likely within noise (see #18), so the top is really a plateau.
3. Compute grew with sample size, so the curve mixes a data effect with a cycle-count effect.

A more accurate summary is too little signal below 20, a plateau from 20 to 100, and a mild decline by 500. Also, if the frontier models’ “+5–6% improvement” is measured against the seed prompt in the same units, the −5% at 10 samples means that run barely beat the seed prompt.

What metric is being measured by “Performance”?

From the post: It is never defined. The clues are that the task is binary classification with ground-truth labels, that “All configurations were evaluated on a fixed holdout set,” and that results are reported as percentage changes.

Inference: It is most likely accuracy on the holdout set, meaning the share of conversations whose predicted label matches the human label. It could also be F1, which balances precision and recall. Three further ambiguities matter when reading the tables:

1. The units are unclear. “−2%” could mean 2 percentage points or a 2% relative drop.
2. The reference points differ by table. The sample-size table compares against the 50-sample optimized prompt, the length table against the unconstrained prompt, and the frontier models’ “+5–6% improvement” most likely against the seed prompt.
3. The scope is unclear. The model must justify each label, but nothing says whether the reasoning is scored or only the label.

For a supervisor, a missed hallucination is usually the costly error, so recall at a fixed precision would be more informative than accuracy.

Why is compute cost not linear in # of samples?

From the post: According to the table, it is close to linear. Relative to the 50-sample baseline, 20 samples cost 2.5× less and 500 samples cost 10× more, both exactly proportional, and the bullet counts agree (about 60 versus 150 calls). The deviations are small: 10 samples cost 6× less, where 5× would be proportional, and 100 samples cost 1.8× more, where 2× would be. The post is also inconsistent here, since its bullet list says 100 samples cost “2-4× the compute cost.”

Inference: Near-linear cost suggests the budget was set in proportion to dataset size, about 3 calls per sample (see #7). With a fixed budget, cost would have stayed flat. The small deviations most likely come from rounding in the post’s summary, or from how runs end: GEPA works in whole cycles, so a run stops a little under or over its cap, and that slack is a bigger share of a small budget.

“Substantive evolution”

Qualitatively judged or quantitatively?

From the post: Qualitatively, as far as the post shows. The phrase appears only as a table label for GPT-4.1, GPT-5.2 and the Claude models, next to “No change (65 chars)” for GPT-4o-mini. No measure of how much the frontier models changed the prompt is reported.

Inference: The nearby numbers are indirect. The seed prompt was apparently about 65 characters, since GPT-4o-mini left it unchanged at that length. Unconstrained runs reached 5,000+ characters, and frontier reflection models gained +5–6%. So “substantive” likely means the prompt grew and changed a lot, and performance rose. A quantitative version would do three things for each reflection model:

1. Report how much text changed, such as characters added or edit distance (the number of character edits needed to turn the seed into the final prompt).
2. Report how much the content changed, such as the number of distinct rules added.
3. Report what the change achieved, meaning holdout performance versus the seed prompt.

“the reflection model is only called ~10-20 times during optimization, while the task model (the one executing your prompts) is called hundreds of times”

Sure, but the prompt length for the former is much higher, right?

Inference: Per call, the reflection model has the longer prompt. A task call contains the prompt plus one conversation. A reflection call contains the current prompt plus 10 examples, each with its full conversation, the model’s output and feedback, so its input is several times larger, up to about 10×. That means 15 reflection calls can carry as many input tokens as 75 to 150 task calls.

If the task model ran 150 to 300 times at a similar price per token, the reflection model would be roughly a fifth to a half of input cost, not 5–10%. The claim holds only if the task side is much bigger than the post describes, for example many more task calls, long reasoning outputs, or a pricier task model. So the underlying concern is right: call counts understate the reflection model’s share of cost. The post’s call figures also don’t reconcile with each other (see #7).

“the reflection mechanism naturally accumulates details across iterations”

Any concerns about the order of examples affecting prompt?

From GEPA’s code and paper (web): The default reflection prompt contains only the current instruction and the current batch, with no memory of earlier batches, so the prompt text is the only thing carried forward. Two safeguards exist: a new prompt must beat its parent on the batch before being scored on the full validation set, and the candidate pool keeps several lineages alive.

Inference: Yes, order can matter in two distinct ways:

1. Across cycles, the process is path-dependent. Early batches decide which rules get written first, and later edits tend to add to those rules rather than rewrite them, so a different batch order could produce a different final prompt.
2. Within a batch, LLMs weigh examples unevenly by position, often favoring the first and last, which can shift which failures the reflector addresses.

The safeguards filter out edits that don’t help, but they don’t remove path dependence. Running “19+” experiments across 7 dimensions works out to about one run per setting, so order effects could explain the 1–2% gaps. Several runs with shuffled order would settle it.

“The challenge is GEPA's default implementation doesn't support length constraints during reflection. We built a custom instruction proposer that encodes the constraint directly into the reflection prompt.”

Is this guaranteed to constrain length as intended?

From the post: The limit lives in the wording of the reflection prompt, and no check on the output is mentioned, even though the overview table calls these “hard character limits.” In practice it seemed to work: the 1,500-character limit produced prompts of about 1,000 characters, and the 500-character limit produced about 400.

Inference: No, not by that mechanism alone. An instruction is a request, not a guarantee. LLMs read tokens rather than characters, so they count characters poorly, and a reflector facing many failures may overshoot. The observed undershoot also means the effective limit was tighter than the stated one. A real guarantee needs a check in code after each proposal:

1. Measure the new prompt’s length.
2. If it is over the limit, ask the reflector to shorten it, allowing a few retries.
3. If it is still over, reject the candidate so it never enters the pool.

From the DSPy docs (web): DSPy’s GEPA exposes an instruction_proposer hook for custom proposers, which is the natural place for this check.

“Better generalization by preventing overfitting to training edge cases”

How do you reconcile this with the -0.8% performance impact?

From the post: All configurations were evaluated on a fixed holdout set “to measure generalization,” so the −0.8% is presumably a holdout result. By the post’s own measure, the cap slightly hurt generalization.

Inference: The claim can be rescued in three ways, but the post directly supports none of them:

1. It could mean a smaller gap between training and holdout scores, which is less overfitting even with a slightly lower holdout score. Training scores aren’t reported.
2. It could be a bet that shorter prompts will hold up better on future production traffic, which a fixed holdout can’t show.
3. The −0.8% could be noise. The pool has 600+ examples and one run used 500, so a holdout kept separate from every run likely has only about 100 examples. At 100 examples, one conversation moves accuracy by 1 point, so −0.8% is less than one conversation. At 85% accuracy, the standard error of a single score (its typical swing from which examples happen to be sampled) is about 3.6 points.

The defensible claim is “the same holdout performance with a 4× shorter prompt,” not “better generalization.”

“achieving 4× compression”

Why is this compression valuable?

Compression pays off in three ways:

1. It cuts latency. The supervisor sits in the live reply path, and longer inputs take longer to process, so every extra token adds to the customer’s wait.
2. It cuts cost. Removing about 4,000 characters saves roughly 1,000 input tokens per call (at about 4 characters per token), multiplied by production volume.
3. It eases upkeep. A person can read and audit a 1,000-character prompt, while 5,000 characters of edge-case rules is hard to review and more likely to contain conflicting rules.

How would you improve this system if you were building it today?

Inference (my recommendations, grouped by stage):

Measurement

1. Run every configuration several times on a larger, stratified holdout and report confidence intervals, so 1–2% differences can be trusted.
2. Optimize for the costly error. A missed hallucination usually costs more than a false alarm, so target recall at a fixed precision rather than raw accuracy.

Optimization

3. Enforce the length cap in code (see #17), and replace GEPA’s default instruction to include “all niche and domain specific factual information” with one that asks for general rules.
4. Test interactions, such as 500 samples with a length cap at a fixed budget, rather than changing one factor at a time.

Operation

5. Feed human-reviewed production disagreements back into the labeled pool, and re-optimize on a schedule, promoting a new prompt only if it doesn’t regress on a fixed test set.
6. At high volume, train a smaller classifier to copy the supervisor’s labels for speed and cost, and route uncertain cases to the prompted model.

“RAG is an often-overlooked aspect of agent architecture that can quietly eat into response time.”

How does the query rewriting stage work for voice conversations? If we want to generate embeddings to retrieve relevant context, how do we structure the lookups?

Inference (background knowledge): In voice, the input is a speech-to-text transcript of the caller’s latest turn. Transcripts are messy, with filler words, misheard product names, and references to earlier turns like “what about the other plan?” A small, fast LLM reads the recent conversation plus that transcript and writes one or a few standalone search queries, resolving the references and fixing likely mishearings. Turns like “yes” or “hold on” can skip retrieval.

Lookups usually follow three steps:

1. Indexing: split help articles into chunks, embed each chunk, and attach metadata such as customer, product, and language.
2. Searching: embed each rewritten query with the same model, run nearest-neighbor search filtered by metadata, and run keyword search (BM25) in parallel to catch exact terms like order numbers.
3. Merging: combine the result lists (for example, with reciprocal rank fusion), remove duplicates, and send the top candidates to the reranker.

A voice-specific speedup is starting retrieval on partial transcripts before the caller finishes.

“reranking typically incurs the largest latency cost, as it must process tens to hundreds of candidate documents”

Are they using an LLM for reranking? If yes, why use that instead of a standard cross-encoder or late interaction Models that are more typical in industry?

From the post: The model is never named. Two details are telling. The post says the latency gains came without replacing the underlying model, and the starting point was a list-wise reranker, which outputs a ranked list. The attention-mask diagram also labels the shared prefix “instruction + query” and draws triangular attention inside each block.

Inference (background knowledge): Very likely yes. Producing a ranked list requires a text-generating model, and triangular (causal) masks are how decoder-only LLMs attend; a BERT-style encoder attends in both directions and would show full squares. The setup matches rerankers like Qwen3-Reranker, which put an instruction and query before each document and score it by the probability of answering “yes.” A point-wise LLM reranker is itself a cross-encoder, since it reads the query and document together. The real contrast is a small bidirectional encoder versus a large causal LLM.

Likely reasons for the LLM:

1. Quality: LLM rerankers generally beat BERT-size cross-encoders on public benchmarks.
2. Control: the instruction can carry customer-specific relevance rules.
3. Continuity: the same model already powered their list-wise setup.

Late-interaction models like ColBERT precompute document token embeddings, so they are faster, but they usually rank less accurately than cross-encoders.

“which drives up latency due to the quadratic time complexity of transformer attention”

Even with local attention/other attention optimizations?

Inference (background knowledge): Mostly yes. Attention optimizations fall into two groups, and neither removes the problem for a typical reranker:

1. Exact attention with faster kernels, such as FlashAttention, still computes every token-to-token score. Memory grows only linearly with length and speed improves by a constant factor, but compute stays quadratic.
2. Restricted attention, such as local (sliding-window) or sparse attention, does cut compute to roughly length times window size. But the model has to be trained that way, and list-wise ranking needs long-range comparisons between documents, which a local window undermines.

The post misses two nuances. The quadratic term only dominates once the sequence is longer than about 12 times the model’s hidden size (Kaplan et al., 2020): roughly 12,000 tokens for a hidden size of 1,024, or 49,000 for 4,096. Below that, the feed-forward layers cost more. List-wise reranking also has to generate the ranked list token by token; FIRST (Reddy et al., 2024) reads the ranking from the first output token’s probabilities instead and reports 50% faster inference.

“We recently moved from list-wise to point-wise reranking at Decagon and saw a significant reduction in reranking latency as a result.“

Quality impact?

From the post: No quality results are reported for this switch. The only quality-related claims concern the later packing step: masking keeps point-wise scoring semantics intact and, per the closing section, preserved scoring quality. Neither claim comes with numbers.

Inference (background knowledge): The effect could go either way.

1. Reasons it could hurt: a point-wise scorer never sees the other candidates, so it can’t judge relevance relative to them or notice near-duplicates. In zero-shot prompting (no reranking-specific training), the Setwise study (Zhuang et al., SIGIR 2024) found point-wise methods very efficient but weak on ranking quality.
2. Reasons it could be neutral or help: models trained specifically for point-wise scoring, such as Qwen3-Reranker or RankLLaMA, are strong. Point-wise scoring also avoids list-wise failure modes, like sensitivity to the order in which candidates are listed, and it gives each document an absolute score you can threshold.

The way to find out is to measure ranking quality offline (NDCG@10 and recall@k) on labeled support queries, then confirm downstream effects, such as answer accuracy or resolution rate, in an A/B test.

“The overall reranking step is gated on the slowest individual call, and the overhead of many sequential requests adds up quickly.”

What overhead? And why are the requests sequential as opposed to parallelized?

Inference (background knowledge): The sentence mixes two pictures. A step that waits for its slowest call implies requests running in parallel, while overhead that adds up across sequential requests implies requests running one at a time.

The overhead is most likely the fixed cost each request pays regardless of document content:

1. Transport: serializing the request, the network round trip, and HTTP or gRPC handling.
2. Scheduling: tokenization, waiting in the server’s queue, and batch formation.
3. Compute: fixed costs per forward pass (launching GPU kernels, Python dispatch), plus reprocessing the identical instruction and query for every document.

Why sequential: sending requests in parallel from the client doesn’t mean the GPU runs them in parallel. If a model replica can only fit a few sequences per forward pass, the extra requests wait in a queue and effectively run one after another. Rate limits or concurrency caps on the endpoint have the same effect. Adding replicas helps, but it costs GPUs and still leaves the slowest-call problem described in question 10.

“Here's how it works: we concatenate multiple (query, document) pairs into a single packed sequence, separated by a delimiter token. We then modify the attention mask so that tokens within each pair attend only to each other, while attention to tokens from other pairs is masked out entirely. This produces a block-diagonal attention pattern (see diagram below) — each document is scored in isolation, but we amortize the overhead of a model call across many pairs at once.“

Walk me through a brief illustrative example to make this more concrete. Include relevant math.

From the post: The text describes packing (query, document) pairs, but the diagram shows one shared prefix (instruction plus query) followed by delimiter-separated documents, each attending only to the prefix and its own earlier tokens. The example follows the diagram.

Inference (worked example): Take a prefix (q1, q2) and documents A = (a1, a2) and B = (b1, b2), without delimiters. The mask (1 = may attend, 0 = blocked) is:

q1 q2 a1 a2 b1 b2
q1 [ 1 0 0 0 0 0 ]
q2 [ 1 1 0 0 0 0 ]
a1 [ 1 1 1 0 0 0 ]
a2 [ 1 1 1 1 0 0 ]
b1 [ 1 1 0 0 1 0 ]
b2 [ 1 1 0 0 1 1 ]

The only change from a normal causal mask is the four zeros blocking B from A.

Attention is softmax(QKᵀ/√d + M)·V, where M is 0 for allowed pairs and −∞ for blocked ones. Since e^(−∞) = 0, token b1 spreads its weight only over q1, q2, and itself, and A gets exactly zero. So B’s hidden states match a standalone run on q1 q2 b1 b2, as long as b1 and b2 reuse positions 2 and 3 instead of 4 and 5.

Each score is then read at the document’s last token, typically as p(yes) = e^(z_yes) / (e^(z_yes) + e^(z_no)), where the z values are output logits. The post doesn’t specify the readout.

Cost: two separate calls process 8 tokens; the packed call processes 6, because the prefix runs once.

Why do packed scores exactly equal standalone scores?

The claim is that every document token has the same hidden state, at every layer, as the matching token in a standalone run of [prefix, document]. The argument goes layer by layer (induction):

1. Layer 0. Embeddings depend only on the token and its position. Position encodings are handled separately.
2. Each next layer. Norms and MLPs act on each token separately, so they keep matching states matched. Attention at a document token mixes only the tokens it's allowed to see. The prefix tokens match because the prefix only ever sees itself. The earlier tokens of the same document match by the previous layer's result. Same inputs through the same weights give the same output.
3. Final layer. The final logits match, and so does the yes-or-no score.

What would happen if they retained the batch structure but didn’t do the attention masking in the model call?

From the post: This case isn’t tested. The post justifies packing on the grounds that pairs don’t need to interact to get correct scores, and says masking keeps point-wise scoring intact.

Inference (background knowledge): That framing implies removing the mask would break point-wise scoring. With an ordinary causal mask instead, the effects split into quality and latency.

1. Quality would degrade in two ways. Scores would depend on neighbors: each document’s tokens would attend to every earlier document in its pack, so the same document could score differently depending on what it’s packed with and in what order. At their batch size of 2, the second document in every pack, half of all documents, would be affected. And inputs wouldn’t resemble training data: a point-wise reranker learns from prompts containing one document, so prompts containing several could produce poorly calibrated scores, and later documents would sit farther from the query, adding position bias.
2. Latency would stay about the same or improve slightly, since the work per forward pass is similar and a standard causal mask can use fast fused kernels like FlashAttention.

In short, you’d keep the speed but lose the guarantee that packed scores equal unpacked scores.

Is attention masking generally available through model provider APIs or would you need to host an open-source model yourself for this to work?

Inference (background knowledge): You would need to self-host, or at least run your own inference code. It depends on the type of API:

1. General LLM APIs (OpenAI, Anthropic, Google) accept text, not attention masks or position IDs, so this technique isn’t possible through them.
2. Hosted rerank APIs (for example, Cohere Rerank or Voyage) accept one query plus a list of documents in a single request. The provider can batch internally, which already removes most per-request overhead, but you can’t control the mask or batch size.
3. Self-hosted open-weight models, on your own GPUs or a dedicated deployment that runs your code, are the only option for custom masks. Hugging Face Transformers accepts custom 4D masks together with position IDs, and PyTorch’s FlexAttention can express shared-prefix and per-document masks while skipping fully masked blocks. Serving engines like vLLM support rerankers such as Qwen3-Reranker but don’t expose a per-request mask, so you’d modify the engine or write your own inference loop.

“With this packing approach in place, the remaining challenge is straightforward: find the optimize batch size that minimizes end-to-end latency for your system.“

Quality?

From the post: Batch size is treated purely as a latency knob, with no quality results by batch size. The post argues packing is valid because pairs don’t interact, and it says masking preserved quality, without numbers.

Inference (background knowledge): In theory, with an exact mask and correct position IDs, each document gets the identical score at any batch size, so quality shouldn’t change. In practice, three things can break that:

1. Position IDs: if they aren’t reset for each document, later documents sit at different distances from the query, and scores shift with position in the pack.
2. Numerical drift: different sequence shapes can make GPU kernels add numbers in a different order, which changes logits slightly and can flip near-ties (Thinking Machines, “Defeating Nondeterminism in LLM Inference,” 2025).
3. Truncation: a maximum packed length can cut off long documents in larger packs.

The check they didn’t report is simple: score the same queries unpacked and packed at each batch size, then compare the largest score difference, rank agreement (Kendall’s tau), and NDCG@10.

“where the overall step is gated on the slowest call”

What does this mean?

From the post: The phrase is part of the case against small packs: with too few pairs per batch you send many requests, and the step waits on the slowest one.

Inference (background knowledge): The reranker sends many requests at once, and the pipeline can’t move on until every one returns. So the step takes as long as the slowest request, not the average one. “Gated” means that one slow call holds the gate shut for everything downstream.

For example, if 20 calls return in 40 milliseconds and a 21st takes 250 milliseconds, the step takes 250 milliseconds.

More calls make a slow one more likely. If each call independently has a 1% chance of being slow, the chance that at least one of m calls is slow is 1 − 0.99^m. That is about 10% for 10 calls and 63% for 100 calls. Dean and Barroso describe this effect in “The Tail at Scale” (2013). Packing more documents into each call reduces m, which is the benefit side of the batch-size trade-off.

“Using too many causes the packed sequence to grow long enough that per-call processing time starts to climb.”

What does this mean?

From the post: This is the other side of the trade-off: more pairs per pack means fewer calls, but longer and slower ones.

Inference (background knowledge): A pack of b documents of L tokens each, after a P-token prefix, is about P + b × L tokens long, and longer inputs take longer to process. “Starts to climb” fits how GPUs behave. At short lengths the GPU has idle parallel capacity, so extra tokens are nearly free; once that capacity is used up, time grows with length, and attention grows with the square of length.

Custom masks make this worse. Unless you use a sparsity-aware kernel such as FlexAttention, attention computes a score for every token pair in the pack and only then discards the masked ones.

Example with P = 100 and L = 300: two documents make a 700-token pack with 490,000 attention scores, while five documents make a 1,600-token pack with 2,560,000. That’s 2.5 times the documents for 5.2 times the attention work, and only about 15% of those scores are allowed by the mask.

Bigger packs are also more likely to contain one very long document, which then becomes the slowest call.

Is there any good theoretical explanation for why a batch size of 2 yields the best latency results?

Inference (background knowledge): Nothing in theory singles out 2; it’s an empirical minimum. A simple cost model does explain the U-shape if total latency tracks total compute (say, calls queuing on a busy GPU). With N documents of L tokens in packs of b:

Latency ≈ N × (F / b + k × L² × b) + constant

F is each call’s fixed cost (overhead plus the shared prefix), and k × L² × b is attention cost that grows as packs lengthen. The first term falls with b and the second rises, so latency is lowest at b* = √(F / (k × L²)). Longer documents push that optimum down.

Batch size 2 beats 1 and 3 when F is 2 to 6 times k × L², meaning each added document is costly relative to a call’s fixed cost. That fits dense custom masks; a sparsity-aware kernel would shrink k and raise the optimum.

With one measurement per point, 1 versus 2 may be noise.

The post seems to focus primarily on the benefits of this approach. Are there any downsides that they haven’t covered?

Inference (background knowledge): The uncovered downsides fall into four groups:

1. Quality: point-wise scoring can’t see other candidates, so it can’t penalize near-duplicates or reward coverage. Neither change comes with quality metrics, and position-ID handling isn’t described, though getting it wrong silently shifts scores.
2. Efficiency ceiling: an optimum of 2 still means making half as many calls as there are documents. If dense custom masks are the cause, speed is being left on the table, and the optimum has to be re-tuned whenever the model, hardware, or document lengths change.
3. Engineering cost: custom masks require self-hosting and custom inference code, can break during library upgrades, and may bypass serving-engine features like continuous batching and prefix caching.
4. Evidence: without latency values, percentiles, or error bars, the size of the win is unknown. Batch size 5 is slower than the list-wise baseline (per the plot), which goes unexplained, and tail latency (p95 and p99), which matters most for voice, isn’t reported.

How would you improve this system if you were building it today?

Inference (my recommendations): I’d make four changes, roughly in order of payoff:

1. Send fewer documents to the LLM. Add a cheap first-pass scorer, such as a small cross-encoder or a ColBERT-style model, to cut the tens-to-hundreds of candidates to about 20. Also cache scores for repeated query-document pairs, which are common in support.
2. Make each pass cheaper. Compute the instruction-plus-query prefix once and reuse its key-value cache (the attention keys and values already computed for those tokens) for every document, and use a sparsity-aware kernel like FlexAttention so cost tracks each document’s own length. That should push the optimal batch size well above 2.
3. Shape the calls and their timing. Pack by token budget rather than a fixed document count, so no call is much longer than the rest, and start retrieval on partial transcripts while the caller is still speaking.
4. Measure it. Track p50, p95, and p99 reranking latency, and gate every change on packed-versus-unpacked score equivalence plus NDCG@10 on labeled support queries.

“When a company receives a contract from a counterparty, someone in their legal department compares it, clause by clause, against their company’s playbook”

How long are company playbooks typically? Are they only text or can they contain other kinds of data too?

[Post] Playbooks "can be hundreds of pages long." A single rule has five parts: the standard position (what the company wants), acceptable deviations (what it will settle for), unacceptable deviations (dealbreakers), guidance on how to apply the rule, and whether the clause is allowed to be missing entirely. Their test set covers "hundreds of provisions."

[Search] The real range is wide. Practitioner guides suggest a first playbook is about an afternoon's work covering the ten or twenty terms that account for most of the redlines a team actually sends. A mature playbook for a single contract type is more like thirty to eighty terms across a few dozen pages. Harvey's "hundreds of pages" describes a large company maintaining playbooks for many contract types at once.

[Inferred] They aren't pure prose. A real playbook usually contains a ladder of fallback positions — ask for this, settle for that, walk away at this — along with the exact wording to paste in for each fallback, dollar thresholds that determine which position applies, and a table of who has to approve what. Harvey's five-field structure suggests they store playbooks as structured records rather than as documents to be read. That matters later, because it's what lets them hand one rule to one agent.

“Playbook rules are conceptually simple, and include the following elements”

How generic are these rules? Is there a single playbook for all kinds of contracts?

[Post] There isn't one playbook for everything. The test data "pairs contracts with playbooks for various contract types," so a playbook belongs to a type. Rules are written generically and then interpreted per deal, which is what the guidance field exists for: "how to interpret and apply the rule given specific deal context."

[Inferred] The normal setup is one playbook per contract type — NDAs, master services agreements, data processing agreements, order forms, vendor agreements. They have to be separate because the same term means different things in each. A liability cap that's perfectly sensible in an NDA would be unacceptable in a services agreement worth millions.

Some terms travel well across types: governing law, how notices get delivered, whether the contract can be assigned to an acquirer. Teams often keep a shared set of those and layer type-specific rules on top.

The practical point is that a rule as written is deliberately incomplete. "We prefer mutual indemnification" doesn't tell you what to do on a particular deal. What fills in the blank is the deal context — whose paper the contract is on, which party you represent, how far into the negotiation you are. That's exactly the information the old system kept losing between steps, and that the new one broadcasts to every agent.

“Rules interact across a document”

Is it possible for rules to be contradictory? If so, what happens?

[Post] Yes, and the post is direct about it. In the old system, "two rules could propose contradictory edits to the same sentence, and there was no rewriting for those edge cases" — the user sorted it out by hand. In the new system, when two rules try to change the same text, the merge step flags it and hands it to the lead agent, which writes a single edit "that satisfies all conflicting rules."

[Inferred] Two different situations are being treated as one here, and only the easier one is actually solved.

The common case is two rules that both want to change the same sentence but aren't really opposed. One rule wants a notice period extended from 15 to 30 days; another wants notices delivered by email as well as post. Both can be satisfied by one carefully written sentence, and that's what the lead agent is doing.

The rarer case is two rules that genuinely cannot both hold — one requires uncapped indemnity for data breaches, another caps all liability at the contract value. No wording satisfies both. The post never addresses this. It's really a defect in the playbook rather than in the contract, and the right response is to stop and tell a human, which fits Harvey's own point that "some decisions cannot be delegated." But nothing they describe would detect it, so the lead agent would presumably draft something plausible and move on.

“it's a quality bar that out-of-the-box AI models and agents miss”

This feels fixable by having “The best edit is usually the smallest one” included as instructions in the system prompt. Would that not work?

[Inferred] Adding that line helps, and they almost certainly had something like it already. It doesn't fix the problem, for two reasons.

The first is that telling a model to make a small edit doesn't help it work out which edit is the small one. To change only the three words that matter, you have to know which three words create the legal problem. That's a hard reading task, and an instruction to "be minimal" gives the model no extra help with it.

The second is that the old setup made minimal edits nearly impossible to produce. It asked the model to rewrite the whole clause, then compared the new version to the original afterward and called the difference a redline. Models trained as helpful assistants tidy as they write — they smooth grammar, reorder for clarity, standardize phrasing. Every one of those shows up in that comparison as a proposed change, even though none of them was the point. The instruction to stay minimal was fighting the model's default writing behavior, and losing.

The new system asks for something different: not "rewrite this clause" but "change these specific words in this specific place." That's a change to what the model produces, not a change in how politely you ask for it.

“still produce edits in a timely fashion”

What are the realistic latency constraints/expectations here? Contract review feels like it should prioritize quality over latency.

[Post] 2.6 minutes before, 3.8 after, which Harvey calls "a reasonable tradeoff to make as we are optimizing for the quality of output." A human review takes 20 minutes to five hours. They also added streaming so users "begin reviewing outputs within seconds."

[Inferred] You're right that quality should win, and Harvey clearly agrees — they accepted a 47% slowdown to get the quality gain. But latency still matters, for reasons that have nothing to do with impatience.

The one that matters most is the first ten seconds. If a lawyer uploads a contract and sees a blank screen, they assume it's broken, or they go do something else and lose the thread. Streaming fixes that without making the review any faster overall, which is exactly why they built it.

The second is that nobody runs the review once. A reviewer changes a deal instruction, or decides to push for the standard position instead of the fallback, and runs it again. Five of those at four minutes each is twenty minutes, which puts you back in the range of just doing the review yourself. That's why the system saves each agent's work and reuses it rather than starting from scratch.

So the constraint isn't a hard number. It's roughly: something useful on screen within seconds, and the whole thing done in less time than it takes to get a coffee.

“For each batch of rules, one model call asked: Does the contract match the rule's standard position?”

Are multiple rules being reviewed simultaneously or one at a time?

[Post] Multiple at once, in batches. One model call took a group of rules and asked, for all of them, whether the contract matched the standard position. Rules that didn't match fell through to a second call about acceptable deviations, and those that failed again fell through to a third about unacceptable deviations.

[Inferred] Batching was there to save money. The contract is the expensive part of the prompt — you pay to send the same fifty pages every time — so asking about twenty rules in one call is far cheaper per rule than twenty separate calls.

The cost is exactly what Harvey complains about. All twenty rules share one call's worth of thinking. A rule that requires tracing a definition across four sections gets the same shallow treatment as a rule that a single sentence settles. In their words: "a single LLM call with limited budget won't be able to think through and give accurate output to solve complex documents and playbooks."

The new design flips this. Every rule gets its own agent, and they run at the same time instead of being crammed into one call. The savings that batching used to provide come back a different way: because every agent reads the same contract, the model provider charges full price only for the first one and a fraction of that for the rest. Same economics, without making rules compete for attention.

“Once classifications were done, a separate retrieval step then located supporting text in the document”

How would this work?

[Post] The post doesn't say. We only get that after classification, "a separate retrieval step then located supporting text in the document," feeding a final call that wrote the summary.

[Inferred] The job is to find the exact piece of contract that justifies the classification, so the interface can highlight it and the lawyer can check the reasoning. There are three normal ways to do that.

The simplest is to have the model quote the relevant text, then search the document for that quote to find where it lives. Cheap, and it works right up until the model paraphrases slightly or invents the quote, at which point you find nothing.

The second is semantic search: split the contract into chunks, convert both the chunks and the rule into numerical representations that put similar meanings close together, and return the nearest chunks. This survives paraphrasing but tends to give you a rough region rather than the exact sentence.

The third, and most reliable, is to number every clause up front and have the model return clause numbers instead of text. That's what the new system does everywhere — "we gave each component of the document a unique identifier."

Why it was a separate step at all: the classification call probably didn't have the whole contract in front of it, so the location had to be recovered afterward. And doing it afterward is precisely how you get the failure they describe, where a suggestion lands on the wrong part of the document.

“We diffed the revision against the original, and that diff became the tracked change.”

What does tracked change mean here?

[Inferred] A tracked change is Word's redline — text marked as inserted or deleted, attributed to an author, each one individually acceptable or rejectable by whoever reads it next. That's what the lawyer sees and what goes back to the other side.

What's interesting is that the phrase means two different things in the two systems, and that difference is the whole design.

In the old system a tracked change was an output. The model wrote a new version of the clause, code compared it to the original, and whatever fell out of that comparison became the redline. Nobody could tell which part of the change was the substantive fix and which part was the model tidying grammar along the way.

In the new system a tracked change is what the agent creates. An agent doesn't hand back replacement text; it hands back "in clause 8.2, replace these words with those words, because of rule 14." Once edits are objects like that, you can tell which rule caused which change, detect that two agents want to touch the same words, and later learn from which edits the lawyer kept. None of that is possible when the edit is just a comparison you ran afterward.

“The core dataset pairs contracts with playbooks for various contract types across hundreds of provisions.”

Synthetic or human generated?

[Post] Not stated. Only that they "built our own evaluation suite with our in-house legal team," that the dataset "pairs contracts with playbooks for various contract types across hundreds of provisions," and that the scoring rubrics were written with lawyers.

[Inferred] Almost certainly a mix, with the pieces coming from different places.

The contracts are most likely real ones from public sources. Companies file their material contracts with the SEC, and those filings are the standard source for anyone building legal AI evaluations. Customer contracts usually can't be used, because their confidentiality terms don't permit it.

The playbooks are most likely written by Harvey's own lawyers to pair with those contracts. A customer's real playbook is a competitive asset and generally isn't available for this.

The rubrics are explicitly human — "after sourcing the rubrics with lawyers." The grading itself is done by models.

This matters more than it sounds. If the same organization wrote the playbooks and designed the system, those playbooks will be cleanly structured and unambiguous in exactly the ways the system handles well. Real customer playbooks are messier: inconsistent, partly self-contradictory, written by different people over several years. The improvements are real, but they were measured on the friendlier version of the problem, and the post gives you no way to tell how much friendlier.

“Risk classification: Ability to identify and categorize issues by their level of risk.“

Accuracy, precision, recall?

[Post] Not stated. One number, 59% rising to 77%, and a description of the task as "a classification problem which is a straightforward task."

[Inferred] A single percentage on a task with three possible labels is almost certainly plain accuracy — the fraction of provisions where the system's label matched the lawyer's. Three things make that a weak way to report this.

Most clauses in most contracts are fine. If 80% of provisions match the standard position, a system that labels everything "standard" scores 80% and catches nothing. Accuracy hides that entirely.

The two kinds of mistake also cost wildly different amounts. Flagging something that was actually fine wastes a few minutes of a lawyer's time. Missing a dealbreaker can cost real money. Harvey knows this — their own list of old failures names both "over-redline" and "miss flags" — so they almost certainly track the two separately internally. They just didn't publish it.

And the three labels are ordered: standard, acceptable, unacceptable. Calling an unacceptable clause "standard" is far worse than calling it "acceptable," because in the first case nobody looks at it again. Accuracy scores both as equally wrong.

The number I'd want is how often the system catches the unacceptable ones, reported on its own.

“Redline quality: A list of considerations that a revision by a competent lawyer would have, both in substance and style. These rubrics are defined for each example.“

Binary or score? Weighting of different dimensions?

[Inferred] "A list of considerations… defined for each example" strongly suggests a checklist per example, where each item is a yes-or-no question the grader answers, and the score is the fraction of items passed.

This is the standard way to build these, and it exists to avoid a specific problem. If you ask a model to rate an edit from 1 to 10, you get different answers on different runs and no way to explain the difference. If you ask "does this edit change only the words that needed changing? yes or no," you get something stable enough to compare across versions of the system.

Weighting is then probably implicit rather than declared: a dimension gets more weight by having more checklist items about it. So a rubric with four questions about legal correctness and one about placement effectively weights correctness four to one, without anyone writing down a weight.

The design I'd expect but that Harvey doesn't claim is a hard failure condition on legal correctness. An edit that is beautifully minimal, perfectly placed, and legally wrong should score zero, not two-thirds. Whether they do that changes what 87% actually means, and there's no way to tell from the post.

“We then used a committee of three frontier models to score independently and aggregate the votes.“

Which models?

[Post] Not named.

[Search] Harvey deliberately runs on models from three companies — OpenAI, Anthropic, and Google DeepMind — and their published lineup names Claude Opus 4.6 (for "agentic reasoning and deep multi-step legal analysis"), GPT-5.2, Sonnet 4.6 (for lower latency), and Gemini 3 Flash (for speed and throughput). The obvious reading of "three frontier models" is one top-end model from each of those three companies. (Source: harvey.ai/blog/why-harvey-is-multi-model-by-design)

[Inferred] The part that actually matters isn't which three, it's that they come from three different companies rather than three sizes of the same model.

Models from the same family have been trained on similar data with similar preferences, so they tend to make the same mistakes and like the same writing. Three of them voting doesn't give you three opinions, it gives you one opinion repeated. Spreading across companies gets you genuinely different judges.

That's especially important here because the system being graded is itself built on one of those companies' models, and models rate their own family's output more generously. Mixing providers reduces that, though it doesn't remove it — one of the three judges is still likely from the same family as the system under test. More on this in Q16.

“equipped with tools to search for various key parts of a document”

What kinds of tools? How do they work?

[Post] Three kinds are named: searching the document, looking up rules and other information, and editing the document. The thing that makes all of them work is that "we gave each component of the document a unique identifier," which "allows agents to cite and edit a specific element unambiguously."

[Inferred] The likely tool set breaks into four groups.

Reading and navigating: fetch a specific clause by its number, get the table of contents, read a whole section, follow a cross-reference or jump to where a defined term is defined.

Searching: keyword or meaning-based search over the contract that returns clause numbers rather than raw text, so every result is something the agent can point at precisely.

Editing: propose a change to a specific piece of text in a specific clause, delete something, insert something after a given clause. Each one gets recorded as a redline tagged with the rule that caused it.

Reporting: file the memo at the end with the classification, the position chosen, and the reasoning.

The reason the identifiers matter so much is worth spelling out. Without them, the only way a model can tell you where it means is by quoting the text. That breaks in three routine ways: contracts repeat near-identical language in several places, the model paraphrases slightly when quoting, and sometimes it invents a quote outright. A clause number either exists or it doesn't, and it points at exactly one thing.

“but was very high latency”

What's a reasonable guess for the latency difference between the single-agent and the multi-agent systems?

[Inferred] You can reason it out from the structure. In the multi-agent system, all the per-rule work happens at once, so the total time is roughly: set up, wait for the slowest single rule, merge, final check. It doesn't matter much whether there are 20 rules or 60.

A single agent does the same work one rule after another. With 30 to 60 rules and a couple of minutes of work per rule, you land somewhere between half an hour and two hours — call it ten to thirty times slower. That's not "a bit slow," that's back inside the range of a human doing the review by hand, which is what makes it unacceptable rather than merely annoying.

Two things make it worse than the simple multiplication suggests. The single agent's context grows with every rule it handles, so it gets slower and worse as it goes — the post's own point that quality degrades as you add tokens. And it's one long chain, so a single stuck step stalls everything, with no way to retry just the rule that broke.

Any concerns about using LLM-based systems to both solve and evaluate?

[Post] The risk isn't discussed. What they do describe as safeguards: the rubrics were written by lawyers rather than models, the graders score against fixed rubrics rather than giving free-form opinions, and three models vote independently.

[Inferred] There are real concerns, and they enter at different points.

Models rate their own family's output more generously than other models' output. Using three different providers reduces this but doesn't remove it, since the system under test is presumably running on one of those three families.

More worrying is what all frontier models have in common. They've been trained to prefer writing that is fluent, thorough, and complete. That's precisely the instinct that "lightest touch" is fighting. A grader with that instinct will quietly reward an edit that rewrites the whole clause nicely over one that changes three words, which means the redline score can climb while deals don't actually move any faster.

There's also the plain fact that 53% to 87% was measured against the rubric the team was optimizing toward. Any metric you tune against stops being an independent measurement of the thing you care about.

And the post reports nothing about whether the graders are any good — no figure for how often the model graders agree with lawyers, and none for how often the three models agree with each other. Without those, you can't tell how much of that 34-point gain is real.

What I'd want: lawyers who didn't write the rubrics grading a held-out set blind, plus the rate at which real customers accept the redlines in production.

“lead agent in charge of distributing work between parallel agents”

What does the lead agent tell the parallel agents to do?

[Post] It "spawns a team of subagents to review each playbook rule." Each one receives the shared deal context — "which party they represent, whether the contract is on your template or theirs, how strict or permissive this negotiation should be, and any deal-specific instructions" — plus any precedent documents the user attached. The subagent then "reads its rule, reads or searches the contract, decides whether the contract complies with the playbook, chooses which position to pursue (standard or a specific fallback), and then drafts a list of suggested edits," finishing with a short memo.

[Inferred] So what actually gets handed over is roughly four things: the one rule it owns, in its five structured fields; the shared deal context; a pointer to the contract — its own branch and the clause numbering — rather than the contract text pasted in, which is what makes the caching work; and a description of what to hand back.

What's notably absent is any information about what the other rules found. The subagents run at the same time and are blind to each other. That's a deliberate choice, and it's the reason reconciliation exists at all: the lead agent's job is to divide the work up front and merge it at the end, not to coordinate anyone in between.

“lead agent reconciles the results to ensure final output is of good quality”

What does reconciliation look like concretely? How does it work?

[Post] It happens in two stages. The merge "merges branches the same way a version-control system would: non-conflicting edits apply cleanly to the final doc, and collisions of two rules touching the same text are escalated for the lead agent to resolve." Then the lead agent resolves those conflicts "the way a senior lawyer would by drafting edits that satisfies all conflicting rules," and "takes a final pass to validate the quality of the review as a whole. This ensures that every rule gets classified and has corresponding edits when needed."

[Inferred] The important thing is how little of this involves a model.

Most of it is ordinary code. Each agent's edits are recorded against specific clause numbers and specific character ranges within them. To merge, you check whether any two agents touched overlapping text. If they didn't — which is the case for the large majority of rules — you apply everything and you're done. No model call, no cost, no risk.

Only the overlaps go to the model. For each one, the lead agent sees the original text, the two competing edits, and the memos from both agents explaining what they were trying to achieve, and writes one edit that does both jobs. This is what those memos are for, and it's why they need to be short: they're the only channel through which one agent's reasoning reaches the lead agent.

Then the final pass checks coverage — every rule has a verdict, and every flagged rule has edits — and reads the merged document as a whole.

The gap is that conflicts are found by checking whether two edits overlap in the text. That catches collisions but not everything that can go wrong. See Q20.

“running up to dozens in parallel”

How does it determine how many need to be run? Based on the number of rules?

[Post] Yes — one agent per rule, always: "every rule in a playbook fans out a subagent." The ceiling on how many run at once comes from infrastructure rather than from any planning: "we added concurrency limits and exponential backoffs to avoid overwhelming shared model infrastructure."

[Inferred] So there are two different numbers. The number of agents needed equals the number of rules, which could be in the hundreds — their own test set covers "hundreds of provisions." The number running at any given moment is capped, with the rest waiting in a queue. "Dozens" is describing the cap, not the size of the job.

This is a deliberately simple design, and it's worth noticing what it gives up. There's no step that decides some rules aren't worth an agent, and no grouping of related rules. If a playbook has four separate rules about limitation of liability, four agents each read the same section, each build their own understanding of it, and then three of the four collide with each other and have to be cleaned up afterward.

What they get in exchange is that the behavior is completely predictable, and caching makes all that duplicated reading nearly free. That's a reasonable trade, but it's a trade — I'd revisit it, see Q37.

How does the multi-agent architecture deal with concerns about rules interacting across a document? Especially since they say “[sub-agents] often would need to apply edits to the same paragraph or sentence while addressing different rules”

[Post] Three mechanisms, doing three different jobs.

Agents can't corrupt each other's work, because each one edits its own copy of the document. Without that, one agent's changes would silently overwrite another's — the post's exact concern: "previous work done by one agent would get overwritten by another agent."

Overlapping edits get noticed rather than silently resolved, because the merge step surfaces them as conflicts.

And conflicts get resolved by the lead agent writing a single edit that satisfies both rules, followed by a read of the whole document.

Separately, a single rule that touches four sections is handled inside one agent, since the agent can search the contract and follow references. That's the direct fix for the old system's problem that "edits were scoped by clause."

[Inferred] Sorting out what this does and doesn't cover:

It genuinely solves agents overwriting each other, and it solves an agent being unable to see past the one clause it was handed.

It partly solves two rules wanting to change the same sentence. Whether the combined edit is any good depends entirely on the lead agent, which is working from two short memos rather than everything the two agents actually figured out.

It doesn't solve edits that conflict in meaning without overlapping in text. Suppose one agent narrows the definition of "Confidential Information" in section 1, and a clause in section 11 was relying on the broad version. Those two edits don't touch the same words, so nothing flags them. The same goes for deleting a clause that something else cross-references. Only the final read-through could catch it, and that's one model call over a long document — not a systematic check.

This is the part I'd push hardest on in an interview. The mechanics are genuinely good; the handling of meaning-level consistency is thin.

“we gave each component of the document a unique identifier”

What does this mean concretely?

[Inferred] Concretely: when the contract is loaded, it gets broken into its structural pieces — sections, paragraphs, numbered clauses, list items, table cells, definitions, exhibits — and each piece is given a label that doesn't change, something like sec-8.2 or p-0147.

From then on, everything works in terms of those labels. The document is shown to the model with them attached, so it reads something like "[p-0147] Neither party shall assign this Agreement...". Tools take a label as their argument: read this section, propose a change to that paragraph. Citations are labels, so highlighting in the interface is exact. And checking whether two agents conflict becomes a matter of comparing lists of labels.

The reason this is worth doing is that the alternative — having the model quote text to say where it means — fails in three routine ways. Contracts repeat near-identical language, so "the Notices provision" might match three places. Models paraphrase slightly when quoting. And as edits accumulate, the original text an agent quoted stops matching what's in the document.

One thing that takes care: labels have to stay stable when the document changes. Inserting a new clause between 8.2 and 8.3 needs a fresh label rather than renumbering everything after it, which is why the internal identifiers are kept separate from the clause numbers a reader sees.

Does the orchestrator look at sub-agent output one at a time or all simultaneously in the same prompt?

[Post] Never stated. What we know: each agent finishes by filing "a short memo"; results are streamed as each agent completes; merging is described as working like version control, with only collisions going to the lead agent; and the lead agent "takes a final pass to validate the quality of the review as a whole."

[Inferred] It's almost certainly a mix of three things, and the word "short" in "short memo" is the clue.

Merging isn't a model call at all. It's code comparing which clauses each agent touched.

Conflict resolution happens one conflict at a time. Each one gets its own small model call containing just the original text, the two competing edits, and the two relevant memos. This keeps each call cheap and focused, and lets several conflicts be resolved at once.

The final validation has to see everything, because checking that every rule got handled and that the document reads consistently isn't something you can do in pieces. So the lead agent probably does see all the memos together at that point — and that is exactly why they have to be short. If each agent handed back its full working transcript, sixty of them wouldn't fit in one call.

That's the general shape of this kind of system: the agents doing the reading burn through enormous amounts of text, and only a compressed summary travels back up to the one that has to see the whole picture.

“Every agent in the hierarchy sees this context”

What does “see” mean here?

[Inferred] "Sees" almost certainly means it's written into every agent's prompt, rather than being something an agent has to go and ask for. Two things point that way: it's small and structured, just a handful of fields, and putting shared text at the start of every prompt is precisely what their caching optimization requires.

The likely exception is the precedent documents, which can be large. Those are more plausibly something an agent searches when it needs them than something pasted into sixty prompts.

Why this matters: it's the fix for one of the old system's most embarrassing failures, where reviews would "miss whose paper the contract was on." In a pipeline, that fact could be known at the classification step and simply absent by the time the redline step ran, because each step only got what the previous step passed along. Putting it in front of every agent makes it impossible to lose.

The cost is that you're paying for those tokens on every one of dozens of calls. That's exactly why prefix caching was worth a section of its own.

“persists with the review”

Any concerns about context window dilution?

[Inferred] The concern is right in general, but this design mostly avoids it, for three reasons.

Saved isn't the same as loaded. That state lives in a database attached to the review, not in a conversation that keeps growing. It gets pulled out selectively, which is what "we reuse the relevant agents" is telling you.

Each agent's world stays small. A subagent only ever sees one rule, the contract, and the deal context. If you ask a follow-up about rule 12, rule 12's agent comes back with rule 12's state — not the accumulated state of the other fifty-nine.

And what's saved is summaries, not transcripts. Classifications, chosen positions, edit summaries, reasoning. Not every search the agent ran and every clause it read along the way.

Where it does bite is the lead agent's final pass, which necessarily grows with the number of rules, and a long session where someone keeps iterating on the same review. The post doesn't say whether either of those gets trimmed.

“we reuse the relevant agents”

How is relevance determined?

[Inferred] Relevance is probably worked out structurally rather than by asking a model, and each of the three triggers has an obvious mechanical answer.

Switching the position on a rule is the easy one. You know which rule the user clicked on, so you re-run that rule's agent and nothing else.

A follow-up question sounds harder, but usually isn't, because the question is attached to a specific flagged issue in the interface. The user clicks on the redline and types their question, so you know which rule they mean without having to interpret the text.

The third case is the interesting one. When the user accepts some suggestions, the document changes, and any rule whose edits were aimed at the text that just changed is now working from a stale picture. Since every edit is already recorded against specific clauses, you can compute exactly which rules those are.

That's the second payoff from the identifier scheme — it was introduced to detect conflicts, and it happens to give you invalidation for free. The post doesn't claim they do this, but they've described every ingredient you'd need.

The risk classification scores in the eval results table seem very low. Any secondary review?

[Post] 59% rising to 77%, with no breakdown by category and no mention of any low-confidence re-check. The product is positioned as a first draft — "reliable first pass reviews," "giving users a well reasoned draft as a starting point." There is escalation to a human, but it's for situations where the deal has nuance the playbook can't capture, not for cases where the model is unsure.

[Inferred] Two separate questions here.

Is 77% actually low? You can't tell from the post, and that's the real problem. If it's measuring exact agreement with a lawyer's label on a three-way judgment call, 77% might be close to the ceiling — two competent lawyers won't agree 100% of the time on whether a deviation is acceptable or unacceptable, and the system can't beat a bar nobody has measured. Harvey never publishes how often their own lawyers agree with each other, which leaves the headline number floating without a reference point. That's the single biggest gap in the post.

Is it acceptable for the product? Yes, provided a lawyer reviews everything. A wrong label costs review time, not a bad contract — as long as the mistake is visible. The dangerous case is the clause the system never flagged at all, because nothing surfaces it for anyone to catch. That's the number worth asking about.

On secondary review specifically: the only second look described is the lead agent's final pass, and what that checks is coverage — that every rule got a verdict. Not whether the verdicts were right. An obvious addition would be re-checking the classifications the model was least sure about, or sending anything near the acceptable/unacceptable boundary to a second model. Nothing like that is mentioned.

“The issue would self-resolve usually on retry”

Why?

[Inferred] Because the failure isn't caused by the input, it's caused by the path the agent happened to take.

Model outputs are sampled rather than deterministic, so the same prompt produces a different first move on a different run. These loops almost always start with one bad early step — searching for a term that isn't in the document, misreading which party a clause binds — and then compound from there. Retrying re-rolls that first step.

There's a second effect that reinforces it. Once an agent has made a few failed attempts, those attempts are sitting in its own context, and the model is now predicting what comes next given a history of failure. It tends to produce more of the same. Starting over wipes that history, which is often more important than the re-roll itself.

And some fraction of "stuck" isn't the model at all — it's a slow or degraded response from the provider, which naturally clears.

Note the word "usually." Retrying doesn't rescue a genuinely hard clause, which is why they pair retries with timeouts rather than retrying forever. And retrying one rule without redoing the review is only possible because each agent works on its own copy.

“To mitigate this, we set timeouts by model, review phase, and document size”

What do timeouts in these different categories mean?

[Inferred] These are three inputs to a single time budget, and each one accounts for a different reason a step might legitimately take longer.

Different models work at very different speeds, especially those that do extended internal reasoning before answering. A single threshold would either cut off a slow model doing perfectly good work, or let a fast model hang for minutes before anyone noticed.

Different stages of the review have different natural durations. Classifying one clause against one rule is quick. Reconciling conflicts across dozens of memos is not. Holding both to the same limit means one of them is set wrong.

And bigger documents take longer for the obvious reason — more to read, more sections to search, more steps before the agent is satisfied. A 300-page contract can't be held to a 10-page contract's budget.

So the limit is looked up from those three facts, probably from a table calibrated against how long things actually took during their evaluation runs, and set well above the slowest legitimate case. The point isn't to make anything faster. It's to make sure that when one rule goes wrong, it costs you that rule and not the whole review.

“we added concurrency limits and exponential backoffs”

How to determine parameters of these?

[Inferred] These numbers aren't found by tuning the agent. They come from constraints outside it.

The cap on how many agents run at once is set by the rate limit Harvey has with the model provider, which is shared across everything Harvey does. You work backwards from it: take the tokens per minute you're allowed, divide by the tokens per minute a single agent consumes, and that's the most you can safely have in flight. Then you divide that budget among customers, so one person's 200-rule playbook doesn't consume the entire allowance and stall everyone else's reviews.

Backoff — waiting before retrying after being rate-limited — follows a standard recipe: wait about a second, double it each time, stop doubling somewhere around 30 to 60 seconds, and add a random amount to each wait. That last part is the one people forget and then regret. If forty agents all get rejected at the same instant and all wait exactly two seconds, they all retry at the same instant and get rejected again. Randomizing the wait spreads them out.

The actual tuning is empirical: raise the concurrency cap until you start seeing rejections and total review time stops improving, then settle just below that point.

One interaction the post doesn't address: time an agent spends waiting in the queue shouldn't count against its timeout. If it does, heavy load causes timeouts, which cause retries, which cause more load.

“We used that fact to redesign our prompts to optimize for prefix cache hits”

How to design prompt to enable this?

[Inferred] First, what the caching actually does. When you send a prompt, the provider does expensive setup work on it before generating anything. If the beginning of your prompt is identical to one it processed recently, it reuses that work — charging a fraction of the usual price and skipping the delay. The catch is that it only reuses the part that matches from the very first character, and it stops at the first difference. One changed word near the top wipes out the benefit for everything below it.

That gives you the design rules.

Put the most shared content first and the most varying content last. The order that works is: system instructions, then tool descriptions, then deal context, then the contract, then the specific rule, then the specific task. Any layout that mixes rule-specific text into the contract destroys the shared portion.

Make the shared part character-for-character identical across every agent. No timestamps, no request IDs, no agent number in a header, no data structures that serialize their fields in a different order each time.

And within a single agent's back-and-forth, only ever append to the conversation. Rewriting an earlier turn invalidates everything after it.

The payoff here is unusually large because the contract dominates the prompt and every single agent reads the same one. That's why one change improved both cost and speed.

“we enabled streaming results such that each rule’s result becomes available as soon as its subagent finishes”

Any concerns about quality given lack of orchestrator review at this stage?

[Inferred] Yes, and this is the one place where the post's own argument works against itself. The reason the lead agent exists is that raw output from individual agents can be contradictory. Streaming shows the user exactly that raw output, before anyone has reconciled it.

Three things can go wrong. A user reads rule 7's suggestion, and then reconciliation rewrites it to accommodate rule 23 — so they have to read it again, and may have already made a decision based on the version that's now gone. If they can accept a suggestion before reconciliation finishes, they can accept one half of a pair of edits that were supposed to be resolved together. And watching answers change on screen makes a system feel unreliable even when every change is the system doing its job correctly.

The standard ways to handle this: stream the classifications early, since reconciliation rarely changes whether a clause is a problem, but hold the actual edits until they've been merged. Or show streamed edits marked as still in progress and don't let anyone accept one until the review completes. Harvey's mention of exposing progress suggests something along these lines, but they don't say whether a rule that's about to be reconciled is visibly marked as such.

“Our evaluation showed that no single model performed best across subtasks of contract review”

Does subtasks mean rule classification vs redlining here or is it more granular where different rules get different models?

[Post] By stage of the review, not by rule: "We built the system to switch models across phases of the same review, selecting each one based on its measured quality, latency, and cost for that specific task."

[Inferred] The stages visible in the post are classification, searching and navigating the document, drafting redlines, resolving conflicts, and the final validation pass. These genuinely do want different things. Resolving a conflict between two competing edits is hard reasoning over a small amount of text, which favors the most capable model available. Navigating a 200-page contract to find the assignment provisions is easy reasoning over a lot of text, done many times, which favors whichever model is fastest and cheapest.

[Search] This lines up with how Harvey talks about its model lineup publicly, which sorts models by exactly that kind of profile: Claude Opus 4.6 for "agentic reasoning and deep multi-step legal analysis," Sonnet 4.6 for "strong performance with a significantly lower latency footprint," Gemini 3 Flash for "where speed and throughput matter." That's a by-stage taxonomy, not a by-rule one.

[Inferred] Sending harder rules to stronger models would be a sensible next step, but nothing in the post claims they do it, and it needs something the post never describes: a way to predict which rules are hard before you've reviewed them. More in Q37.

“We built our agent harness to normalize the differences between model providers.”

What does this mean concretely?

[Inferred] It's a layer of their own code sitting between the agent logic and each provider's API, so that the agents are written once against Harvey's own interface rather than three times against three different ones.

The differences it has to absorb are more than cosmetic. The request and response formats differ, including where the system instructions go and what the streaming events look like. Tool calling differs — what the mechanism is called, how tool schemas are written, whether the model can request several tools at once, what happens when it produces malformed arguments. The controls for how much the model thinks before answering differ and don't map cleanly onto each other. Caching differs: one provider wants you to mark explicitly where the cacheable part of the prompt ends, another does it automatically, so a prompt laid out efficiently for one isn't necessarily efficient on the other. And the operational details differ throughout — how rate limits are signalled, what the error types are, how tokens are counted.

This matters more for Harvey than for a typical chat product because their entire scalability story assumes models are interchangeable parts. Picking the best model per stage, setting timeouts per model, backing off per provider — none of that is manageable if each model is its own integration.

The part a harness doesn't solve is that prompts aren't portable. A prompt tuned around one model's habits often does worse on another. So you get the ability to swap models mechanically, but you still have to re-run your evaluations every time you do.

“Every accepted, edited, or rejected redline helps agents understand a legal team's preferences.”

In what way is this feedback actually incorporated?

[Post] It isn't yet. Note the tense: "This will allow us to build towards a personalized system, from users to accounts." No mechanism is described.

[Inferred] There are four ways to do this, and they differ a lot in cost and in how soon you could ship them.

The cheapest and almost certainly first is to use past decisions as examples. Store which redlines the team accepted and which they rejected, and when an agent is about to work on a given rule, pull up how that team handled the same rule before and put it in the prompt. No training required, works per customer, and it uses the same infrastructure as the historical-contract search in Q35.

The second is to improve the playbook rather than the model. If a team rejects your liability fallback every single time, that's not a model problem — the playbook is wrong. Surface it as a proposed change to the rule for a lawyer to approve. This keeps a human in the loop and improves the artifact the customer can actually read and audit.

The third is to train a small model on the accept/reject history and use it to filter out suggestions the team is unlikely to want before they ever appear.

The fourth is fine-tuning the main model on preferences. This is the most powerful and the least likely soon: per-customer fine-tuning doesn't scale, fine-tuning on everyone's data averages away the personalization that was the point, and one customer's contracts can't be used to improve another's reviews.

The complication running through all of these is that a rejection doesn't tell you why. A lawyer might reject a technically correct redline because the counterparty is important and this isn't the fight to pick, or accept a mediocre one because they're behind on the deal. Without capturing the reason, the signal stays noisy — which is probably part of why this is still future work.

“This opens the ability to search across thousands of documents for previous examples to trust agent outputs and quickly understand what a team has approved in the past and why.”

Who is doing this searching and in what part of the pipeline?

[Inferred] The sentence actually describes two different users doing two different things at two different points.

One is the agent, during the review. Before deciding how to handle a provision, it searches the company's past contracts for how that provision was settled before, and lets that shape which position it pushes for. This changes the output, and it's the personalization path.

The other is the lawyer, after the review. That's what "to trust agent outputs" points at: the interface shows a suggestion next to a note saying your team has agreed to this exact language fourteen times before. This doesn't change the suggestion at all, only whether the lawyer believes it.

My guess is they ship the second one first, because the worst case is much milder. A bad retrieval there is a weak citation the lawyer ignores. A bad retrieval in the first case is a bad redline that goes out the door. Both need the same underlying thing: an index of the company's signed contracts broken down by provision, recording what language was finally agreed and whether it was negotiated.

Worth noting what changes between today and that future. Right now the user chooses which precedent documents matter. Once the system chooses, retrieval quality becomes a new way for the review to be wrong.

How would you improve this system if you were building it today?

In rough order of value.

Fix the measurement before touching the architecture. Publish how often two lawyers agree with each other on the same clause, so 77% has something to be compared against. Report how often the system catches the unacceptable deviations specifically, since that's the expensive miss. And have lawyers who didn't write the rubrics grade a held-out set blind. Right now the headline numbers can't be interpreted, and every decision about what to build next rests on them.

Detect conflicts in meaning, not just in text. Their merge catches two edits touching the same words. It can't catch an edit that narrows a definition in section 1 when section 11 depends on the broad version. When the contract is parsed, build a map of which clauses define terms, which clauses use them, and which clauses cross-reference which. Then check the merged document against that map in code. It's a deterministic check, not a model call, and it closes the gap that worries me most.

Group related rules instead of one agent per rule. Four rules about limitation of liability currently produce four agents that each read the same section, each work out the same thing, and then collide. Cluster rules by which part of the contract they target and give one agent the cluster. That removes duplicated work and removes the conflicts before they happen rather than resolving them afterward.

Escalate when the model is uncertain, not only when the deal is unusual. Today escalation is about deal nuance. Add a confidence score to each classification and route the uncertain ones — especially anything sitting near the line between acceptable and unacceptable — to a stronger model or to a person, marked as needing judgment. This is the cheapest way to reduce the risk of a silent miss.

Route by rule difficulty as well as by stage. They pick a model per stage of the review, but difficulty varies far more between rules than between stages. A rough difficulty signal — how long the rule is, how many conditions it has, how many sections it touches — would let the easy majority go to a fast model and concentrate the budget on the hard minority.

Start the feedback loop now, as retrieval. Accept/reject history is the most valuable thing they'll accumulate, and you don't need to train anything to use it. Index it per customer and feed relevant past decisions to the agent as examples. And capture a reason when a lawyer rejects something, even a one-click reason, or the data stays ambiguous forever.

Make streaming honest. Either hold edits back until they've been reconciled, or show them marked as provisional and don't allow accepting one until the review finishes. Otherwise streaming quietly gives away the consistency guarantee the lead agent exists to provide.

Is the main difference between prompts and context that context is information + instructions and prompts are just instructions?

Yes, pretty much. Context is also often dynamic.

“we need strategies for managing the entire context state (system instructions, tools, Model Context Protocol (MCP), external data, message history, etc).“

In what form is MCP included in the context?

MCP isn't a protocol object in the window — it gets flattened into ordinary text. Three forms, MECE by when they land:

1. Connection time — tool schemas. Each server advertises tools; the client renders each as name + description + JSON Schema, injected into the tool-definitions block. This is the dominant cost: tens to hundreds of tokens per tool, paid every turn. Twenty servers × 15 tools is easily 20k+ tokens before you say anything.
2. Connection time — server instructions, resources, prompt templates. Servers can return an instructions string (typically appended near the system prompt) plus listings of available resources and prompt templates.
3. Call time — tool results. Whatever content blocks the server returns land in the message history and persist there.

Point 1 is why MCP makes the post's "bloated tool sets" failure mode the default rather than a mistake you have to commit.

“This results in n² pairwise relationships for n tokens.”

Is this true for modern architectures as well? What about local attention?

True of the mathematical object, increasingly false of the implementation. Three things to keep apart:

1. Compute vs. memory. FlashAttention removed the n² memory cost (it never materializes the attention matrix) but keeps n² FLOPs. GQA and MLA shrink the KV cache, not the pairwise count.
2. Sparse and local attention genuinely break n². Sliding-window attention (Mistral, Gemma) costs n·w. Frontier models interleave — a few global layers among many local ones. DeepSeek's NSA learns which blocks to attend to. [Search] 2026 work is predominantly hybrid: sparse + linear + a few full-attention layers.
3. It doesn't rescue the conclusion — it strengthens it. A token in a sliding-window layer cannot see distant tokens directly; it depends on multi-hop propagation up the layer stack. So long-range degradation is, if anything, more likely under local attention. The post's "attention budget" claim survives even though its stated mechanism is a simplification of what frontier models actually run.

“Additionally, models develop their attention patterns from training data distributions where shorter sequences are typically more common than longer ones.”

Don’t models typically get fine-tuned on representative data lengths even if pre-training data has more shorter sequences?

[Post] Doesn't address it — just asserts the pretraining-distribution claim.

[Inferred] Your objection is correct and the post's framing is dated. The standard modern recipe is:

1. Pretrain short (4k–8k) for most of the token budget — cheap.
2. A long-context extension phase: continued pretraining at 32k → 128k → 1M, using RoPE base frequency scaling (ABF) or YaRN, on data deliberately upsampled for long documents and synthetic long-range tasks.
3. Long-context SFT / RL on tasks requiring long-range dependence.

Three things blunt the correction:

1. Volume. The long-context phase is typically low single-digit percent of total training tokens. Parameters are still overwhelmingly shaped by short-sequence gradients.
2. Data quality. Genuine documents with real long-range dependencies are scarce. Much long-context training signal is constructed — concatenated documents, synthetic needle tasks — which teaches retrieval more than reasoning across the whole span.
3. Measurement. Needle-in-a-haystack is near-saturated, but multi-hop and aggregation-over-long-context benchmarks (RULER and its successors) still degrade sharply with length.

Net: the post's mechanism is stated too simply; the empirical claim holds for reasons that survive your correction.

What’s a two sentence summary of position encoding interpolation?

Rather than asking the model to handle position values it has never seen, you rescale the positions of a longer sequence down into the range it was trained on — position 8,000 in a 32k input is fed as if it were position 2,000 in the original 8k range. This works because interpolating inside the trained range is far safer than extrapolating outside it, but it compresses the positional signal so adjacent tokens become harder to distinguish, which is the "degradation in token position understanding" the post mentions.

“smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome”

Dual objective optimization or satisficing?

[Post] Written as if it were a joint optimum, and never resolved. The post does give itself an escape hatch elsewhere: "minimal does not necessarily mean short."

[Inferred] As literally written it's ill-posed — you cannot minimize tokens and maximize success without a stated exchange rate. Three readings:

1. Scalarized dual objective. max P(success) − λ·tokens. Coherent, but λ is never stated, and in practice λ is tiny next to the cost of a task failure.
2. Constrained. min tokens subject to P(success) ≥ threshold. This is what "smallest possible set" implies grammatically, and it is the defensible reading.
3. Satisficing, in practice. Nobody maps the frontier. Teams reach "good enough," trim context until quality visibly drops, and stop.

[Inferred] The more useful reframe: it is effectively lexicographic — quality first, tokens as tiebreaker — because over the relevant range the two objectives are not opposed. Adding low-signal tokens hurts both. A real tradeoff only appears once every remaining token is genuinely informative, which most production prompts never reach.

“System prompts should be extremely clear and use simple, direct language that presents ideas at the right altitude for the agent.”

How good are LLMs at writing system prompts for other LLMs? Could I give an LLM examples of good and bad system prompts and expect it to come up with a good system prompt for my system?

[Inferred] Split into two capabilities, because the answer differs sharply:

1. From examples of good/bad prompts alone — weak. Style transfer works; you'll get something that looks like a good prompt. But prompt quality isn't a property of the text, it's a property of text × task × model. With no task data the model has no signal about your failure modes. Expect a plausible first draft, not a good prompt.
2. From examples of good/bad task outputs plus a metric — strong. This is the real answer, and it is a solved-enough tooling problem. [Search] The two current standards are DSPy's MIPROv2 (bootstraps demonstrations, proposes candidate instructions, Bayesian search over combinations) and GEPA (reflective evolutionary search — the model reads execution traces and failures, then mutates the prompt). GEPA generally wins at lower rollout budgets. TextGrad and OPRO are the earlier academic line.

[Inferred] Concrete path, matching the post's own recipe: hand-write a minimal prompt → build a 30–100 case eval set with a scorer → run GEPA or MIPROv2 against it. The eval set is the work; the optimizer is cheap. Without evals you are doing vibes whether a human or an LLM writes it.

“then add clear instructions and examples to improve performance based on failure modes found during initial testing”

How to address risk of overfitting?

[Post] Never mentions the risk. This is a real gap — as written, the recipe is textbook dev-set overfitting.

[Inferred] Mitigations, MECE by where you intervene:

1. Data. Hold out a test set you never inspect during iteration. Refresh the dev set from production traffic periodically. Keep a regression suite of already-fixed cases to catch whack-a-mole.
2. Edit policy. Prefer general fixes to specific ones — this is the post's laundry-list-vs-canonical-examples point restated as bias-variance. A rule naming a specific input is high-variance; a rule naming a principle generalizes. If you cannot articulate the principle behind a patch, that is the signal you are fitting a sample.
3. Budget. Cap prompt growth. Every N iterations, delete the oldest instructions and re-run evals — most will not be load-bearing. Prompts accrete monotonically unless you force a deletion step.
4. Model. Re-test on a different or newer model. Instructions that help only one model are usually compensating for that model's quirk, not encoding your task.

“One of the most common failure modes we see is bloated tool sets that cover too much functionality or lead to ambiguous decision points about which tool to use.”

How can you detect and mitigate this issue?

Detection, MECE by signal type:

1. Static — no traffic needed. Embed tool descriptions and flag high-similarity pairs. Run the post's heuristic as an actual exercise: two engineers independently label which tool each of ~50 realistic requests should use, then measure agreement.
2. Behavioral — from traces. Per-tool call rate (never-called tools pay token rent every turn for nothing); tool error and retry rate; thrash patterns where tool A is followed by tool B on the same subgoal; definitions as a share of total context.
3. Interventional. Ablate: remove a tool, re-run evals. The only signal that is causal.

Mitigation, MECE by action:

1. Merge near-duplicates into one tool with an enum parameter.
2. Delete anything below a usage floor.
3. Tier — a small always-loaded core set, plus dynamic loading of the rest (tool search, or per-task MCP server loading).
4. Disambiguate in the description — state explicitly when not to use it and which tool to use instead. Cheapest fix, usually the highest yield.

“However, teams will often stuff a laundry list of edge cases into a prompt in an attempt to articulate every possible rule the LLM should follow for a particular task. We do not recommend this. Instead, we recommend working to curate a set of diverse, canonical examples that effectively portray the expected behavior of the agent.”

One person’s laundry list is another person’s diverse set. How to distinguish?

[Post] No criterion offered. Genuine soft spot in the post.

[Inferred] The distinction is real and testable. Three tests, cheapest first:

1. Generalization test. Does this example teach a rule applicable to inputs you haven't written down? Canonical examples cover a region of the behavior space; laundry-list items are point masses covering only themselves. If removing an item breaks only the exact case it names, it is laundry.
2. Coverage-vs-count test. Map examples against the task's dimensions (input type × difficulty × output form). A diverse set is few examples spread wide. A laundry list is many examples clustered in one corner — usually whatever broke most recently.
3. Ablation test. Remove each item, re-run evals. Decisive, and the only empirical one.

[Inferred] Two fast heuristics that usually settle it without running any of the above:

• Provenance. Laundry lists grow append-only from incident response; canonical sets are curated top-down from a task taxonomy. If your examples section is a log of past bugs, it is a laundry list regardless of how varied it looks.
• Form. Laundry-list items are exceptions ("if X, don't do Y"). Canonical examples are demonstrations (full input → output pairs).

“agents built with the “just in time” approach maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools”

What does it mean to “maintain lightweight identifiers”? Where and how?

[Inferred] "Maintain" covers three different storage locations, MECE by where the identifier sits:

1. In the context window, as plain text. The agent ran ls or grep -l and the paths are now in the message history. That is the maintenance — a path costs ~10 tokens, the file ~10,000. No machinery required. This is most of what Claude Code does.
2. In the environment, re-derived on demand. The identifiers are not held at all; the filesystem is the index, and the agent regenerates paths with glob/grep whenever needed. Zero context cost, one round trip.
3. In external persistent storage. A NOTES.md, a to-do list, or the memory-tool file system the post introduces later. The durable form — and the only one that survives compaction.

[Inferred] The honest answer: this is mostly not a data structure you build. It is a discipline enforced in your tool contracts — make search return paths + line numbers + a two-line snippet rather than file contents, and the identifiers maintain themselves. The engineering work is in tool return shapes, not in an identifier registry.

“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time”

These feel like very rudimentary ways to access relevant files? Is lexical similarity actually sufficient?

[Inferred] For code, the objection is weaker than it looks, for three reasons:

1. Code isn't prose. Identifiers are exact, near-unique strings. Want UserAuthService? Grep it — every definition and call site, perfect precision. Embeddings are worse here: they blur exact tokens, which is the opposite of what you want.
2. It's agentic, not single-shot. The comparison is not one grep vs. one embedding query. It is one embedding query vs. fifteen greps with a model reading results and reformulating between each. The loop supplies the semantics; grep supplies the lookup. Recall misses get recovered in later iterations.
3. Freshness and zero infrastructure. No index to build, invalidate, or sync. On a repo the user is actively editing, a stale index is worse than none.

[Inferred] Where it genuinely fails: conceptual queries with no shared vocabulary — "where do we handle rate limiting?" when the code calls it ThrottleGuard. The model's world knowledge partly covers this (guess synonyms, grep again) but not reliably. That is the case for hybrid: semantic search over docstrings or file summaries to find the neighborhood, then grep for precision inside it.

“model to summarize and compress the most critical details”

How does the model know what to include in the summary and what the most important details are? Do you just ask it to figure that out itself or is it more sophisticated than that?

[Post] Answers more than you would expect. It names the categories — the model "preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages" — and says engineers should be "carefully tuning your prompt on complex agent traces." So: it is a prompt, and the categories come from the engineer, not the model.

[Inferred] But it is more structured than "summarize this":

1. Schema-driven, not free-form. Production compaction prompts specify an output skeleton — goal, decisions made, current state, open problems, files touched, next step. The model fills slots. Far more reliable than open-ended summarization.
2. Deterministic pre-filtering runs first. Tool result clearing (the post's own example, now a Claude Developer Platform feature) involves no model judgment at all. Cheap deletions happen before expensive summarization.
3. Recency passthrough. Recent turns are copied verbatim rather than summarized. The five-files detail is this same idea applied to files.

[Inferred] What is not happening: there is no training signal on "did this summary preserve what mattered." Importance is hand-specified and tuned offline against traces — which is exactly why the post frames it as a recall-then-precision prompt engineering exercise rather than a learned component.

“The agent can then continue with this compressed context plus the five most recently accessed files”

Why five? What happens if the files are very big?

[Inferred] On five — almost certainly an empirical constant, not a derived one:

• It is a recency heuristic proxying for relevance. In coding, files you just touched are the ones you are about to touch. Cheap and usually right.
• Five is roughly where marginal value crosses marginal token cost. At ~500–2,000 tokens per source file, five is ~2,500–10,000 tokens — low single-digit percent of a 200k window. Twenty files would be a meaningful fraction of a freshly cleared window.
• It is the kind of number tuned once on internal traces and then frozen.

On big files — the more interesting question, and the post ducks it. Three possibilities:

1. A token budget binds, not the count. Any sane implementation caps total tokens, so "five files" is really "up to five, subject to a budget." Otherwise one vendored 500k-line file blows the fresh window immediately.
2. Truncation or re-read-on-demand. Either partial inclusion, or keep only the path and let the agent re-read the relevant span with offset/limit — which is precisely the just-in-time pattern the post argues for earlier.
3. If neither: compaction becomes self-defeating — you summarize to free space, then immediately refill it.

[Inferred] Worth flagging the tension: re-injecting file contents is pre-loading, which contradicts the post's own just-in-time thesis. Re-injecting paths would be the consistent choice. The likely justification for pre-loading anyway: after a reset the agent has lost the reasoning that made those files relevant, so paying tokens to restore working state beats a slow rediscovery loop.

“Start by maximizing recall to ensure your compaction prompt captures every relevant piece of information from the trace, then iterate to improve precision by eliminating superfluous content.”

How would recall and precision actually be measured in this situation? Or is it more qualitative?

[Inferred] Both are measurable, but only once you fix a "unit of information" — which is the whole difficulty. Three levels, MECE by cost:

1. Qualitative human review — what most teams actually do. An engineer reads a long trace, lists what matters, and checks the summary against it. Recall = fraction of that list present. Subjective and slow, but catches the failures that matter.
2. Proposition-level with an LLM judge — the practical middle. Decompose the trace into atomic claims ("the auth bug is unfixed", "we chose Postgres over Mongo"). A judge checks each against the summary. Recall = claims preserved / claims extracted. Precision = summary claims supported by the trace / total summary claims — note this catches hallucination, not just verbosity.
3. Downstream task accuracy — the only ground truth. Generate question-answer pairs from the full trace, then ask them of an agent given only the summary. Recall failure = questions it can no longer answer. This measures the thing you actually care about (can the agent continue) and is the standard formulation in long-context and memory benchmarks.

[Inferred] One precision note: the post's "precision" is not the classification-theoretic one. It means token efficiency — no superfluous content. So the operational metric is closer to compression ratio at fixed recall.

“the agent regularly writes notes persisted to memory outside of the context window”

1. How does the model know what to include in the notes? Is it application-dependent?
2. Why written to memory outside of context window? How does it know when to retrieve?

(1) What to include

[Post] Says both things, in tension. Engineered on one hand (Claude Code's to-do list, a NOTES.md convention); emergent on the other — the Pokémon agent, "without any prompting about memory structure," develops maps of explored regions, tracks achievements, and maintains combat strategy notes.

[Inferred] Reconciliation: the trigger and affordance are engineered (a tool exists, the prompt says to use it); the schema is often emergent. And yes, strongly application-dependent — what to persist is whatever must survive a reset in that domain: architectural decisions for coding, entity relationships for research, map/inventory/strategy state for a game.

(2) Why outside the window, and when to retrieve

Why outside: [Post] "provides persistent memory with minimal overhead" — notes survive compaction and resets; in-context content by definition does not.

[Inferred] A second reason the post does not give: notes are rewritable. Context is append-only — you cannot revise an earlier message, only append a correction, and both then sit in the window. A file can be edited in place, so stale state gets replaced instead of accumulating alongside its correction.

When to retrieve: [Post] Doesn't say, beyond "these notes get pulled back into the context window at later times" and "after context resets, the agent reads its own notes."

[Inferred] Three mechanisms in practice, MECE by who decides:

1. Unconditional on reset — the post-compaction prompt says "read your notes file."
2. Model-decided — the memory tool is a filesystem; the agent reads when it judges it needs to. Unreliable without prompting.
3. Always-on injection — small files (a to-do list) get re-rendered into context every turn.

[Inferred] The real gap: retrieval timing is the weakest link in the whole scheme. An agent that does not know it forgot something has no trigger to go look.

“After context resets” What does reset mean here?

[Post] Undefined. Used once, in the Pokémon section, next to "coherence across summarization steps" — implying reset ≈ compaction event.

[Inferred] It conflates three things worth keeping separate, MECE by what survives in-band:

1. Compaction. The window is replaced by a summary of itself. Some information survives in-band. This is what the post means in that sentence.
2. Hard clear. The window is wiped to system prompt plus a fresh start; nothing survives in-band, only external files carry over. This is what "reset" most literally means, and what a game harness does when it hits the limit.
3. New session. A separate process started later — the same information situation as (2) but across time. This is what the memory tool targets: "maintain project state across sessions."

[Inferred] Why the distinction matters: the note-taking argument is only load-bearing for (2) and (3). For (1), a well-tuned compaction prompt can carry state in-band. Notes still win because they are lossless for what they cover and do not degrade across repeated resets — a summary of a summary of a summary drifts; a file does not.

How would you improve this system if you were building it today?

Ordered by expected value:

1. Make context management measurable, not advisory. The post's entire toolkit is heuristic with no evaluation story. Build a continuation benchmark: sample real traces, compact them, then measure whether the agent can answer trace-derived questions and complete the task. That converts "tune your compaction prompt" into a regression test.
2. Tiered editable state instead of one summarization pass. Compaction today is monolithic and lossy in a single shot, and re-summarizing a summary compounds drift. Better: verbatim recent turns, plus a structured state document (decisions, open problems) that is edited rather than re-summarized, plus cold storage that is searchable but unloaded.
3. Treat tool definitions as retrievable context. MCP made bloated tool sets the default. Load a small core set and retrieve the rest on demand via tool search. This directly attacks the failure mode the post names but does not solve.
4. Give note-taking an index. Per Q16, retrieval timing is the weak link. Keep a cheap always-in-context table of contents of what is in memory, so the agent knows that something exists before deciding whether to read it. Knowing you forgot is the precondition for looking.
5. Reconcile just-in-time with the five-files behavior. Post-compaction, re-inject file paths plus one-line summaries, not contents. Cheaper and consistent with the rest of the argument.
6. Measure the write side, not just the read side. Everything in the post is about what enters context. Nobody measures what the agent writes — note quality, note staleness, contradictory notes accumulating. That is where long-running agents actually fail.

“customer-reported misses in Detection 360”

1. What is Detection 360?
2. How do customers realize misses happened?

(1)

An in-product submission channel: "Submit a missed attack or a false positive incident from within the product's Detection 360 tab" (navigation: Investigate → Detection 360). The case object carries sender, recipient, subject, submitted-by, case number, status (Submitted/Resolved), and a VIP flag. The platform page summarizes the loop: "Detection 360 investigates, explains the outcome, updates the models." Case lifecycle runs "from initial classification through investigation, remediation, and deployed detections." Turnaround is marketed as "hours, not days," and the loop reportedly got "94% faster."

(2)

Abnormal only ever uses analyst-initiated framing ("An analyst reports a vendor impersonation attack"). Their AI Security Mailbox "connects to existing phishing-report buttons, abuse mailboxes, G Suite alerts, and third-party tools" — but no page states that end-user reports automatically create Detection 360 submissions. They are documented as separate paths.

[Inferred] The realistic channels, MECE by who notices:

1. End user reports it — phishing button or abuse mailbox, then a SOC analyst escalates
2. SOC proactive hunting — searching mail for indicators pulled from threat intelligence
3. A different tool catches it downstream — EDR fires on the payload, web proxy blocks the link, a SEG flags a sibling message
4. Post-incident, working backwards — someone clicked, credentials were phished, money moved
5. External notification — the impersonated vendor, a partner, or law enforcement
6. Purple team or phishing simulation — a deliberate test that got through

Where does this agentic pipeline fit into the broader system? Given latency constraints

It is clearly offline and asynchronous. The post gives four tells:

• It is triggered by a submission, not by an arriving message
• It evaluates "across a larger slice of the customer's real traffic" — historical replay
• It loops "multiple times"
• End-to-end is "hours"

So the pipeline sits beside the detection engine, not inside it. Same architecture as the 2024 rules post: expensive LLM work happens once per attack pattern offline; a cheap deterministic artifact runs online.

The three-tier picture:

[Inferred] This places the agentic loop in the middle speed tier — faster than retraining, far slower than a scoring decision. Which is exactly the gap the 2024 post identified: "adding new [features] involves retraining the model, which takes weeks." The agents fill that hole. That is the one-sentence answer to how it fits.

“how they normally communicate”

In what form?

Sender-history profile, computed per sender-recipient pair and per sender-tenant:

• Has this sender written to this recipient before, and how often
• Typical send times and cadence
• Typical infrastructure — sending IPs, domains, authentication posture
• Typical recipients within the organization
• Typical subject/topic distribution and tone
• Signature blocks and formatting conventions

Form: almost certainly a structured summary — counts, rates, first-seen dates — plus probably a handful of retrieved prior messages for the agent to read.

“how the message compares to typical organizational patterns”

In what form?

• Does this organization receive Teams event invites at all? From which platforms?
• Which vendors are established relationships versus first contact?
• What is the normal distribution of external senders, of financial-request language, of invite mail?

“what signals point toward intent or impersonation”

In what form?

[Post] Not specified, but the example output enumerates a real subset: sender domain and its subdomain lineage, authentication results (SPF/DKIM), body link destinations, subject content, delivery mechanism. [Post] elsewhere: "thousands of rich signals."

[Inferred] The likely full set adds display-name-vs-address mismatch, first-time-sender flags, phone numbers extracted from the body, brand mentions inconsistent with sender identity, and the text-model scores from the NLP post (urgency, tone).

[Inferred] Form: most likely a text serialization of the feature record — name/value pairs with short natural-language annotations — because the agent's output reasons over them in prose. Standard pattern: render structured features into a prompt, let the model reason, give it tools to query more.

[Inferred] The design question worth raising: with "thousands" of signals you cannot render them all into a prompt. So something selects. Either the agent has tools to query attributes on demand — which the post hints at, "its tools let it verify those intuitions against real data" — or a retrieval step pre-selects. On-demand querying is the better design and is what the post implies. Same feature-selection-as-retrieval problem flagged in Q4 on the 2024 post; it recurs here.

“This is a **trusted platform abuse** attack - using Microsoft Teams' event registration system to deliver phishing content.“

If the Deep Research Agent already determined it’s an attack, what’s the Adaptive Detection Agent doing?

[Post] The Deep Research Agent's output is "a detailed behavioral profile of the attack" — prose, explaining why this one message is malicious.

[Post] The Adaptive Detection Agent's job is to "find the attributes that represent the underlying behavior behind the attack," check statistical and semantic relevance, apply second-order abstraction, test against traffic, and iterate.

Three ways to state the difference:

1. Explanation vs. classifier. The DRA answers "why is this message malicious?" The ADA answers "what decision rule separates this class of message from all normal traffic?" A perfect explanation may cite features that do not generalize, or that are not even available as production attributes.
2. Prose vs. an executable artifact grounded in the production feature space. The DRA can say "uses legitimate Microsoft Teams infrastructure." The ADA must translate that into attributes the engine actually computes at scale, in sub-second, and verify each one separates on real traffic. The DRA has no obligation to check whether "uses Teams infrastructure" also fires on ten thousand legitimate meeting invites a day. The ADA does.
3. One message vs. a population. The DRA reasons about a single known-bad message with zero false-positive cost. The ADA operates against a one-in-a-million FP budget over a traffic distribution.

[Inferred] The analogy: the Deep Research Agent writes the incident post-mortem; the Adaptive Detection Agent writes the regression test. Knowing exactly why the bug happened does not hand you the test that catches the next one.

[Inferred] One more asymmetry: the DRA's determination is retrospective, on a message already known to be a miss. The ADA must produce something that works prospectively, on messages nobody has labeled.

“The problem is that without any deeper understanding, teams end up with rules that are technically correct but practically useless. A system might flag that 100% of attacks in a cluster lack an attachment, which is true but meaningless since most legitimate emails don't have attachments either. Statistical relevance alone produces a lot of noise without the next critical piece”

This seems like it’s only true for the most rudimentary definition of statistical relevance?

Yes. The critique is correct, and this is the strongest pushback available on this post.

Their example is a statement about P(no attachment | attack) — recall on the positive class alone, with no reference to the negative class. Every standard criterion rejects it immediately:

So what they describe is not "statistical relevance" — it is support on the positive class. A first-year decision-tree splitting criterion catches this. They picked a weak example and undersold their own argument.

[Inferred] The real case for semantics is different, and stronger. Three genuine reasons statistics alone is insufficient here:

1. Multiple comparisons at tiny n. A submission yields maybe a handful of attack messages against thousands of candidate attributes. At small n and high dimensionality, many attributes separate perfectly by chance. Statistics cannot distinguish causal from coincidental at that sample size. Semantics functions as a prior over hypotheses that prunes the search space before testing — exactly what you need when the multiple-testing burden is otherwise fatal. Note this is a well-posed statistical argument, not an anti-statistical one.
2. Durability under adversarial shift. Statistical relevance is measured on the past and says nothing about which correlations survive the attacker's next move. "Sender domain equals X" is maximally statistically relevant and minimally durable. Semantics is how you predict stability — their own second-order-thinking point, and the argument they should have led with.
3. Cost asymmetry is not in the statistic. At a base rate under 1 in 10 million and an FP target of 1 in a million, an attribute with excellent in-sample lift can still be catastrophic at production volume. Understanding what an attribute means is how you anticipate where it fires outside your sample.

[Inferred] Accurate framing: semantics is a prior and a durability filter, not a substitute for statistics.

[Inferred] The charitable reading, in their defense: at n of about 5 attack messages, most rigorous statistical criteria are genuinely underpowered — you cannot estimate lift reliably from five examples. So their practical situation may justify leaning hard on priors. That is probably what they meant; the attachment example just does not communicate it.

“It has enough statistical sense”

How? Reasoning, tool-use or something else?

[Inferred] Tool use, not reasoning. The LLM does not compute statistics in-context. It calls something that queries traffic and returns counts and rates, then reasons over the returned numbers. That is the only design that works, for three reasons:

1. LLMs are unreliable at arithmetic over large numbers, and these are counts over millions of messages
2. The data is not in the context window — it is in their feature store and data lake
3. The post explicitly describes an iterate-and-verify loop, which requires an external oracle

[Inferred] The likely tool surface: "for attribute A and recipient R, what fraction of the attack messages have value v, and what fraction of normal traffic does?" plus "run this candidate detector against this traffic slice and return the hits."

The "statistical sense" that actually lives in the model is choosing which query to run and interpreting the result — not computing it.

[Search] Consistent with their broader practice: their backtesting infrastructure already does exactly this kind of replay, "re-hydrated exactly as they would have been during online scoring."

[Inferred] Worth naming the pattern: LLM-as-orchestrator over a statistical oracle. Same division of labor as the 2024 post — LLM does translation and hypothesis generation, deterministic software does evaluation. The same team making the same architectural choice twice, two years apart, is a real signal about their design philosophy.

“The second-order approach is to ask: what's true about the domain that makes it suspicious? Is it a domain the organization has never seen before? Is it newly registered? Those characteristics will still hold when the attacker switches to a different domain next time.”

Is the Adaptive agent simply trying to answer yes or no to these questions, or use them in its downstream determination in the same iteration as well?

[Post] answers this indirectly: "its tools let it verify those intuitions against real data before committing to a rule." So the sequence is hypothesize, verify, commit — and the verified abstraction becomes a predicate in the rule, not a fact that is consulted and discarded.

[Post] The end state confirms it: "The result is detectors that generalize well beyond the specific messages they were built from." That only holds if the abstraction (domain novelty, domain age) is what the shipped detector tests, not the raw domain.

[Inferred] The flow for the PayPal example:

1. Observe — all the attacks share sender domain *.teams-events.com
2. Hypothesize the second-order property — "a domain this organization has never seen before," and/or "recently registered"
3. Verify against traffic — does never_seen_domain actually separate these attacks from this recipient's normal mail?
4. Commit — if yes, the abstraction goes into the detector; the raw domain does not

So: both. It is a hypothesis that gets answered yes/no and, if yes, becomes part of the rule in that same iteration. The yes/no is the verification gate; the attribute is the payload.

[Inferred] Worth naming: this is generate-and-test over a hypothesis space — classic program synthesis with an execution oracle. The LLM proposes candidate abstractions; traffic data accepts or rejects them.

“they latch onto the most distinctive surface features of the example they were shown”

What does distinctive mean here?

[Inferred] Precisely, it means maximally identifying of the specific example, relative to the corpus prior — the lowest-probability, highest-information strings in the input. For the PayPal message that is the exact domain jorjarounsevell.s08.usa1.teams-events.com, the exact phone number, the exact subject line.

Three framings of why models gravitate there — same phenomenon, different vocabulary:

1. Information-theoretic. A rare string carries high self-information. Asked "what characterizes this message," the rare strings are the highest-signal answer to that question as literally posed.
2. Learning-theoretic. From n of 1, the minimum-description-length hypothesis that perfectly separates the example is the example itself. Memorization is the optimal fit. Generalization requires a prior preferring abstract hypotheses, and nothing supplies that prior unless you put it there deliberately.
3. Shortcut learning. The model takes the feature with the highest in-sample discriminative power at the lowest reasoning cost. Exact string matching is both.

[Inferred] The essential point: distinctive is measured on the example; predictive must be measured on the class. Those coincide only if the example is representative in the right way — and for an adversary-generated example they systematically do not, because the attacker controls the distinctive features precisely so they can rotate them.

So in this domain, "most distinctive" and "most durable" are close to anti-correlated. That inversion is the entire reason second-order thinking has to be engineered in rather than emerging naturally.

“We worked to guide the agent toward behavioral abstractions over raw values”

By prompting, fine-tuning or what?

[Inferred] The wording points away from fine-tuning and toward three softer levers, in order of likely impact:

1. Attribute vocabulary design — probably the biggest lever, and structural rather than persuasive. If the attributes exposed to the agent are already behavioral — never_seen_sender, from_fqdn_age_in_days, both real names from their 2024 DSL — then a first-order rule becomes hard to express. You do not have to talk a model out of matching a raw domain if no attribute matches raw domains. This is guidance by construction, and it is far more reliable than instruction.

2. Prompting. Explicit instruction plus few-shot examples contrasting first-order and second-order rules for the same attack. Cheap — and "we worked to guide" / "something we spent a lot of time tweaking" is precisely what iterating on a prompt sounds like.

3. The verification loop as an implicit filter. A first-order rule passes local eval but generalizes badly in the broad eval — it will not catch other instances of the attack type. [Post] treats that as an explicit signal: "Did it find other instances of this attack type that were previously missed? If so, that's a strong signal." So the evaluation selects for generalization even when the model's first instinct is first-order. Guidance via reward rather than instruction.

[Inferred] Why probably not fine-tuning: you would need a labeled corpus of good-vs-bad detectors, which is exactly the scarce thing this system exists to produce; "guide" and "tweaking" are not fine-tuning vocabulary; and [Search] their documented fine-tuning work (7B Mistral/Llama-2 via LoRA) is described for detection and classification, not agent behavior.

“First, the agent runs a fast local evaluation. It takes the attack messages it already identified along with a random sample of normal traffic for the recipient and tests the detector against them.”

The random sample would need to be huge for this to work, right?

Yes — and the post effectively concedes it, twice.

[Post] The local stage is explicitly not a precision estimate: "This is intentionally lightweight. The goal is to iterate quickly." And the broad stage exists because the sample is insufficient: "it lets us catch any false positives hiding in the broader traffic that didn't show up in the smaller sample."

So the local sample's job is screening, not estimation — kill obviously broken candidates cheaply before paying for a full replay.

[Inferred] The arithmetic, worth having ready. Observing zero false positives on a sample of size n bounds the true rate only weakly. By the rule of three, with zero events in n trials the upper bound of the 95% confidence interval is about 3/n:

A "random sample of normal traffic for the recipient" — one mailbox, days or weeks of mail, so hundreds to low thousands of messages — is three to four orders of magnitude short of supporting a precision claim. It is entirely adequate for what it actually does.

[Inferred] The second, subtler problem: the sample is drawn from the wrong distribution. Randomly sampling one recipient's normal mail gives mostly easy negatives — internal mail, known vendors, newsletters. The false positives that matter are near misses: legitimate Teams invites, real PayPal notices, genuine first-time vendor contact. Random sampling surfaces those only in proportion to their frequency, which is low. So you systematically fail to detect the exact failure mode the detector is most likely to have.

[Inferred] The fix: stratified hard-negative sampling. The Deep Research Agent has already characterized the attack as "Teams event registration abuse." Use that to sample negatives conditioned on the characterization — real Teams event invites, real event-platform mail, real PayPal correspondence. Small, targeted, far more informative per message than a random draw. Standard hard-negative mining, and it fits their architecture cleanly since the characterization is already in hand.

“Every message the detector would have flagged gets reviewed, and we LLM-label each one to determine what it actually is.”

If we’re using an LLM to label it again, wouldn’t that just lead to artificially inflated agreement?

Good instinct, and partly right — but the failure mode is narrower than self-grading.

First, why it is not circular in the obvious way: [Post] the detector is not an LLM. It is a rule over attributes; the LLM wrote it offline. So at evaluation time you have a symbolic classifier being graded by a language model — different mechanisms, not a model scoring its own forward pass. That defuses the strongest version of the objection.

[Inferred] But three genuine correlation risks remain, MECE by where the leak enters:

1. Shared priors between writer and grader. If the same model family that reasoned "Teams invite plus urgency equals attack" also labels flagged messages, it will tend to agree — not because the detector is right, but because both inherit the same notion of what phishing looks like. Legitimate mail that merely looks phishy to that model gets labeled malicious, and a real false positive is scored as a true positive.
2. Context contamination. If the labeler sees the detector's rationale or the Deep Research Agent's attack profile, it is being primed with the hypothesis it is supposed to test. Strongest version of the problem, and the most easily avoided.
3. No ground-truth anchor. [Post] gives no indication that LLM labels are ever calibrated against human labels. Without that you cannot distinguish "the detector is precise" from "the labeler agrees with the detector." The number is uninterpretable in isolation.

[Inferred] What to do, in order of return:

• Blind the labeler — it sees the message and nothing about the detector or the attack profile. Cheap, eliminates risk 2 outright.
• Use a different model family for labeling than for generation — decorrelates errors. Same logic as their multi-tokenizer ensemble in the NLP post, applied to judges.
• Anchor on a human-labeled calibration set — periodically measure the labeler's own precision and recall against expert labels, then report detector precision corrected for labeler error.
• Human review on the tail — auto-accept where the labeler is confident and volume impact is small; escalate high-volume or VIP-recipient detectors.

[Search] They already know how to do this. Their 2024 post Testing GenAI Products describes using "a separate judge LLM, with access to our accuracy rubric" and running "separate 'unrestricted' red team LLMs." The judge-LLM discipline is established internally — this 2026 post just does not say whether those safeguards apply here.

“The agent loops through this cycle multiple times, refining the detector with each pass until it can reliably catch the attack without compromising on safe emails.”

Stopping criteria?

[Inferred] What a real implementation needs, MECE by termination reason:

1. Success — precision on the broad eval above a threshold, and recall on the submitted attacks at or near 100%, and firing volume within budget. All three, not just the first.
2. Budget exhaustion — max iterations, or max LLM/eval spend. Necessary because the loop is not guaranteed to converge.
3. Plateau — no improvement over k consecutive iterations. Cheaper than burning the full budget.
4. Abandonment — declare the attack not separable with the available attributes and escalate to a human. [Post] never mentions this possibility, and it must exist. Some attacks genuinely cannot be caught by a rule over current attributes; the honest outcome is "this needs a new feature" or "this needs the model," not a marginal rule. A loop with no failure exit will eventually ship something that satisfies the letter of the criteria and not the spirit.

[Inferred] The real risk the post never names: eval-set overfitting through iteration.

Every loop uses evaluation results to modify the detector. That is adaptive data analysis — after k rounds the measured precision on that traffic slice is optimistically biased, and the bias grows with k. You are effectively doing gradient descent on your test set.

Mitigations, strongest first:

• A final held-out gate — a fresh traffic slice never seen during iteration, evaluated exactly once. Fail there and the detector does not ship, regardless of how good the iteration numbers looked.
• Fresh slice per iteration, so no single slice gets fit repeatedly.
• Penalize iteration count — a detector that needed 12 rounds is more suspect than one that passed in 2, because it has absorbed more evaluation information.

This connects to Q32: the two-stage eval already has the right shape; it needs a third, sacrosanct stage that iteration never touches.

“Every submission kicks off the full loop”

What is a submission? Does the loop get kicked off by a single false negative?

[Search] A submission is a Detection 360 case, filed in-product: "Submit a missed attack or a false positive incident from within the product's Detection 360 tab." The case object carries sender, recipient, subject, submitted-by, case number, status, and a VIP flag.

So yes — a single customer-reported message can start the loop. Note it can be a false positive as well as a false negative, though this post only describes the missed-attack path.

[Inferred] But the loop does not stay at n of 1, and the post quietly tells you so. It says the agent tests against "the attack messages it already identified" — plural. That means between submission and generation, one reported message gets expanded into a cluster: the Deep Research Agent (or something upstream) finds related messages in the customer's traffic. This has to happen, because you cannot do statistical attribute selection on a single example.

[Inferred] Two consequences:

1. Cluster quality gates everything downstream. Expand too narrowly and you are back to n of 1, with no evidence to verify the second-order abstraction against. Expand too broadly and you are characterizing a mixture, so the detector ends up loose or incoherent. The post says nothing about how this expansion works, and it is arguably the most load-bearing undocumented step in the pipeline.
2. A single submission is an unvalidated trigger. [Search] No public Abnormal source addresses adversarial or poisoned submissions. An attacker with access to a tenant's Detection 360 — or who can get a legitimate message reported as an attack — could induce the system to auto-generate and deploy a detector that fires on legitimate traffic. [Post] There is explicitly no human in the loop: "No analyst queue, no manual review, no waiting." The evaluation stages are the only defense, and they are graded by an LLM (see Q33).

How would you improve this system if you were building it today?

Stage 1 — Understand the attack

• Fix the input bias, or at least measure it. Detection 360 misses are the misses somebody noticed, not a random sample (Q21). Instrument it: periodically sample random traffic for expert labeling to estimate the unreported miss rate, so you know how much of the problem the loop can even see. Without this you cannot tell whether "hours to protection" covers 10% or 90% of your misses.
• Document and harden cluster expansion (Q36). Report cluster size and purity alongside every generated detector, at minimum.

Stage 2 — Generate the detector

• Enforce second-order by construction, not by prompting (Q31). Expose only behavioral attributes, or type-check candidate rules and reject raw-literal predicates. Guidance that lives in the schema cannot be talked out of; guidance that lives in a prompt can.
• Generate a population, not a single detector. Sample K candidates, evaluate all, keep the precision/recall Pareto front. More LLM calls, but calls are cheap relative to a bad detector.

Stage 3 — Evaluation

• Hard-negative sampling in the local stage (Q32), conditioned on the attack characterization the Deep Research Agent already produced.
• Blind the LLM labeler, use a different model family, add a human-labeled calibration set (Q33) so labeler error can be corrected for rather than ignored.
• A held-out final gate the iteration loop never touches (Q35). The change to argue hardest for — without it, the reported precision is optimistically biased by construction.

Stage 4 — Precision optimization

• Explicit stopping and abandonment criteria (Q35), including the option to fail and escalate rather than ship a marginal detector.
• Penalize iteration count as a proxy for overfitting risk.

Stage 5 — Deployment

• Blast-radius controls. Auto-deploy in score-contributing mode first; promote to verdict-forcing after a live observation window. Cap how much a newly generated detector can quarantine per hour. Auto-rollback on FP-rate breach.
• Tiered human review. Fully automatic for low-volume, low-impact detectors; mandatory review above a volume or VIP-recipient threshold. Preserves the "hours, not days" claim for the common case while capping tail risk — and that framing is how to sell it internally without gutting the product story.

Cross-cutting

• Lifecycle and decay. Same gap as the 2024 post, but worse here, because generation is now fast enough to accumulate rules quickly. Per-detector monitoring (fire rate, precision, and marginal catch — messages it catches that nothing else does), automatic demotion when a detector stops catching, expiry by default.
• Reconcile per-tenant vs. global (Q21). There should be an explicit promotion path: a tenant-local detector that proves out across multiple tenants gets promoted to global, at a higher evidence bar.
• Submission trust model (Q36). Provenance checks, rate limits, and a poisoning red-team exercise. [Search] They already run red teams for prompt injection on their GenAI products; this loop deserves the same and there is no public evidence it gets it.
• Feed detectors back into the model. Detectors are a fast patch for a slow retrain. Mine the accumulated detector population for recurring attribute combinations — those are missing features for the core model. Over time the detector layer should shrink as the model absorbs what it keeps rediscovering. Without that, you are building an ever-growing rule layer and calling it progress.

Do different payment processors send different kinds of data? If yes, would this affect the approach you use?

Yes. Ramp likely has a "normalization" or "ETL" (Extract, Transform, Load) layer that maps various processor formats into a standardized internal schema before the AI agent ever sees it.

What format do they use for the input to the LLM?

Not stated in the blogpost but probably something along the following lines:

1. System Prompt with Guardrails
2. Transaction Metadata, User Request Data, Retrieved merchants, specified actions provided as a structured list like JSON or Markdown

Why use an LLM instead of a simpler model?

Simpler models

1. can’t provide explanations as easily
2. are harder to use for multimodal and unstructured data
3. would struggle with zero-shot because they don’t have world knowledge from pre-training

“By bringing in merchants using these two strategies, the LLM can inspect the names, websites, and categories of a good selection of merchants. It can then decide whether to create a new merchant or modify existing ones.”

It doesn’t look like they’ve mentioned how to produce a final list for the LLM. What’s our best guess for how they did the final ranking from the two lists?

Simple approach

1. Take k/2 from each list to get k total results

Cross-Encoder Re-ranker

1. Pull n >> k from each list to get 2n total results
2. Use a cross-encoder (using all the features you have for both the transaction and the merchant) to rerank and filter for the final k
3. Have LLM pick from this final list

Did they use any kind of few-shot prompting? If not, is that likely to have been helpful?

Unclear from the blogpost whether they used it.

It could be helpful, especially for edge cases like the government/Service Fee stuff and making it clear to the LLM what kind of explanation you want to provide users. Examples of both correct and incorrect responses could be helpful.

“We require that the LLM always chooses one of the provided actions”

It doesn’t look like they’ve mentioned how they actually enforce this restriction. What’s our best guess for how they did this?

They say “If the LLM hallucinates, we inform it of its mistake and have it retry until we get a valid response.” so it looks like they have retry logic to help with this.

They could also use Structured Output (potentially through Constrained Decoding).

“If the LLM hallucinates, we inform it of its mistake and have it retry until we get a valid response.”

What guardrails could they have put in place to avoid getting stuck in infinite loops?

1. Limit number of retries
2. Dynamic temperature adjustment
3. Meta-Prompting for Errors (provide explanation of failure to decrease likelihood of future failures in retries).

Once you hit the maximum number of retries without a valid response, you can escalate to humans.

“We take our primary LLM's rejection reasoning and then use a second LLM to rewrite it in plain language.”

Is the second LLM there because the primary LLMs rejection reasoning is unlikely to be very accessible to users? If yes, how could they have set up the second LLM to address that gap?

Yes, it’s better to exclude the human-optimized explanations from the primary LLM to avoid overloading it and keep it focused on its main job.

The second LLM can be prompted with something like

“You are a helpful Ramp Customer Support agent. Your goal is to explain a technical rejection to a user in a friendly, clear, and non-technical way. Do not use terms like 'MCC' or 'LLM' or 'Card Acceptor.' Use the provided evidence to explain the 'Why' simply.”

“It is also important to recognize LLMs' knowledge that has been distilled as part of their training.”

If this is the case, would it be helpful to replace the LLM every so often to incorporate this new knowledge? Or would a better strategy be to just store all instances of name changes and use a mapping between old names and new names?

Leveraging the User-Provided Data is probably a better way to update the LLM’s knowledge. You only need to do it once for each merchant whose name changed.

Storing mappings is complicated. Multiple companies can have the same name, only distinguishable using metadata like location, industry etc. Probably not worth it.

They mention “multimodal retrieval augmented generation (RAG)”.

Where is the multimodal part used?

Related receipt images; could use a Vision-Language Model (VLM) or a specialized OCR-to-text pipeline

How is the LLM as a judge supposed to know if the “agent's LLM action resulted in an improvement according to the user's request”?

The Judge is likely using a Rubric-Based Evaluation. The prompt for the Judge LLM for changes could look like this:

"Compare the Original Merchant with the New Merchant created by the Agent.

1. Does the New Merchant better match the Receipt Evidence (Line items: Fuel)?
2. Does the New Merchant fulfill the User's Intent (Requested: Gas Station)?
3. Is the New Merchant a valid entity (not a hallucination)?

Decision: If 'Yes' to all, mark as IMPROVEMENT. If the agent rejected the request, check if the User's input was 'Placeholder' or 'Malicious'—if so, mark it as REASONABLE REJECTION."

They said using LLM as a judge “Requires Cycles”; what does this mean?

Not sure what “Cycles” refers to here but I guess the general idea is that it requires a lot more work to set up. Maybe some degree of iteration on the prompt, input data etc to get it working correctly.

What is the “Mean Daily Improvement Rate on Classification Correction Requests”?

The 98.8% figure means that on a typical day, the Judge finds that 98.8% of the agent's changes resulted in a more accurate merchant record than what existed before. It is essentially an "Accuracy Score" for the actions the AI chooses to execute.

The reasonable rejection rate seems really low (one-third of legitimate user feedback is being ignored)? Should they be concerned about this?

Probably a worthwhile risk given that accepting a bad user request will mess up the database.

How would you improve this system if you were building it today?

1. Agentic Search for relevant merchants
   1. An agent could look at the merchant’s website, social media or Google Maps page to understand it better. Addresses from receipts can also be reverse-engineered to find Google Maps pages which can then provide a lot of additional useful data.
2. Speculative Decoding or Router Architecture (easy requests handled by fast model, harder requests handled by better model) to speed up responses
3. Consider adding some LLM as a judge negatives to the few-shot examples in the prompt

“citations were attached to the cell as a whole”

Does “attached to the cell” here mean that the citation mapping was (full generated text) <> (source document)? Or was it (sentence in generated text) <> (source document)?

I think they mean that the original was (full generated text) ↔ (source document) and they moved to (sentence in generated text) ↔ (source document).

“Previously, review tables generated sources through an algorithm that combined model-generated text and fuzzy matching.”

What would this methodology might have looked like concretely?

This could potentially look something like the following:

1. Generation step: The model produces a free-form answer for the cell (e.g., a summary of a contract clause) without any grounding mechanism built into generation itself.
2. Post-hoc matching step: Separately, the system takes that generated text (or key phrases from it) and runs a fuzzy string-matching algorithm (e.g., something like Levenshtein distance, n-gram overlap, or approximate substring search) against the source document to find the closest matching passage.
3. Attach citation to cell: Whatever passage scored as the best fuzzy match gets attached as "the citation" for that cell — but since matching happens on the whole generated text rather than per-sentence, the granularity is coarse

How can citations at a sentence level be generated?

This isn't explicitly outlined in the text some potential ways include:

1. Prompting-based: The model could be instructed via system/prompt engineering to output structured citations (e.g., "for each sentence, output the corresponding source index") as part of its generation — essentially asking the model to self-report indices as a formatting/output requirement.
2. Constrained decoding / structured output: Rather than relying on the model to "remember" to cite correctly via prompting alone, the system could constrain the model's output format at the token level (e.g., forcing it to emit index tokens at sentence boundaries), which is more reliable than prompting alone but requires deeper integration with the inference pipeline.
3. Retrieval-augmented indexing: The source document could be pre-chunked and indexed (e.g., by sentence, with unique IDs), and the model's job is to select from a known, indexed candidate set of sentences rather than generate a citation from scratch — reducing hallucinated citations by construction.
4. Hybrid: Prompting to elicit reasoning + a separate verification/alignment step that checks whether the model's claimed index actually matches the claim.

Sharing the reasoning with users seems kind of obvious. Is there any reason why they didn't do this in the previous iteration?

1. Latency/cost?
2. Risk of hallucination?

“This also resulted in a boost in output speed. By reassessing our execution pipeline, we were able to adopt an implementation that kept per-cell latency low and the overall architecture simple while achieving sentence-level granularity. At scale — a 30-column, 1000-document table produces 30,000 concurrent cells — the latency of this approach is substantial.“

I don't understand why this new approach is faster. Where is the latency boost coming from exactly?

They don’t explain it in the blogpost but it could be because:

1. Parallelization: processing citation-lookup across sentences concurrently rather than sequentially per cell
2. Reduced post-processing: eliminating the separate fuzzy-matching pass (which requires comparing generated text against source text as a distinct step) by having the model point to indices directly during generation — removing an entire post-hoc computation stage

Let's say I've built this pipeline and I want to get applied legal researchers' resources to assist in the evaluation. How can I, in advance of securing these resources, estimate how much manpower I need? One approach I had in mind was using synthetic queries in LLM as a judge to get a preliminary estimate of the improvement, and then use that to calculate the sample size I need for statistical significance at a desired margin of error.

This is probably reasonable but we can enhance this by estimating p̂ (preliminary win-rate) on a much smaller sample (30-50) which we can then use to calibrate the LLM judge by computing the agreement rate.

“Reliability measures the rate at which Harvey fails to provide the user with an answer for parsing reasons or because of unexpected model outputs.”

How do you measure “unexpected model outputs”?

Not explained in the blog post, but could feasibly include:

1. Schema/format violations — the model was expected to return output in a structured format (e.g., answer + reasoning + citation index), but instead returned malformed structure, missing fields, or extra unparseable text.
2. Invalid citation indices — the model points to a document index/offset that doesn't exist or is out of bounds, which would fail downstream parsing even if the "answer" text itself looks fine.
3. Refusals or non-answers — model declines to respond, or returns something like an apology/meta-commentary instead of the expected output type.
4. Truncated or incomplete generations — output cut off mid-structure (e.g., due to token limits) such that required fields never complete.
5. Encoding/formatting anomalies — unexpected characters, broken JSON, or malformed markup if the pipeline expects a specific serialization.

Most of these can probably be measured/detected by LLM as a judge or some simple heuristics.

“We measured quality through side-by-side preference testing.”

Is there any concern about just comparing the two of them and asking which one is better, as opposed to quantifying the degree of improvement? If I want to justify the significant resources dedicated to this project, the current setup (which doesn’t distinguish between narrow wins and big wins) seems like it makes it a bit difficult.

We could try a graded preference scale: "A much better," "A slightly better," "Tie," "B slightly better," "B much better." This is still a comparative judgment (avoids the calibration problems of absolute scoring across raters) but recovers magnitude information that pure binary throws away.

How would I convince a business leader that this project was worth doing?

I think some way to get a sense of the time-savings from the improvements would be helpful (maybe ask ALRs to estimate).

If we’ve had previous user research showing that latency is a driver of churn, reduced latency can be presented as a major win. It’s fair to assume that improved response quality (as detected by the ALR preference data) is good for the bottom-line (increased retention, positive word-of-mouth etc.).

How would you improve this system if you were building it today?

1. Citation verifications step (latency and cost trade-off though)
2. More structured reasoning trace
3. Selective reasoning depth depending on what kind of cell output is being considered
4. Task-based outcome metrics in evaluation (actual time-to-complete a review task, error/override rates in production)
5. Failure handling: if the primary pipeline produces an unparseable or low-confidence output, retry with a simpler/more constrained generation mode (e.g., force extraction-only, no reasoning) rather than surfacing a hard failure to the user
6. Tiered latency budgets by query complexity. Not all cells need the same processing depth; routing simple yes/no extractions through a faster path and reserving the full reasoning+citation pipeline for interpretive questions would reduce average latency without sacrificing quality where it matters

How does query understanding fit into the broader search pipeline? What are the remaining stages?

Piecing together Instacart’s other engineering posts, the pipeline is roughly:

1. Query understanding — normalization/spell correction, category classification, rewrites, SRL tagging
2. Retrieval/recall — hybrid: lexical (Postgres full-text search, GIN indexes, a customized ts_rank), embedding-based retrieval (their ITEMS bi-encoder over a FAISS ANN service), and category-based retrieval
3. Merge/fusion — reciprocal rank fusion or a weighted convex combination of lexical and semantic scores
4. Ranking — multi-objective, balancing “relevance, popularity, and personal preference”; the embedding score is a feature here
5. Serving layer — ads insertion, store-level availability filtering, dedup, blending, pagination
6. Post-serving — out-of-stock replacement suggestions, feedback logging

What kinds of “traditional machine learning models” have historically been used for search-related tasks at Instacart for the different use-cases they outlined?

[Post] Only two are named, both in the Fig 1 caption: query classification “relied on a FastText model for multi-label classification,” and query rewrites “were generated by a separate system that mined user session behavior” — which is a mining/statistics pipeline, not really a model.

[Post] For category classification the legacy setup is described as a “massive multi-class classification problem” predicting “the top-K most likely categories from a flat list.”

[Search] Adjacent Instacart posts name ITEMS (Instacart Transformer-based Embedding Model for Search), a Sentence-Transformers bi-encoder fine-tuned on their conversion data, and an earlier MiniLM-L3-v2 bi-encoder used for hybrid retrieval. Both are retrieval-side, not QU.

[Inferred] SRL is never given a legacy model. The pre-LLM standard for e-commerce query tagging is BIO sequence labeling — CRF, BiLSTM-CRF, or BERT token classification. Reasonable guess, unstated in the post.

Why would the methods they've historically used struggle with broad queries?

• No lexical bridge. FastText is a bag-of-n-grams linear classifier over the surface string. “healthy food” shares no characters with Yogurt, Produce, or Whole Grains. The model can only memorize the association from conversion data — it can’t derive it.
• The label distribution is genuinely flat and multi-modal. Top-K with a fixed threshold either cuts real categories (low recall) or admits noise (low precision). There’s no single correct answer to fit toward.
• It learns popularity, not semantics. Conversions for a broad query are dominated by whatever is merchandised or best-selling, so the model converges on “what people buy” rather than “what the query means.”
• No compositional reasoning. “healthy” is a constraint over attributes (low sodium, high fiber, no added sugar), not a category. Deriving that requires world knowledge the model doesn’t have.

“QU operates upstream and doesn’t benefit from direct feedback like clicks or conversions. The pseudo-labels we derive from user behaviors are inherently noisy.”

Wouldn't this be a problem at any layer of the stack, regardless of what methods you use?

It’s a matter of degree rather than kind — but the degree is large enough to change what’s possible:

• Ranking gets directly attributable labels. An item was shown at position 3 and clicked. The label sits on the same object the model scores.
• QU’s outputs are never shown to a user. A conversion is credit-assigned through retrieval → ranking → UI. If someone searches “bread” and buys bananas, you can’t separate a bad rewrite from bad retrieval from bad ranking from a user who simply changed their mind. That’s credit assignment through a long, non-differentiable pipeline.
• Confounding is worse upstream. QU’s own output determines the candidate set, so the feedback you observe is censored by the thing you’re trying to evaluate. Ranking has a version of this — position bias, selection bias from retrieval — which is why IPS and click models exist. But ranking has a label to debias; QU only has a proxy.
• QU labels are set-valued and partly subjective. “What are the correct categories for ‘healthy food’?” has no unique answer, which is why the fallback is human evaluation — expensive, and the reason the post reaches for LLMs at all.

“This heterogeneity introduced inconsistencies”

What kinds of inconsistencies?

The realistic failure modes:

• Semantic disagreement between components. The classifier maps “gluten free bread” to Bakery while the rewriter expands it to “sourdough” — which contains gluten. Two parts of QU assert contradictory things about the same query.
• Taxonomy version skew. Each model trained against a different snapshot; categories get renamed, split, or merged, so one model emits IDs another considers dead.
• Preprocessing skew. Different tokenization, casing, stemming, and spell correction per system, so “zip-lock” and “ziplock” are treated identically by one component and differently by another.
• Training-window skew. Rewrites mined over 30 days, classifier trained over 90 — different implicit definitions of “popular.”
• Metric incomparability. One team’s precision is human-labeled, another’s is a conversion proxy. You can’t reason about end-to-end quality because the numbers don’t compose.
• Training/serving skew duplicated N times, each with its own bugs.

“slowed down development cycles”

Why?

• N systems × (data pipeline + training + eval + serving + monitoring) is N times the maintenance and on-call burden, with no shared leverage.
• A change to a shared input — a taxonomy revision, a new catalog field — forces N retrains and N separate validations.
• Each system needs its own labeled eval set, and building one is the long pole. Every new task starts from zero.
• No transfer. Improving the rewriter’s handling of “organic” does nothing for the classifier. Effort doesn’t compound.
• Launching a new QU capability means a new bespoke model — a multi-quarter project — versus writing a prompt.

“Trained on diverse textual data, LLMs possess world knowledge that enables them to make logical inferences from user queries”

Isn't this also true for models like BERT, which I'm assuming were part of their historical pipeline?

Yes, BERT has world knowledge — but three differences are load-bearing:

• Scale. BERT-base is 110 million parameters trained on roughly 3 billion words. A frontier LLM is three to four orders of magnitude larger on both axes. “Italian parsley ≈ flat parsley” is exactly the kind of long-tail fact that only survives at that scale.
• Access mode — the decisive one. BERT’s knowledge lives in its representations, and the only way to extract it is a trained head on your labeled data. If you have no clean labels (Q4), you cannot get at it. An LLM exposes the same knowledge through natural-language instructions, requiring zero labels. That’s the whole unlock.
• Generation and steerability. Rewrites are a generative task — BERT can tag spans but cannot produce “curly parsley” as a substitute; it has no decoder over that space. And instruction-following is what makes “generate a broader query, not a synonym” expressible at all. That single capability is what makes the three-rewrite-type design in section 2 possible.

Short version: BERT has world knowledge. It doesn’t have label-free, generative, steerable world knowledge.

“We build data pipelines that retrieve and inject Instacart-specific context”

What are good ways to determine what context would be most helpful for a given task?

A practical method, in order:

1. Build an error taxonomy first. Sample 100–200 failures and label why each was wrong. Missing-knowledge errors (the model doesn’t know “verdant machine” is a brand) → context helps. Reasoning or format errors → context won’t help; prompting or fine-tuning will. This one step prevents most wasted effort.
2. The human-oracle test. Could a smart new hire with no Instacart data access answer this? If not, what would they look up? That lookup is your retrieval source. Cheapest, highest-signal step available.
3. Leave-one-out ablation. Build context as independent blocks — conversions, catalog, taxonomy, session — and ablate each against a fixed eval set. Keep only what moves the metric; every block costs tokens, latency, and adds distraction risk.
4. Prefer high-precision, low-volume context. Top-5 converted categories beats top-50. Irrelevant context measurably degrades LLM output, so more is not better.
5. Check for label leakage. If your context essentially contains the answer for head queries, your eval overstates gains on the tail — precisely where you need it to work.
6. Make sure it works cold. The tail queries you care about have no conversion history by definition. Include at least one source that works with zero engagement data — which is exactly why Instacart pairs conversion history with catalog embedding similarity.

“For the most advanced use cases, we fine-tune models on proprietary data”

Is there a good way to know in advance or in a less resource-intensive way whether fine-tuning is likely to be helpful for your use case before you dedicate the full set of resources needed for it?

Cheapest-first sequence:

1. The ceiling test. Can a frontier model with your best prompt and context solve the task well? If no, fine-tuning a smaller model almost certainly won’t — fine-tuning transfers behavior, it doesn’t invent capability. If yes, you have a teacher, and the problem reduces to a known-feasible engineering exercise. Instacart is exactly this case, which is why their story works.
2. Name what fine-tuning is for. Three distinct goals with very different success rates: (a) cost/latency — distill a known-good big model; high success. (b) format and style adherence — very high success, small data. (c) new knowledge — low success; RAG is usually the better tool.
3. Measure the few-shot slope. Run 0-shot → 5-shot → 30-shot. A steep improvement means the model learns this task from demonstrations, and fine-tuning is just many more demonstrations. A flat curve means the bottleneck is knowledge or capability, and more examples won’t fix it.
4. Run a small LoRA pilot. 1,000–3,000 examples, a few GPU-hours. The curve from 500 → 1,000 → 3,000 tells you whether more data pays. Hours, not quarters.
5. Check data feasibility before compute feasibility. Can you produce 5,000–10,000 high-quality examples without human labeling? If not, the project’s real cost is annotation, and that’s what usually kills it.
6. Check task stability. If the label definition churns monthly (taxonomy revisions), fine-tuning bakes in something you’ll repeatedly have to re-bake. Prefer RAG.

“Our legacy approach treated this as a massive multi-class classification problem”

Is every element of every level of the taxonomy hierarchy a class in this model? Would something be able to classified as both “Meat” and “Short Ribs”?

[Post] Yes to both, and the post’s own example proves it: for “butter milk” the legacy model predicts (“Dairy”, 0.95) and (“Milk”, 0.92) — explicitly described as “distinct, non-hierarchical outputs.” So the label space is flat and spans levels, and an ancestor plus its descendant can both be returned, because the model has no representation of the hierarchy.

Terminology flag: the post is internally inconsistent. The body says “multi-class”; the Fig 1 caption says “a FastText model for multi-label classification.” The behavior described — top-K, multiple simultaneous labels — is multi-label. Multi-class means exactly one label.

[Inferred] The pathology this creates: independent labels mean probability mass splits across redundant categories, the top-K budget gets spent on duplicates, and you get no signal about how specific to be. The new approach fixes this by predicting a category path (“the LLM’s predicted category path”), which is hierarchy-aware by construction — one output encodes both the level and the lineage.

“It directly powers recall and ranking, helping us retrieve items from the right categories and intelligently expand the search when a query is broad or ambiguous”

1. By “right categories”, do they mean they apply filtering?
2. How does search expansion work?

(a)

[Search] Instacart’s embeddings post lists “category-based retrieval” as a retrieval source alongside keyword and embedding retrieval.

[Inferred] Almost certainly not a hard filter in the general case. Two likelier uses:

• As a candidate generator — predicted categories trigger an additional retrieval pass (“fetch popular items in category C”), whose results merge with lexical and embedding candidates.
• As a ranking feature — item’s category is in the predicted set → boost.

Hard filtering on a QU prediction is dangerous: one classification error zeroes out recall for the entire query. Filtering is plausible only for high-confidence narrow queries, or in combination with the similarity guardrail.

(b) [Inferred] Expansion here means widening the candidate pool across taxonomy branches, not rewriting the text. For “healthy food,” many categories score highly, so you retrieve from each and blend — which is also what produces category-shelf-style result pages for broad queries. Note this is a different mechanism from the query rewriting in section 2, even though both get called “expansion.”

“First, being trained on noisy conversion data (e.g., a user searches “bread” but buys bananas) means it can produce irrelevant suggestions”

Shouldn’t the noise average out and thus be mitigated?

Averaging only rescues you if the noise is zero-mean and independent of the label. Neither holds here:

• The noise is systematic, not random. “bread → bananas” happens because users do multi-item shops in one session. The mislabeling correlates with what’s popular overall — bananas are among Instacart’s top items. So every query accumulates bias toward the same popular categories. More data makes that bias more confident, not smaller. This is the crux: you’re not averaging away noise, you’re converging on a biased estimate.
• There’s a feedback loop. The current system determines what’s shown → what converts → tomorrow’s labels. The noise is self-reinforcing, not i.i.d.
• Volume asymmetry. Averaging needs samples. Head queries have plenty, so it looks fine there. Tail queries — the entire target of this project — have a handful of conversions, so variance never washes out. This is exactly why the post pairs “noisy labels” with “data sparsity” as separate but compounding problems.
• Precision is the operative metric. Averaging shrinks mean error but doesn’t remove the tail of visible failures. A small persistent probability mass on a wrong category still surfaces to a user as a visibly bad result.

“we retrieve the top-K converted categories for each query as initial candidates”

Wouldn’t this struggle with low cardinality queries? Do they do any kind of rewriting or normalization to mitigate that?

[Post] Not addressed. No normalization or backoff step is described on the classification path.

[Inferred] This is a genuine gap, and arguably the weakest seam in the section. If a query has three historical conversions, the candidate set is tiny and possibly wrong — and a re-ranker cannot recover a category that was never in the candidate list. The LLM’s world knowledge is wasted if the retrieval step already threw away the right answer.

Standard mitigations, all of which Instacart plausibly has but doesn’t mention:

• Back off to a rewritten/normalized form. They already generate synonyms and broader queries — retrieve conversions for those instead. The post never connects the two systems, but it’s the obvious composition.
• Back off to embedding neighbors. Use the query embedding to find similar historical queries and pool their converted categories. They do exactly this on the SRL path (“brand names with high semantic similarity, ranked by embedding scores”), so the machinery exists.
• Back off to item-level signal. Retrieve items by text/embedding, then use those items’ categories as candidates. Works with zero query history.
• Let the LLM generate rather than re-rank when candidates are thin, with the guardrail catching invalid paths.

Note the structural asymmetry: the classification section is head-query-shaped, and the SRL section is where they explicitly solve the tail. The post presents both as “the LLM approach” without flagging that only one handles cold start.

“we use an LLM to re-rank them with injected Instacart context”

1. What context?
2. Do they use structured outputs? If not, how to ensure ranking output format is correct?

(a) For category re-ranking you’d want: candidate category paths with human-readable names rather than IDs; conversion counts or rates per candidate; a few exemplar item titles per category (this is what disambiguates a category whose name is vague); and the taxonomy’s own description of each node.

(b) In rough order of robustness:

1. Constrained decoding / grammar — vLLM guided decoding, Outlines, XGrammar. Invalid output is structurally impossible.
2. Provider-side JSON schema mode if using a hosted API.
3. Rank by index into the candidate list rather than emitting category strings. This makes hallucinating a nonexistent category impossible by construction and shrinks the output to a few tokens. For a re-ranking task specifically, this is the cleanest option and I’d expect it in a mature system.
4. Plain JSON prompt + parse + retry — weakest, but workable offline.

One clarification worth making: their guardrail (semantic similarity filter) is not a format check. It validates relevance, not schema. A malformed response fails before it ever reaches the guardrail.

Also: this is an offline/batch path, so retries are cheap and the format-safety pressure is far lower here than on the real-time SRL path.

“finally, we apply a post-processing guardrail. This filter computes a semantic similarity score between the embeddings of the original query and the LLM’s predicted category path, discarding any pair that falls below our relevance threshold.”

What model do they use for this score? Wouldn’t it struggle for the same reasons their original approach did?

[Search/Inferred] Two obvious in-house candidates: ITEMS (their Sentence-Transformers bi-encoder fine-tuned on conversion data) or the older MiniLM-L3-v2. ITEMS is the likelier one — already trained on query↔item pairs.

[Inferred] On the objection — the guardrail mostly escapes the original failure modes, for three reasons:

• Much easier task. The legacy classifier had to choose among thousands of categories. The guardrail only has to reject an obviously unrelated pair. A crude, low-resolution filter can be valuable where a crude classifier is not.
• Decorrelated errors. The LLM fails by hallucinating plausible-sounding but irrelevant categories; those are usually lexically and semantically far from the query, which is exactly what an embedding catches. The failure modes don’t overlap, which is what makes the two-stage design work.
• Tunable in the safe direction. Set the threshold loose so you only kill clear garbage. You’re not asking it to be right; you’re asking it to be right about egregious cases.

But the instinct is partly correct in one place: if the guardrail’s embedding model was trained on the same noisy conversion data, it inherits the same popularity bias — and may wrongly reject correct but rare query-category pairs. That would silently degrade exactly the tail queries this whole project exists to fix. The post doesn’t discuss this, and it’s the risk worth measuring before shipping.

“Our legacy system mined candidate rewrites from user session data”

What does this mean?

[Inferred] The standard technique is logging within-session query reformulation pairs. A user searches q₁ = “gluten free bread,” gets poor results, reformulates to q₂ = “gf bread,” and converts. You record (q₁ → q₂). Aggregate over millions of sessions, score by frequency and by the conversion lift of q₂ over q₁, then threshold.

Common variants:

• Full reformulation chains within a session, not just adjacent pairs
• Click/purchase-graph co-occurrence — two queries that lead to the same items are related
• Abandon-then-succeed pairs — highest signal, because the user has effectively labeled q₁ as insufficient

[Inferred] Why it caps near 50%: it can only produce a rewrite for a query that (a) someone has typed before and (b) was followed by a reformulation. Novel and tail queries have neither, by definition. It’s a memorization system with zero generalization — which is precisely the gap an LLM fills.

“This proved too ambiguous.”

How would they assess whether it was too ambiguous?

[Inferred] That reads like a qualitative, eyeballed finding, not a measurement. The rigorous versions:

• Type-distribution audit. Sample N queries, have raters label each generated rewrite’s type (synonym / substitute / broader / other) and usefulness. If output collapses onto synonyms — say 70% when you needed a spread — the instruction is under-specified. This directly diagnoses “ambiguous.”
• Incremental recall — the cheap quantitative version. How many items does the rewrite retrieve that the original didn’t? “one percent milk” adds roughly zero. No humans needed, and it operationalizes “not useful” exactly.
• Output-diversity check. Run the same prompt repeatedly at temperature; tight clustering means the model has silently picked one interpretation of an ambiguous instruction.

The generalizable lesson: “generate rewrites” doesn’t specify an objective. Splitting into three types is really specifying three separate objectives so each becomes individually evaluable — the evaluability is arguably the bigger win over the quality.

“This led us to design specialized prompts for three distinct rewrite types”

Are these three rewrite types all done every time for every query or is each of them triggered by its own unique conditions being met?

[Post] Not stated. It says each type is “handled by a dedicated prompt,” and that coverage rose above 95% “with 90%+ precision across all three types” — implying all three are generated broadly.

[Inferred] The likely design is generate all three offline, trigger conditionally at serve time:

• Generation: run all three across the precomputed query set. You can’t know in advance which you’ll need, and offline compute is cheap. “Coverage over 95%” is a claim about the generated inventory, not about firing rate.
• Serving: the post says rewrites matter “especially when the original query does not return sufficient results.” So the natural trigger is result-count or result-quality based. If the original returns enough good items, use nothing. If thin, escalate in order of intent-drift risk: synonyms first (safest, preserves intent) → broader (widens, some drift) → substitutes (changes the product, riskiest).

That escalation ladder is inference, not theirs — but it’s the only ordering that makes sense given the risk profile of each type.

“increased our query rewrite coverage to over 95%”

This feels like a very gameable metric in isolation?

Note they do pair it — [Post] “over 95% with 90%+ precision across all three types.”

[Inferred] Why coverage alone is nearly meaningless here: an LLM emits something for any input. Coverage measures the model’s willingness to answer, not quality. You can drive it to 100% by removing your abstain path. The legacy 50% was low because a mining system genuinely couldn’t produce candidates; an LLM’s pre-filter coverage is ~100%. So the honest framing is “50% from mining” vs “95% surviving guardrails” — and the interesting number is how much the guardrails removed, which they never report.

What would make it credible:

• Coverage on the tail specifically. Head coverage was never the problem, and an aggregate number is dominated by head volume.
• The coverage/precision curve as the guardrail threshold sweeps. A single point on that curve is unfalsifiable — you can always pick the flattering point.
• Incremental recall and conversion lift on queries where a rewrite actually fired.
• Null-result-rate reduction — the thing rewrites exist to fix, and the metric a business would actually pay for.

Coverage is an input metric. Nobody cares about it if it doesn’t move an output metric.

“90%+ precision across all three types”

How is this measured? Example?

Given the post’s own claim that QU lacks “direct feedback like clicks,” this is almost certainly human evaluation on a sampled set — possibly LLM-as-judge calibrated against human labels. The likely protocol, with the type-specific question a rater would answer:

• Synonym — “Would a shopper consider these the same product request?”
  “1% milk” → “one percent milk” ✓ · → “skim milk” ✗ (different product)
• Substitute — “If the original were unavailable, is this an acceptable alternative?”
  “Italian parsley” → “curly parsley” ✓ · → “cilantro” ✗ (different flavor profile)
• Broader — “Is this a strict generalization that still contains the original intent?”
  “2% reduced-fat chocolate milk” → “chocolate milk” ✓ · → “beverages” ✗ (too broad) · → “whole milk” ✗ (sibling, not parent)

Precision = rewrites judged correct ÷ rewrites emitted, computed per type. Recall isn’t reportable because there’s no enumerable ground-truth set of all valid rewrites — which is exactly why they fall back to coverage as a recall proxy, and why Q19’s critique lands.

[Inferred] Important caveat: precision defined this way measures semantic correctness, not usefulness. “one percent milk” would pass a synonym-precision check while being precisely the failure they described one paragraph earlier. The metric they report does not capture the problem they say they solved.

“we are now adopting context engineering to make rewrites more convertible, personalized, and session-aware”

What would those improvements look like in practice?

[Post] One concrete mechanism: “injecting user engagement signals, such as the top-converting product categories from their subsequent searches in the same session.”

[Inferred] Unpacking each:

• Convertible — bias rewrites toward products that actually sell and are actually in stock at the user’s selected retailer. Concretely: score candidate rewrites by the historical conversion rate of the items they retrieve, and gate on store-level availability. “Italian parsley → curly parsley” is worthless if that store stocks neither.
• Personalized — condition on purchase history and dietary pattern. “milk” from a user who has only ever bought oat milk should rewrite toward plant-based, not dairy. Same for brand affinity: a repeat buyer of one brand gets rewrites that preserve it.
• Session-aware — condition on queries already issued this session. Someone who searched “ground beef,” then “ricotta,” then “noodles” is making lasagna, so “noodles” should rewrite toward lasagna sheets, not ramen. This is the same idea as the post’s closing ambition of separating “lasagna ingredients” from “quick lasagna recipe.”

[Inferred] The cost, which the post doesn’t mention: all three push work from offline-and-cacheable to per-request, because the context is now user- and session-specific. That breaks the head-query cache — the exact mechanism that makes their economics work (Q22, Q29). That’s very likely why this is framed as future work rather than shipped.

“populating a cache for our most common “head” queries”

How to determine which queries? What threshold? Is it dynamic/time-windowed?

[Post] Never specified. Only: “the power-law nature of search traffic,” and — crucially, buried in the takeaways — “smart caching meant only 2% of queries needed real-time inference.”

[Inferred] That 2% is the answer in disguise: the cache is sized to cover 98% of query volume. The natural construction:

• Selection. Rank distinct queries by frequency over a rolling window; take the prefix covering 98% of impressions. Under a power law that’s a small fraction of distinct strings covering the overwhelming majority of traffic.
• Window. Should be rolling (30–90 days) and refreshed on a batch schedule, or it can’t track seasonality — “pumpkin spice” is head in October and tail in April.
• Dynamic promotion. Log cache misses; any query crossing a miss-count threshold gets queued for the offline teacher pipeline and promoted on the next run. This is what keeps the real-time model’s load pinned near 2% as traffic shifts.
• Key normalization. The cache key should be normalized text (lowercase, whitespace, punctuation, probably spell-corrected), or hit rate craters on trivial variants.
• Invalidation is the hard part they don’t mention: entries must expire when the taxonomy, catalog, or prompt changes, or the cache silently serves stale tags indefinitely. This is usually the biggest operational liability in a design like this.

What model used for Teacher and why?

[Post] Never named. Only “a powerful offline process,” “much larger frontier model,” “massive LLMs,” and Fig 5’s unlabeled “baseline.”

[Inferred] For work published November 2025, the plausible set is a GPT-4o/4.1-class, Claude Sonnet-class, or Gemini Pro-class hosted model, or a large open model (Llama-3-70B/405B) run in-house. Choosing Llama-3-8B for the student mildly suggests comfort with that family, but a hosted frontier API is the more common teacher because batch pricing makes it affordable.

Why a big model as teacher: you need the teacher’s quality to be the ceiling you distill toward. You only pay for it on head queries, in batch, with no latency constraint — so it’s expensive per call but amortized across a cached result serving millions of impressions. That amortization is the entire economic argument for the architecture.

Is their approach equivalent to standard teacher-student distillation or something different? Why this fine-tuning approach vs another?

It is not classical knowledge distillation, and the difference is worth being precise about:

• Classical KD (Hinton et al.) trains the student on the teacher’s full output distribution — soft logits, KL loss, temperature. Requires logit access and a shared tokenizer/vocabulary.
• What Instacart did is sequence-level distillation — take the teacher’s final generated text as a hard label, train with ordinary cross-entropy. Colloquially everyone calls this “distillation,” but structurally it’s supervised fine-tuning on machine-generated labels.

Why that variant:

• It works through an API. No logits needed, so the teacher can be a closed frontier model.
• It’s tokenizer- and architecture-agnostic. Teacher and student need nothing in common.
• The training code is just standard SFT. No custom loss, no distillation infrastructure.
• The decisive one: the teacher’s advantage comes largely from injected RAG context. Hard-label SFT is how you bake that retrieved context into the student’s weights — so the student doesn’t need to run retrieval at inference. That is what buys the latency. The student isn’t just a smaller model; it’s a model that has internalized a retrieval step.

One refinement worth naming: [Post] the teacher’s outputs pass “a post-processing guardrail [that] validates the tags against our catalog” before becoming training data. So it’s filtered / rejection-sampled distillation, which meaningfully raises label quality over raw teacher output.

Why LoRA over full fine-tuning [Inferred]: far less memory, hours instead of days, cheap to sweep variants, trivial rollback (keep one base, swap adapters). The cost is a lower capacity ceiling — which didn’t bind here because the task is narrow.

How did they know in the first place that the Teacher LLM works well for this task? Who evaluates the teacher?

[Inferred] The standard answer, and near-certainly what happened:

• A human-labeled golden set. A few hundred to a few thousand queries with expert-annotated SRL tags. Expensive — which is why it stays small — but you build it once and it anchors everything downstream. This is exactly the “costly and time-consuming human evaluation” the post cites in its challenges section.
• Stratified by traffic band (head/torso/tail) and query type (brand-led, attribute-heavy, ambiguous). A teacher that’s excellent on head and weak on tail is useless here, since the tail is the whole point.
• The catalog guardrail as an unsupervised check. Tags that fail catalog validation are wrong by construction — a cheap, large-scale lower bound on teacher error, no humans required.
• Downstream proxy. Do teacher-produced tags improve offline retrieval/ranking metrics? Noisier, but scales.
• The A/B test is the final arbiter, but it validates the system, not the teacher in isolation.

The structural problem worth sitting with: Fig 5 compares the student against the frontier model — i.e. the student is scored against the teacher, not against ground truth. If the teacher has a systematic blind spot, the student inherits it perfectly and the chart still shows 95.7% F1. “Matches the teacher” and “is correct” are different claims, and only a golden set separates them. They report the first and imply the second.

“For the real-time SRL model, we fine-tuned an open-source Llama-3–8B model using LoRA (Low-Rank Adaptation). The model was trained on the dataset from the offline “teacher” pipeline.”

What does this fine-tuning process look like concretely? How do I set up the necessary tools, datasets, infrastructure etc for this?

[Inferred] Concrete recipe as I’d run it today:

1. Data. JSONL of {messages: [system, user, assistant]} — user = the query (plus any context you want internalized), assistant = the JSON tag output. Target 5,000–50,000 examples. Hold out 5–10%, stratified by query frequency band so you can read tail performance separately. Dedupe aggressively; near-duplicate head queries will otherwise dominate the loss.

2. Framework, easiest → most controllable:

• Axolotl — YAML config, batteries included, best default choice
• Unsloth — fastest single-GPU (~2× speedup), ideal for 8B on one card
• HuggingFace TRL SFTTrainer + PEFT — most transparent, best when you need to customize
• LLaMA-Factory — good UI/CLI, broad model support

All wrap the same underlying PEFT LoRA implementation.

3. Hyperparameters, in order of impact:

• r = 16 (8–32 is plenty for narrow extraction); lora_alpha = 2r
• Target all attention and MLP projections (q, k, v, o, gate, up, down). Targeting only q,v is the old default and it underperforms — this is the single most common mistake.
• LR 1e-4 to 2e-4, cosine decay, 2–3 epochs, bf16
• Mask the loss to completion tokens only — don’t train on the prompt

4. Hardware. One 80GB A100/H100 fits 8B LoRA comfortably at bf16 with gradient checkpointing. On 24GB, use QLoRA (4-bit base via bitsandbytes) — quality is close for extraction tasks. Cloud: Lambda, RunPod, Modal, or SageMaker. A run at this scale is hours and tens of dollars, not a capital expense.

5. Eval, tracked per epoch. Three metrics: per-field F1 and exact-match on held-out, catalog-validation rate, and JSON parse rate. Parse rate should reach ~100% within one epoch — that itself confirms format learning is done and remaining error is semantic.

6. Merge and serve. peft’s merge_and_unload() → save as a plain HF model → serve with vLLM or TensorRT-LLM. Benchmark p50/p95/p99 at realistic concurrency, not batch-size-1 — that gap is where latency projects die.

7. Guarantee the schema in production. vLLM guided decoding (XGrammar backend) or Outlines, so parse failures become structurally impossible rather than merely rare.

“Merging the LoRA adapter weights directly into the base model”

What does this mean?

[Inferred] LoRA trains two small matrices per targeted weight — A (r×k) and B (d×r) — and inference computes h = Wx + BAx: the base projection plus a low-rank correction on a separate side path. Since BA has the same shape as W, you can precompute W' = W + BA once and store it. Inference becomes h = W'x — a single matmul, identical in cost to the unmodified base model.

Why it’s meaningfully faster:

• Eliminates two extra small matmuls per targeted module, per layer, per generated token. That’s roughly 7 modules × 32 layers × every token.
• Small matmuls are launch-overhead-bound, not FLOP-bound — so the cost is wildly disproportionate to the tiny parameter count. This is why 30% is plausible from a change that touches about half a percent of the weights.
• Removes the branching side path, which unblocks fused kernels and CUDA graph capture in the serving stack.

What you give up: you can no longer hot-swap adapters or serve many fine-tunes from one base-model copy in memory. A real cost in a multi-tenant setup; irrelevant here, since they serve exactly one.

Numerically it’s exact — merging changes nothing about model outputs beyond float rounding. It’s a free win when you don’t need adapter swapping.

What is typical acceptable latency in search systems like theirs? Given that they’re only focused on the first part of the pipeline, what latency budget are they allocated?

[Post] Their numbers: ~700ms out-of-the-box on A100 → 300ms target, achieved via adapter merging + H100. FP8 quantization would have cut another 10% (rejected for a recall drop). End-to-end impact: “only a marginal latency increase.”

[Search/Inferred] Industry context:

• E-commerce whole-page search typically budgets ~200–500ms p95 end-to-end; past ~1s, conversion loss is measurable. The canonical citations (Amazon’s “100ms costs 1% of sales,” Google’s 500ms/20%-traffic result) are old and over-quoted, but they’re why everyone budgets tightly.
• Within that, retrieval is usually tens of ms — [Search] Instacart’s own embeddings post cites under 8ms for on-the-fly query embedding — ranking is tens of ms, and the remainder is network, ads, and rendering.

[Inferred] The key structural point: 300ms for a single QU component is enormous — larger than many teams’ entire backend budget. It is only affordable because of the cache. At a 98% hit rate, the average added latency is about 0.02 × 300ms ≈ 6ms, which is why “marginal” is honest rather than spin. The 300ms is a tail cost paid by 2% of queries — and those are cold-start queries where the alternative was bad results anyway, so the user’s willingness to wait is higher and the regret from getting it wrong is larger.

That trade — accept a large latency hit on a small, high-regret slice — is the actual transferable design insight here, more so than the 300ms number itself.

Is “average scroll depth” just the number of items users scrolled past before clicking?

[Post] Defined only parenthetically: “(users find items faster).”

[Inferred] Roughly yes, but the exact definition changes the interpretation, and they don’t give it. The three common variants:

• Rank of first engagement — position of the first item clicked or added to cart. Lower is better. Most likely what’s meant, given “find items faster.”
• Max scroll depth — deepest item position rendered during the session, clicks or not. Measures effort, but conflates “found it fast” with “gave up fast.”
• Impressions before first add-to-cart — counts items seen rather than rank; handles grid layouts and lazy loading more cleanly.

Why the ambiguity matters: −6% is clearly good under the first reading. Under the second it’s ambiguous — abandonment also reduces scroll depth. A user who sees garbage and leaves immediately scrolls very little. Ruling that out requires pairing it with a conversion or engagement-rate metric. The complaints figure partially covers it, but not cleanly.

[Inferred] A grocery-specific wrinkle: users often scroll deliberately to browse alternatives even when results are excellent (“what yogurts do you carry”). That makes scroll depth a noisier relevance proxy in grocery than in, say, a support search box — worth knowing if you ever borrow the metric.

How would I convince a business leader that this project was worth doing?

Cost

• Reduced overhead from mode consolidation?

Benefit

• More conversions -> more revenue

How would you improve this system if you were building it today?

[Inferred] throughout, roughly ordered by expected value:

1. Build the golden eval set first. The biggest gap in the entire writeup (Q26). Without human-labeled ground truth, the teacher is unvalidated and the student is only ever measured against it. A few thousand stratified, expert-labeled queries is the highest-leverage artifact in the system — it makes every downstream decision cheap and every claim falsifiable.

2. Unify the three QU tasks into one model. Their own takeaway is “consolidate, don’t complicate,” yet they still ship three separate paths with different architectures. One fine-tuned model emitting categories + tags + rewrites in a single structured response is one forward pass instead of several, and it eliminates the cross-component inconsistency they named as the original problem (Q5). Multi-task training on a shared backbone usually helps each task, since they draw on the same underlying understanding.

3. Close the loop from serving back to training. Log cache misses, guardrail rejections, and null-result queries; feed them to the teacher on a schedule; retrain the student periodically. Prioritize queries where student and teacher disagree — that’s where the information density is. Their architecture is a one-way pipeline; making it a cycle is cheap and compounds.

4. Replace the exact-match cache with a semantic cache. String matching misses “gluten free bread” / “gluten-free bread” / “glutenfree bread.” Embed the query, ANN-lookup against cached keys, accept above a conservative threshold. They already run an ANN service. Higher hit rate translates directly into lower GPU spend.

5. Test whether you need 8B at all. A 1–3B model (Llama-3.2-3B, Qwen-3-4B class) may match on a task this narrow, at a fraction of the latency and cost. One experiment either saves real money or justifies the 8B — both outcomes are worth having.

6. Add abstention / confidence. The student always answers. Fig 5’s 96.4% precision means roughly 1 in 28 outputs is wrong with no way to identify which. A model that can say “unsure” lets you route those to a fallback — broader retrieval, or a synchronous call to a larger model with a longer budget — instead of confidently mis-tagging.

7. Audit the guardrail for popularity bias (Q15). Check whether the similarity filter disproportionately rejects correct-but-rare pairs. If so, make the threshold frequency-adaptive, or replace pure embedding similarity with catalog-grounded validation — “does any real item match this category path?” — which they already do on the SRL side.

8. Instrument the business metric. Conversion and null-result rate on the affected slice, in the A/B readout.

9. Modernize serving. Built today: server-side guided decoding, speculative decoding (SRL outputs are short and highly patterned, so draft-model acceptance rates should be excellent), and continuous batching in vLLM would likely land under 300ms without the H100 upgrade — which changes the hardware budget conversation entirely.

10. On their stated direction — session-aware, multi-intent — it’s right, but it breaks the cache (Q21). I’d resolve that by splitting rather than merging: keep query-level tags cached and stateless, and add a separate, small, cheap session-intent model whose output is combined at serve time. Putting session state inside the cached artifact destroys the 98% hit rate that makes the economics work.

“Attackers may launch attacks from compromised accounts making them even harder to identify from regular email”

What signals can you actually use if this happens? IP address, communication style, etc?

Per-sender behavior — writing-style deviation against that sender's own history, send-time-of-day shift, sudden send bursts, recipients never previously contacted

Mailbox configuration — new forwarding rules; rules that auto-delete or bury replies in Archive/RSS (the classic tell — the attacker hides the victim's incoming replies); delegate and permission changes.

Authentication / session — sign-in geography and ASN, impossible travel, new device fingerprint, legacy-protocol auth (IMAP/POP, which bypasses MFA), OAuth app consents

The IP address situation is a bit more complicated. It’ll be the same as the regular account holder (and thus not usable) if the attacker uses webmail or the mobile app. It’ll be different (and thus usable) if the attacker uses stolen credentials with an SMTP client which is much rarer.

“are identity of parties in the communication often targeted?”

What does this mean?

The intended reading: is this person's role one that attackers commonly go after? It's a prior on the identity, not a property of the message.

Role targeting — CEO/CFO (impersonated in BEC), AP clerks (invoice fraud), HR/payroll (W-2 and direct-deposit diversion), IT helpdesk (credential resets), executive assistants (authority by proxy)

“1 in 100,000 emails is advanced spear-phishing“

How was this estimate generated?

How a number like this necessarily gets produced: numerator = attacks their own engine caught, plus customer-reported and analyst-confirmed cases, plus retro-hunts. Denominator = total messages processed. So it's an observed detection rate, not true prevalence — a lower bound, because false negatives are by construction uncounted.

Three things that move it a lot:

• Denominator choice. Abnormal sits behind the existing gateway (API, post-delivery), so its denominator is already spam-filtered mail. That inflates the rate relative to raw internet email — the 65-in-100 spam is already gone.
• Definition. "Advanced spear-phishing" is their category, no external standard.
• Customer mix. Their customers are enterprises that bought advanced email security — plausibly more targeted than average.

“Account sign-ins, mail filters, and other account activity”

Do we typically have access to all of this information?

Yes, for the API-based model. Microsoft 365 exposes it through Graph API plus the Office 365 Management Activity API: Entra ID sign-in logs, mailbox audit logs, inbox-rule create/modify events, message content and headers, mailbox permissions and delegates, OAuth consents. Google Workspace has equivalents (Admin SDK Reports API for login/user-account audit, Gmail API).

Two caveats that matter:

• Licensing and consent. Full sign-in log retention depends on the customer's Entra ID tier, and you only get the scopes the admin consented to at install.
• Lag. Audit events surface with delay — historically minutes for the Management Activity API. So they are not available at the sub-second scoring moment for the message in hand. They feed the account-takeover detector and the feature store, not the inline decision. This connects directly to Q17.

“Malware in attachments”

How is this identified?

Four methods, cheapest to most expensive:

1. Hash / reputation — SHA-256 against known-bad corpora (VirusTotal, internal). Milliseconds; catches only knowns.
2. Static analysis — OLE/VBA macro extraction from Office files; PDF JavaScript, embedded files and launch actions; recursive archive unpacking; YARA rules; ML classifiers over PE headers and section entropy. Catches variants; defeated by packing.
3. Dynamic detonation — run the file in an instrumented sandbox (CAPE/Cuckoo lineage; commercially Joe Sandbox, VMRay) and watch process spawns, registry writes, network callbacks. Catches unknowns; takes 30 seconds to several minutes, and evasion-aware malware sleeps or checks for VM artifacts.
4. Multi-engine AV — several commercial engines in parallel.

“Adversarial attackers — To make matters worse, attackers actively manipulate the data to make it hard on ML models:”

For each of the methods listed here, what is one counter strategy each?

“The precision must be very high — to build a product to prevent email attacks we must avoid false positives and disruption of legitimate business communications”

Can we have different thresholds for different remediation severities (quarantine, block, etc)?

Yes, and this is standard practice: decouple score from action. The model emits a calibrated probability; a separate policy layer maps score bands to actions. Thresholds should be monotonically increasing in action severity, because each action has a different FP cost:

Three second-order points:

• Vary by attack class too. A suspected payroll-diversion BEC against the CFO justifies a lower bar than a generic promo-shaped phish, because the FN cost differs by orders of magnitude.
• Vary by recipient. Lower thresholds for finance, AP, and executives.
• The post-delivery architecture makes this much cheaper. Because Abnormal acts after delivery via API, it can decide late and revise — retracting a message when link detonation finishes or when the same campaign gets confirmed at another tenant. An inline gateway must decide once, immediately, with whatever it has.

One dependency: bands are only meaningful if the score is calibrated to an actual probability

“Is this pattern of communication unusual in any way?”

What are a couple of examples of stuff that would get flagged here?

Six kinds, organized by what's anomalous about the graph edge:

1. Edge never existed — first-time sender to this recipient, especially one that immediately asks for an action.
2. Edge exists, wrong direction or role — this vendor has always emailed AP, now it's emailing the CEO. Or the CEO mails a junior employee three orgs away they've never contacted.
3. Edge exists, properties changed — same display name, different sending domain than the previous 400 messages from that vendor. Or same person, new reply-to.
4. Volume/timing — an internal account suddenly mails 200 external recipients; a user sends at 3am local when they never have.
5. Missing expected structure — a "reply" that belongs to no thread the recipient has; a vendor invoice arriving off its normal monthly cadence.
6. Tenant-level shape — a domain that has never touched this organization now contacting five finance employees at once. That's campaign shape, invisible at the single-message level.

“Our ensemble of detection models”

Why ensemble as opposed to single model?

Five reasons, roughly in order of how much they matter here:

1. Adversarial robustness. A single model is one thing to reverse-engineer. The post says attackers A/B-test against filters — if they find the perturbation that flips your one model, they win everywhere. With detectors keyed on different data (text vs. graph vs. identity vs. links), evading all of them at once is much harder, and the evasion that beats one usually trips another.
2. Heterogeneous sub-problems. Invoice fraud, credential phishing and lateral ATO phishing have different base rates, signals and label volumes. A single model trained on all of it is dominated by the common class and underfits the rare, expensive one.
3. Precision-budget arithmetic. At a 1-in-a-million FP target, you can OR together several high-precision, low-recall detectors and accumulate recall without any one of them being perfect. A single model has to hit that operating point alone.
4. Operational independence. Ship and roll back separately, attribute failures to a component, patch a novel attack in hours with a signature while retraining runs for weeks.
5. Variance reduction. The textbook decorrelated-errors argument — the least important reason in this setting.

What do they mean by “redundant detectors”? Why do they use it and what does that look like in practice?

Redundancy ≠ ensembling. Ensembling combines weak signals into a better score. Redundancy means deliberately overlapping coverage: more than one independent detector capable of catching the same attack on its own. It's defense in depth for the classes where a miss is catastrophic.

Why: if detector A fails — new evasion, model drift, a feature-pipeline bug, a bad deploy — detector B still fires. The failure modes are meant to be uncorrelated.

For invoice fraud specifically, the stack would look like:

• The multi-modal deep model scoring the message end-to-end
• A dedicated invoice sub-model over attachment OCR and invoice layout
• A stateful check, not a model: does the bank account on this invoice match what this vendor has historically been paid to?
• Analyst heuristics — "reply-to ≠ from AND first-contact sender AND banking keywords AND urgency"
• Signatures on known campaign infrastructure

“for the convenience of including multiple modalities of data inside the same model”

A GBDT could also take in text and image embeddings as features though, right?

Yes — mechanically nothing stops you. The post's stated reason is convenience, and it undersells the real difference.

Four differences, and the first is the substantive one:

1. Gradient flow. Feed a GBDT frozen embeddings and the text encoder never learns from your detection labels. The figure shows the NLP sub-model's "Text Embedding" feeding a joint Neural Net — gradients propagate back into the encoder, so it learns representations tuned to this task (phishing register, urgency, payment language) rather than generic semantics. Trees cannot backprop into an encoder. This is the whole ballgame.
2. Dimensionality. A 768-dim embedding is 768 axis-aligned split candidates to a tree. Embedding information lives in directions, not coordinates, and trees split one coordinate at a time. GBDTs are weak exactly in the dense high-dimensional continuous regime. Note the converse: on genuinely tabular features GBDTs usually still beat NNs — which is why the figure retains a separate Tabular Sub-Model.
3. Variable-length data. An email has 0–N links, 0–N attachments, 0–N images. A tree needs a fixed-width row, so you hand-pool ("max link risk," "mean image score") and throw information away. A network can pool or attend in a learned way.
4. Engineering — the post's stated reason. One model, one training loop, one serving artifact, instead of a pipeline of separate embedders whose versions must be pinned and re-materialized every time they change. That last part is precisely the pain in his other Abnormal post on re-scoring past attacks.

Honest counterpoint: GBDT + frozen embeddings is a genuinely strong baseline, is what the post recommends starting with, and is cheaper to serve. The upgrade only earns its keep once you have enough positive labels to fine-tune — and labels are scarce here because of the base rate.

“we must be very careful that the model performs well autonomously and can handle distributional shifts”

How do they ensure this happens?

Pre-deployment

• Calibrate the score to a real probability (Platt/isotonic), correcting for negative subsampling
• Backtest on held-out future time slices, never random splits. Random splits leak badly here because campaigns repeat within a window.
• Re-score historical attacks with today's pipeline to confirm the new model still catches old ones.

At deployment

• Shadow mode: score live traffic, take no action, compare against production.
• Gradual rollout by tenant with a pre-agreed abort criterion.

Monitoring

• Watch the inputs, not just outputs: per-feature distribution drift (PSI/KL), null-rate spikes, cardinality changes. Feature-pipeline bugs look identical to genuine drift and are far more common.
• Watch firing rate per tenant at a fixed threshold — the earliest available warning.
• Asymmetry to plan around: precision is observable within days (customers report false positives); recall is not observable at all. You need proxies — analyst-labeled samples, purple-team and simulated attacks, cross-tenant confirmation.

Structural mitigations

• Per-tenant normalization. Features expressed relative to an org's own baseline transfer across orgs far better than absolute features.
• Keep the heuristic/signature layer, which can be updated in hours when the model degrades.

“Study the attacks, trends, and false negatives”

What are the sources for false negatives?

Seven channels, by how you learn about the miss:

1. End-user reports — the "report phishing" button in Outlook/Gmail. Highest volume, lowest quality.
2. Customer SOC escalation — the security team finds it in their own hunt and forwards it.
3. Incident-driven — post-breach investigation after money moved or credentials were used. Lowest volume, highest cost, most informative.
4. Retro-hunting on new intelligence — you learn a domain or hash is malicious later and sweep historical mail. Includes cross-tenant: an attack caught at customer B lets you re-search A and C. Abnormal's later material calls this "federated intelligence."
5. Delayed signal — the URL was benign at delivery and weaponized afterward (benign-then-swap). Only re-crawling links post-delivery surfaces these.
6. Threat-intel feeds and industry sharing — ISACs, vendor feeds, public reporting.
7. Phishing simulation vendors (KnowBe4, Proofpoint Security Awareness) — attacks in your stream with known ground truth.

The bias to name explicitly: every channel is biased toward attacks that were noticed. The FNs you never learn about are the successful ones never attributed to email at all. Measured recall is therefore optimistic by an unknown and unmeasurable amount — which is the deeper reason the post's base-rate figures in Q3 should be read as lower bounds.

“Build a portfolio of detection models, heuristics, and signatures”

What does signature mean here?

A signature is an exact-match rule keyed to a specific known artifact. It fires on identity, not on learned pattern — the narrowest, most brittle, most precise layer of the stack.

In email that means: a SHA-256 of a known-bad attachment; an exact URL, domain, or IP of known phishing infrastructure; a YARA rule matching a byte pattern inside a file; a specific sender address or subject string from an observed campaign; a structural fingerprint of a campaign's HTML template.

Why an ML-first company keeps signatures at all: response time and explainability. When a campaign is live right now, you push a signature in minutes; retraining takes weeks. And when a customer asks "why was this blocked?", a signature has a one-line answer.

“Ensure each sub-model is representing its sub-problem”

What are examples of sub-models representing sub-problems in this context?

All four feed a joint Neural Net → "Is Attack?"

[I] The design point that's easy to miss: the sub-problem is the modality, not the attack type. Short text is split from long text because a display name and an email body need different tokenization and different encoders — a subject line has no sentence structure to model, and a 24-character display name in a BERT-shaped encoder is mostly padding.

[S] The lesson in full: "Ensure each sub-model is representing its sub-problem well before trying to combine them into a larger network, or you will never make progress." Meaning: validate each branch in isolation — can the Link Sub-Model alone separate phishing URLs from benign ones? — before wiring it into the joint net. Otherwise a broken branch is invisible inside a joint loss, and you can't tell whether poor performance is an architecture problem or one garbage input.

[I] Sub-problems you'd expect beyond the figure: an OCR/vision model over attachment and landing-page images, an invoice-layout model, a per-sender writing-style model, and a communication-graph embedding.

“combine results of all models for a final decision on whether the email is malicious”

What would this look like in practice?

Four combination mechanisms, which are not mutually exclusive:

1. In-model fusion — the figure's approach. Sub-models feed a joint neural net; combination is learned, inside the network, and there's only one score at the end.
2. Stacking — each detector emits a score; a small meta-model (logistic regression or shallow GBDT) takes those scores plus context (tenant, attack class) and outputs the final probability. Needs clean held-out data or it leaks.
3. OR-of-thresholds — each high-precision detector has its own threshold; malicious if any fires. This is what "redundant detectors" implies, and it's the natural fit for recall-through-redundancy. Simple and explainable, but FP rates add.
4. Policy layer on top — score bands map to actions (Q7), plus overrides: allowlists, VIP-recipient escalation, tenant policy, and "if signature X fired, act regardless of score."

Realistically it's a hybrid: a learned score from the core multi-modal model, OR'd with a small set of high-precision specialist detectors and signatures, then a policy layer choosing the action.

One design constraint worth stating: attribution matters operationally. Customers ask why a message was actioned, and SOC analysts need to triage. So the combiner has to carry the firing reason forward — which argues against fully opaque fusion as the only layer.

“All this must be done at latencies of less than a second”

Where did this latency constraint come from?

Abnormal isn't inline. Their own architecture page describes a "pure API architecture that operates post-delivery," "outside the mail flow," with "no MX record changes," and notes that messages "briefly exist in the mailstore before remediation — typically measured in seconds."

[I] So the real constraint is time-to-remediate, not protocol timeout. The message is already sitting in the user's inbox. The window between delivery and retraction is the window in which someone can click, and mobile push notifications make that window very short — users read on phones within seconds of arrival. Every second is exposure.

The tension this creates, which the post doesn't acknowledge: several things on its own feature list cannot fit in a second —

• Sandbox detonation of an attachment: 30s to minutes (Q5)
• Headless rendering of a JS landing page behind a CAPTCHA: seconds
• Audit-log events (sign-ins, rule changes): minutes of lag (Q4)

So in practice this has to be a two-tier design: a fast path scoring on what's already available (text, headers, cached feature store, graph lookups) and a slow path that re-scores when enrichment lands — with the ability to retract a message after the fact. The post-delivery architecture is exactly what makes that second bite possible, and it's the same property that makes tiered thresholds cheap in Q7.

Read "less than a second" as an engineering target for the online scoring path, not an end-to-end SLA.

“How can we detect invoice fraud better? To do so we must deeply understand the natural language and images in an invoice to identify Abnormal patterns.”

What is invoice fraud exactly and why is it a challenge compared to the stuff they’re already doing?

What it is: getting a business to pay a real-looking invoice into an attacker-controlled bank account. Three variants:

• Vendor email compromise (VEC) — the attacker compromises the vendor's mailbox, sits inside a genuine payment thread, and sends a real invoice with changed banking details. This is what the post's "sit on a compromised account for months... inserting an illegitimate invoice into a conversation about payment at just the right time" bullet describes.
• Vendor impersonation — a lookalike domain of a real supplier; no compromise needed.
• Fake vendor — an invoice from a supplier that doesn't exist, aimed at a large AP department paying on autopilot.

Why it's harder than what they already do — seven reasons:

1. No malicious artifact. No malware, no credential-harvesting URL, no payload. Every signal a legacy gateway is built on is simply absent.
2. The text is legitimate business text. "Please find attached invoice #4417, net 30." Body-level NLP has nothing to grip — which is precisely why the post says you must understand the invoice, not the email.
3. The signal lives in the image/PDF. Detection requires OCR plus layout understanding: extract vendor name, remittance account/IBAN, amount, invoice number — a document-understanding problem, a different and much harder extraction task than reading an email body. That's what "images in an invoice" is pointing at.
4. It requires cross-message state. You cannot classify the message in isolation. You need a per-vendor payment-detail store: "this supplier has been paid to account ending 8812 for two years; this invoice says 4409." That's a data-engineering problem, not a model problem.
5. ~100× worse base rate. <1 in 10 million vs. 1 in 100,000. Far fewer positive labels, unchanged precision bar.
6. Every authenticity check passes. In VEC the mail genuinely comes from the vendor's real domain, valid SPF/DKIM/DMARC, inside a real thread with real history. The graph features that catch first-contact phishing all report "safe" — the exact opposite of the Q8 signals.
7. Highest per-incident cost, in both directions. Six- and seven-figure wire losses on a miss; blocking a real invoice stops the business paying its suppliers. Both FN and FP costs peak here simultaneously.

“How can we better identify account takeovers? This is a difficult anomaly detection problem”

Do they actually use traditional anomaly detection algorithms as either models or features?

I don’t think they use traditional anomaly detection algorithms.

• The dominant industry pattern is per-entity baselining and deviation scoring, not off-the-shelf unsupervised detectors. You model each user's own history and score how far today deviates — z-scores, quantiles, or a learned per-user profile. That is anomaly detection in the statistical sense, but it isn't "run an isolation forest on it."
• Why the classical algorithms underperform here: they find rare, and in enterprise email rare ≠ malicious. A salesperson traveling, a new laptop, a quarter-end send burst are all rare and all benign. At a one-in-a-million FP budget, an unsupervised detector's alert volume is unusable standalone.
• So the realistic role is as features into a supervised model, not as the decision-maker. Anomaly scores ("how unusual is this login for this user") become inputs alongside signals trained on labeled attacks.

How would you improve this system if you were building it today?

The attack side changed more than the defense side.

• LLM-written lures destroyed the fluency signal that 2019 NLP leaned on — no more grammar errors, and personalized spear-phishing now runs at spam economics.
• The attack went multi-channel: the callback step moved to cloned voice, SMS, Teams/Slack, and calendar invites. Email-only telemetry now sees a fragment.
• Targeting is automated: retrieval over public data to rank who has payment authority.

Seven changes, most to least important:

1. Model intent, not content. Since fluency no longer discriminates, stop asking "does this text look phishy" and model the request: what action, by whom, of whom, with what authority, at what point in a thread. "First-contact sender asks AP to change remittance details" is attack-shaped no matter how well written. This is an extraction-and-reasoning task an LLM does well and 2019 embeddings could not.
2. Replace the OCR→NLP pipeline with a vision-language model reading the rendered email, the attached invoice, and a screenshot of the landing page in one pass. That single component attacks all three of the post's stated open problems — invoice fraud, phishing sites, image-encoded text — at once.
3. Learned graph representations. 2019 used graph features; today use a GNN or temporal-graph model over the tenant's communication graph so relationship anomalies are learned rather than hand-engineered. Abnormal's 2026 material mentions graph neural networks, so this has happened.
4. Cross-channel identity fusion. One identity risk score across email, Slack/Teams, Zoom, calendar and the IdP — so a suspicious Teams message following a suspicious email is one incident, not two independent low-confidence events that each fall below threshold.
5. Adversarial evaluation as a first-class pipeline. Continuously generate LLM-based evasions of your own detector and train against them; treat detector robustness as a monitored metric, not a periodic red-team exercise.
6. Prompt-injection hardening. Any LLM component now reads attacker-controlled text. Email content must be data, never instructions — this is a new attack surface the 2019 design didn't have.
7. Fix some of it with workflow, not classifiers. For the highest-cost class, out-of-band verification of remittance-detail changes against a vendor record removes more residual risk than any marginal model improvement. Worth saying plainly rather than treating everything as a detection problem.

What I'd keep unchanged — these aged well: start simple and add complexity only when you can show it's better; heuristics as model features; redundant detectors; per-tenant baselining; calibration and thresholding discipline. Modest GBDT-on-tabular too — trees still beat NNs on genuinely tabular features.

It’s a little confusing to me that parents of items can be both brands (Coke) and sub-brands (Coke Zero). Why isn’t regular Coke a sub-brand of Coke the way Coke Zero and Cherry Coke are?

One way to think about this is that in their knowledge graph, a Sub-brand is created to capture a deviation from the core product (e.g., "Zero" for no sugar, "Cherry" for flavor). Because "Regular Coke" is the original standard, it has no deviating attributes to define it against the parent. It is the definition of the parent brand itself.

Depending on how the knowledge graph is used, having a Coke -> Regular Coke edge could increase latency without any actual benefit.

“Unstructured product description is passed to our in-house brand classifier”

What is their in-house brand classifier?

Exact model unclear from the blogpost but a reasonable assumption would be some sort of lighter-weight language model like BERT used as a closed-set classifier (trained on a limited set of labels).

“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction.”

Let’s say they used BERT as their initial classifier. Does lack of confidence here imply a relatively flat distribution (high entropy) over output probabilities?

That’s one way to measure it. We could also use margin (difference between top two predicted probabilities).

“The extraction output is passed to a second LLM, which retrieves similar brands and example item names from an internal knowledge graph to decide whether the extracted brand is a duplicate entity.”

Why do we need a second LLM for this? And how is it retrieving the similar brands and item names? Is it through embedding similarity search for the extraction output or by navigating the knowledge graph in some way?

By second LLM, I think they just mean the same model but called a second time with a different prompt.

1. The first call extracts the brand.
2. The second call, augmented through the similarity search, checks whether the extracted brand already exists.

I think the retrieval is just through embedding similarity based on the RAG system they describe later.

“the in-house classifier is retrained with the new annotations”

Does it make sense to retrain every time a new brand is added? And what would the retraining look like exactly?

I think retraining every time a new brand is added is probably not computationally efficient.

For the retraining step, if the classifier is a fixed-output model (e.g., a Softmax layer with N brands), the "head" of the model would need to be expanded to include a new node for the new brand.

For language models, you could freeze the earlier layers and simply retrain the final few layers of the model on the additional data + some older examples (to avoid catastrophic forgetting). LoRA could also work well.

If the cost of full retraining seems really low though, might as well do that.

“Last year, we stood up a model to label all organic grocery products”

Does it make sense to have a separate model to do this labeling as opposed to having it be just Organic/Not Organic be one part of the taxonomy mapping of a larger model?

One of the main downsides to having a single model do all sorts of taxonomy mapping is certain features may exist for some items but not for other items. Food items can be organic/not organic but batteries can’t. Having a model try to figure out what the feature value is for a feature that doesn’t exist for the item being considered can increase computational cost and negatively impact performance.

{Worth thinking through the other tradeoffs and how to make a final decision}

Where are the training data labels coming from in the organic product labeling use-case?

Since they mention “highest precision”, they must have some labels.

Unclear but either human labels or powerful LLM (that is really good at generating labels but too slow/expensive to run in production).

“missed cases where organic is misspelled / dropped or has a slightly different presentation in the data”

What do they mean by a ‘slightly different presentation in the data’? That it’s not included in the product title but might be present in an image or some other metadata?

Could also mean something like “Org," or "Org.” or the word is concatenated with other text e.g., OrganicMilk or Milk(Organic).

I feel like the LLM reasoning approach could have been used for the entirety of their catalog. Using a distilled language model for what I’m guessing is < 1 million unique items doesn’t seem like it should be that slow or costly, at least not anymore. Is that fair?

# of unique items isn’t the right number to think about; unique merchant SKU data is. Different merchants can encode their data differently and each of these different encodings would have to be processed.

Determining what would actually make the most sense in production would need more information about the cost and latency structure.

“LLMs conduct online searches of product information and pipe the search results to another LLM for reasoning”

What exactly do they mean by ‘LLMs conduct online searches’ and how is the first LLM here different from the second one?

It could look something like this:

1. The LLM is given a SKU (e.g., "Horizon Whole Milk 64oz").
2. The LLM generates a structured Search Query (e.g., site:horizonorganic.com "Whole Milk" ingredients organic certification).
3. A software wrapper executes this query via a Search API (like Google Search API or Bing Search API).
4. The raw HTML or snippets from the top results are scraped and fed back into the system.

The difference LLMs may not actually be different; they might just be referring to different calls with different prompts and context. It’s probably better to have a smaller LLM generate the search query and a larger one process the results to generate a response.

“Products from different categories are characterized by different sets of uniquely defining attributes. For example, an alcohol product is uniquely defined by attributes such as vintage, aging, and flavor.”

If we’re storing these in a database, does it make sense to have a column for each possible field or use a non-relational database?

Not a good idea to have a column for every field.

You can also use a hybrid structure where you have a column in a relational database storing category-specific attributes in a JSON.

We can also have a tall table - Entity-Attribute-Value (EAV) Model - where a single product could have multiple rows.

“For each unannotated SKU, we first leverage OpenAI embeddings and the approximate nearest neighbors technique to retrieve the most similar SKUs from our golden annotation set.”

What exactly are they feeding into the embedding model? Also, is this just equivalent to dynamic few-shot prompting?

An SKU at DoorDash is represented by a list of attributes, including:

• Item Name (e.g., "Dove Silk Glow Body Wash")
• Size/UOM (e.g., "500 ml")
• Brand (if available)
• Category

To generate the embedding, DoorDash likely feeds a concatenated string of these available attributes (e.g., "Dove Silk Glow Body Wash 500 ml") into the OpenAI embedding model. They are embedding the raw merchant-provided text to find semantically similar items in their human-verified "golden" dataset.

And yes, this process is functionally equivalent to Dynamic Few-Shot Prompting.

How do you pick the right number of examples to provide as context for the LLM in the previous step?

Test out different thresholds (for number and for similarity), and evaluate metrics of interest: accuracy, latency/cost etc.

“Ultimately, the generated annotations are used to fine-tune an LLM for more scalable inference”

What does this mean?

Could mean training a distilled model on outputs of a larger teacher model. Scalability is improved both in terms of the new model being faster/cheaper but also no API limits if we use an open-source model.

How would I convince a business leader that this project was worth doing?

Any cost-savings are definitely worth highlighting.

For other impacts, they’d need to figure out a way to connect improvements in the performance of these intermediate steps to actual downstream user impact (ads CTR, search fulfilment rate, etc).

How would you improve this system if you were building it today?

1. Use Contrastive Language-Image Pre-training (CLIP) to create unified embeddings that represent the text and the physical packaging in the same vector space.
2. Graph-Augmented Retrieval (GraphRAG)
   1. Instead of just finding "items that look like this one" via vector similarity, GraphRAG allows the LLM to traverse the Knowledge Graph's relationships (e.g., "This item belongs to the same distributor as Brand X")
3. Synthetic Data Generation
   1. Especially useful for edge-cases like misspellings.

“Attackers are often even using ML systems of their own to adapt their attacks!”

How would attackers use ML systems? Would they run their planned attacks through ML models to see if they would be picked up and adjust important features if so?

That’s one way to do it. They can either build their own models or use any of the following:

• Free/consumer filters — Gmail, Outlook.com — as a rough proxy.
• Secure email gateways bought as a legitimate tenant (a few hundred dollars a month), giving unlimited test sends.
• VirusTotal and public sandboxes, for the attachment/URL side.

There are other ways to use ML systems as well.

1. Reconnaissance and targeting — classifiers over scraped LinkedIn/corporate data to rank who has payment authority, cluster orgs by vendor relationships, and time the attack to real events (an acquisition, a quarter close).
2. Content generation — LLMs write fluent, on-register lures at scale, killing the grammar-error signal that older filters leaned on, and translate campaigns into new markets. Also style transfer: fed a few of a real executive's emails, mimic their cadence.
3. Identity spoofing beyond text — voice cloning for the callback step of a BEC, deepfake video for "verify me on Zoom." This is the fastest-growing branch since 2023.
4. Operational automation — ML to triage which compromised inboxes are worth working, and to run conversation threads with victims semi-autonomously.

“We cannot afford to throw out older attacks simply because they do not have the latest features. We must instead re-compute these features.”

Are there circumstances where it may be impossible to recompute those features?

Yes. Some examples include:

1. URL detonation doesn’t work because page is no longer live
2. Any internal features require underlying store to be append-only and timestamped
3. TTLs for data (because of GDPR erasure requests, data-residency rules, PII minimization commitments, contractual deletion)

“This includes changes to underlying detection code, new datasets, new features, and the development of new models.”

What are examples of new data sets that would affect the pipeline? Everything else makes sense to me, but I think I need an illustrative example of datasets to understand it better.

The model: a message arrives with its own contents. A dataset is a side table keyed on something in that message — sender domain, sender address, recipient, URL, attachment hash — looked up and joined in before features are computed.

Four sources of such tables:

1. The customer's own mail history. A per-tenant communication graph: for each sender-recipient pair, first-seen date and message count. Enables this vendor has emailed this AP clerk 400 times vs. never before. This class makes time travel hard — the correct join for a 2019 message is the graph as of 2019.
2. The customer's other systems. A new Okta or HR-directory integration gives you the real reporting structure, so "CFO wires request to a direct report" separates from "CFO asks someone three orgs away."
3. Third parties. A domain-intelligence feed keyed on sender domain, supplying registration date and hosting ASN — newly-registered-domain becomes computable the day you add it.
4. Internal analyst or model output. A curated list of known-good vendor domains, or campaign cluster IDs from another job.

“we make an unintentional code change that modifies a feature used by a model (but we do not retrain the model)”

What is a likely example of such a code change?

Four classes:

1. A refactor that alters a boundary case. Someone rewrites the sender-domain parser to handle subdomains properly. Correct by any reasonable spec — but a model that learned "this domain string is safe" now sees a different string on the same mail, and its lookup misses.
2. A dependency upgrade. Bump the tokenizer, language-detection library, or HTML parser. Text features shift slightly across every message; no line of your own code changed, so nothing flags it.
3. A change to a null or default. Missing sender-age gets -1 instead of 0, or a timeout that used to return "unknown" now returns "clean." The model learned the old sentinel's meaning and reads the new one as a real value.
4. A units or normalization change. Timestamps switch from local to UTC, a count becomes a rate, a score gets rescaled to 0–1. The feature is now more sensible and entirely wrong relative to the trained weights.

Why rescoring catches these and unit tests don't: the bug is a mismatch between two artifacts — code and model weights — that no single component's tests can see. Running the whole stack over Golden Labels surfaces it as a drop in rescored recall, which is precisely the argument the post is making for testing the stack rather than the parts.

“Process to evaluate the entire detection system to produce performance analytics like precision, recall. We call this “rescored recall.””

This isn’t equivalent to actual recall, right? Because they don't have all true positives.

Yes, it’s not equivalent.

“We must have correct and unbiased features”

The word "unbiased" implies the existence of biased features. Is this just a reference to time travel?

Yes, I think so.

“Should be able to update all historical samples”

Let's say the data you have doesn't let you update all historical samples, but only a subset. Do you just not use it then? That seems a little inefficient.

Not sure how Abnormal handles this internally but some alternatives to throwing it out include:

1. Mark the feature missing and let the model handle it. Tree models natively route missing values; for others you carry an explicit is_null indicator. Cost: the model learns a pattern from missingness itself, and if missingness correlates with age, it correlates with label. Mitigation is to inject the same missingness into recent samples at the observed rate so the indicator carries no time signal.
2. Restrict the metric's window, keep the sample. Report rescored recall on the full corpus for stack-wide features, and on the post-cutoff window for the new feature's contribution. The old attacks still train models and still count in every other evaluation. This is usually the right answer for a feature added six months ago.
3. Two-tier the corpus. A fully-hydrated recent tier for anything requiring complete features, plus a degraded historical tier used where partial features suffice. Explicit, but adds bookkeeping to every consumer.
4. Impute. Almost always the worst choice here. An imputed value on a rare attack is a fabricated signal on your most precious samples, and it's invisible downstream — unlike a null, which announces itself.

“Time travel ensures unbiased data and also helps us avoid future leakage: avoiding bringing any data from the future (to a past sample) that leaks labels into training features.”

What are a couple of examples of features that could be affected in this way?

1. Sender-recipient relationship count. A behavioral feature like "number of prior messages from this sender to this org." Recomputed today over the full history, an attacker domain that ran a 2019 campaign shows hundreds of messages — inflating it into a familiar sender. Worse, the count includes the campaign's own later messages, which exist because the attack happened. Time-traveled to the attack timestamp, the correct value is zero or near-zero, which is the actual signal.
2. Domain reputation from a threat feed. Query the feed today for a domain used in a 2019 attack and it returns "known malicious" — because the industry blacklisted it in response to that campaign. That feature is a near-perfect label proxy in training and worthless at inference, where the domain is hours old and unlisted. A model that leans on it will look excellent on Golden Labels and fail live.

“To do so, we rely on the most recent updated Golden Labels from the night before”

Is there a reason they're doing daily batching? Couldn't this miss recent attack patterns?

Why daily rather than continuous:

1. Cost. Full regeneration re-extracts features for every historical sample against time-correct join tables — a Spark job over years of data. Running it hourly multiplies compute by 24 for a corpus that changes by a fraction of a percent.
2. Labels aren't available in real time anyway. A message detected today may not be labeled for hours or days — customer reports, analyst review, retroactive discovery. A faster pipeline would mostly re-process unlabeled data.
3. Reproducibility. A dated, branch-tagged snapshot means two engineers comparing baseline vs. experiment are measuring against identical data. A continuously-updating corpus makes results non-comparable across runs.
4. Off-peak windows. Overnight batch avoids contending with production scoring for cluster capacity.

On your concern — the gap is real but it's not where the risk lives. Two things to separate:

• Detection latency — how fast a new attack pattern gets blocked. Not affected by this pipeline. That runs through the live scoring path and, presumably, faster mechanisms: signature/campaign clustering, analyst-pushed rules, threat-intel updates. Rescoring is an offline evaluation and training-data system.
• Evaluation and retraining latency — how fast a new pattern shows up in your metrics and your next model. Bounded at ~24 hours by this design, plus label delay, which likely dominates.

“We can either set up two configurations, “Baseline” and “Experiment,” or run this on two different code branches.”

What are the trade-offs between these two options?

What each can express

• Two configs, one branch. Only changes the config schema anticipated: swap a model, skip a stage, point at a different artifact. Anything requiring new code is out of reach.
• Two branches. Anything at all — new feature logic, changed extraction code, a new join, a restructured decision tree. This is the only option when the change is code, which is exactly the failure mode the post worries about in the "unintentional code change" paragraph.

Trade-offs

1. Confounding. Configs vary one declared knob against identical code — a clean A/B. Branches carry every difference between them, including drift from main if the branch is stale. Diagnosing "which of my six commits moved recall" is on you.
2. Compute. Configs can share upstream stages: run feature extraction once, fork only at MODEL_SCORING. Two branches generally can't share intermediates, since you can't prove the shared prefix is identical — so you pay roughly double.
3. Reproducibility. A config is a serializable object you can attach to a result and re-run later. A branch is a moving pointer; rebasing or force-pushing silently invalidates the comparison unless you record the commit SHA.
4. Path to production. A config swap is an experiment that then needs code written to ship it. A branch experiment is already the artifact you'd merge, so the thing you measured is the thing you deploy.

Rule of thumb: configs when the change fits the schema — cheaper, cleaner, more reproducible. Branches when it doesn't, accepting more compute and more attribution work. The post's stated aspiration, CI/CD auto-running rescoring on affected stages per code change, is the branch path automated.

“In many cases, GenAI-powered solutions are easier to maintain than traditional ML models”

Why is that?

1. The artifact you maintain is a prompt, not a training pipeline. A traditional supervised model for invoice parsing needs labeled data, a feature pipeline, a training job, a model registry, and a serving stack. Each is a thing that breaks. With GenAI the vendor owns the model; you own a text file and some post-processing.
2. Change cost is asymmetric. A new vendor with an odd invoice layout means, for a traditional model, collecting and labeling examples of that layout, retraining, revalidating, redeploying — days to weeks. For GenAI it means editing the prompt and rerunning evals — hours. This is the maintainability consequence of the flexibility bullet.
3. No feature engineering to keep alive. Document parsing models traditionally depend on OCR output, bounding-box geometry, layout heuristics, regexes per field. Those are brittle and accumulate special cases. An LLM absorbs that into the model's pretrained understanding.
4. No label drift treadmill. Traditional models degrade as the input distribution shifts and need periodic relabeling and retraining. GenAI doesn't require you to own that loop.
5. Fewer specialists needed. A prompt plus evals is legible to an engineer or analyst; a fine-tuned layout model needs someone who understands its training setup.

Why didn’t they ask the models making the predictions to generate confidence scores? Even if those scores are likely to be over-confident, that feels like something that can be addressed through calibration.

There are a few reasons why this might not work great (lack of independence, instability across runs, interpretability) but I think it’s still worth at least discussing.

““Calibrator” refers to using a separate single GenAI model to evaluate the outputs generated by other models and assign confidence scores”

1. Should this be a different model than those used for extraction? If so, which ones make sense for which use-case?
2. What would actually be included in the prompt for this model?

1. Should it be a different model?

Yes, for the same reason self-scoring fails: correlated errors. If the extractor and the judge are the same model, a misread "0" as "O" looks correct to both. A different model gives you an independent error distribution — the same property that makes their voting ensemble work.

Which model:

• Judging structured extraction against a document (their case) — needs strong document/vision grounding, since the judge must re-read the invoice image, not just inspect the text. That points to a frontier multimodal model, and specifically not the weakest member of the extraction ensemble.
• Judging free-text output with no ground-truth artifact — a cheaper text-only model is usually fine, since the task is coherence and consistency rather than verification.
• High-volume pre-filtering — a small fast model as a first pass, escalating only the uncertain cases.

2. What goes in the prompt?

Six things:

• The source document (image or OCR text) — without it the judge is grading plausibility, not correctness.
• The field definition, ideally the same one the extractor got, so both are answering the same question about "invoice total."
• The candidate value to be judged.
• A rubric that decomposes the judgment: is the value present in the document, is it in the right place, is the format right, is there a competing candidate elsewhere on the page.
• A required structured output — score plus a short justification, ideally with the justification first so the score follows from stated evidence rather than being emitted cold.
• A discretized scale with anchored labels (e.g. five bands, each defined) rather than a free 0–100. LLMs cluster at round numbers, and defined bands are more stable across runs — which was Spotify's stated complaint about calibrators.

“The calibrator model is also trainable by learning from feedback and can potentially improve over time.”

Does this mean fine-tuning, few-shot learning, prompt iteration or something else?

Not stated in the blogpost but my best guess would be few-shot examples plus prompt iteration, not fine-tuning.

The reasoning: their whole framing is that GenAI wins on fast development and low maintenance. Fine-tuning a judge would undercut both — it needs a labeled dataset of (document, extraction, correct-or-not) pairs, which is the labeling burden they were trying to escape by not training an extractor in the first place. And the invoice distribution keeps shifting as vendors change, so you'd be retraining continually.

“confidence scores generated by calibrators are difficult to interpret and sometimes counterintuitive”

In what ways is this true?

Difficult to interpret — the number has no defined referent. A voting score of 80% means four of five models agreed; you can state what generated it and reproduce it. A calibrator's 0.8 means whatever the model's next-token distribution landed on. It isn't a frequency, isn't tied to an observable event count, and can't be explained to an auditor as a control.

Counterintuitive — the specific ways this shows up:

• Non-monotonicity. Two extractions where one is clearly harder get similar or inverted scores. The judge is responding to surface features (clean formatting, plausible-looking numbers) rather than to actual difficulty.
• Clustering. Almost everything lands at 0.85–0.95, so the score doesn't separate the cases you care about. Then an occasional 0.3 appears for no visible reason.
• Score-justification mismatch. The judge writes a reason that identifies a real problem, then attaches a high score anyway — the text and the number are generated somewhat independently.

“the scores are not consistent across multiple runs”

What’s the underlying reason for this and are there any good ways to address it?

Underlying reason. Sampling is the obvious part — temperature above zero means the score token is drawn from a distribution, and if that distribution has mass spread across "8", "85", "9", you get different numbers on identical inputs. But the deeper issue is that the underlying distribution is flat. A well-grounded judgment concentrates probability on one answer and survives sampling; a weakly-grounded one doesn't. The instability is a symptom that the judge has no firm basis for the number, not just that decoding is stochastic.

Two secondary causes: LLMs are sensitive to irrelevant perturbations (ordering, whitespace, which examples are in the few-shot block), and even at temperature zero, batching and hardware nondeterminism in floating-point reduction make outputs non-reproducible run to run.

What actually helps:

• Discretize the output. Five anchored bands instead of 0–100. Sampling noise that would move 87 to 91 usually doesn't cross a band boundary. Biggest gain for the least effort.
• Sample it repeatedly and aggregate. Run the judge n times and take the median or modal band. This converts an unstable point estimate into a stable one — and note it's the same move as majority voting, which is where they ended up.
• Force the reasoning before the score, so the number is conditioned on stated evidence rather than emitted cold. Reduces variance but doesn't eliminate it.

“However, the methodology is not always transparent – it varies by provider, and for many models it is unknown how they were calculated.”

Don’t they all work the same way? If not, what are the differences?

The math is identical everywhere — log of the softmax over the vocabulary at one decoding step. What varies is everything between that number and what reaches you.

The three that actually matter for Spotify's method:

1. Availability. Not a uniform feature. Anthropic doesn't support logprobs at all; Gemini's support lives on Vertex AI rather than the standard API. A 2025 study of OpenRouter found only 164 of 813 endpoints (23%) returned them. A multi-vendor ensemble can't get logprobs from every member — which alone may explain why they dropped the approach.
2. Transformations applied before the number you see. Temperature scaling, top-k/top-p truncation with renormalization, repetition penalties, logit bias, and constrained JSON decoding all alter the logits first. Constrained decoding is the worst offender here: zeroing invalid tokens inflates confidence in whatever survives. Providers rarely document which of these apply.
3. Tokenization. Averaging over tokens makes the denominator tokenizer-dependent. "1,234.56" might be 3 tokens in one model and 8 in another, so the same extracted value gets a different average across ensemble members and isn't comparable.

Points 2 and 3 are what break their specific method — exponentiating an average of logprobs assumes a clean posterior, and under those transformations it isn't one.

“The confidence score was then calculated by exponentiating the average logprobs of the tokens.”

Why exponentiated?

To get back to a probability scale.

What does a single dot here represent?

Almost certainly a bucket of extractions grouped by confidence score, not an individual extraction. Two reasons:

• The y-values are accuracies — 60%, 75%, 96%, 22%, 12.5%. A single extraction is right or wrong, so accuracy is only definable over a group.
• The specific fractions give away small bucket sizes: 60% = 3/5, 75% = 3/4, 12.5% = 1/8, 66% = 2/3, 25% = 1/4. Roughly 8–12 dots per panel, each holding a handful of invoices.

“Majority voting is an ensemble method that selects the final output by choosing the most common response from multiple prompts or among multiple GenAI models.”

Are they using majority voting for the actual prediction too or just for the confidence scores? If it’s only for the confidence scores, couldn’t you potentially have the single extractor model have a different output than the majority vote?

I think they’re using it for both, which means the concern I raised is not applicable.

What models are generally considered best for this kind of use-case?

Frontier multimodal: Gemini 3 Pro led March 2026 invoice benchmarks (94.75%), with Azure Document Intelligence (90.52%) and Claude Sonnet 4.5 (90.27%) behind it. Claude Sonnet 3.5 degraded least as document quality dropped from clean PDFs to photos — resilience matters more than peak accuracy for real vendor mail.

Open-weight: Qwen3-VL-8B-Instruct is now state-of-the-art among open vision-language models.

Specialized engines (Azure, ABBYY, Rossum) remain competitive, particularly on tables and line items.

Two things that matter more than the rankings:

1. Pipeline structure beats model choice. A 2026 study on scanned financial documents found feeding full PDFs directly to a VLM was the worst configuration in every test; OCR-first multistage pipelines won by a wide margin. The document-specialized MiniCPM-o-2.6 also beat the much larger general-purpose Gemma-3-27b under identical settings.
2. For the ensemble, diversity beats individual accuracy. The voting signal comes from independent error distributions, so mix vendors and architectures. Logprob support is irrelevant here, so Anthropic's lack of it doesn't rule Claude out the way it would for the logprob approach.

“Literature often suggests using four to seven models in an optimal ensemble.”

What literature?

I haven’t found anything that supports this specific number so far so more extensive research may be needed.

In their use-case, they found that majority voting works best. Is that consistent with what the literature on this has found more generally?

Where the literature agrees. Consistency-based confidence is the mainstream approach, resting on Spotify's same premise: genuine knowledge produces reproducible answers, fabrications diverge. A recent head-to-head audit found self-consistency had the highest rank correlation with correctness on GPQA (ρ=0.31), ahead of verbalized confidence (0.21) and P(True) (0.19) — matching their ordering.

Their logprob failure also replicates, with a better diagnosis than the post offers. A 2026 paper on document field extraction notes these methods measure confidence in the generated token sequence, but extraction errors come from what the model can't observe — unreadable source material, ambiguous layouts, OCR noise. A model confidently transcribing OCR noise emits high logprobs for a wrong answer.

Where it's less flattering. That paper puts self-consistency across five calls at only 0.744 AUC on field extraction — better than logprobs, but not good, and at five times the cost. The audit concurs that no signal was a reliable standalone abstention score.

One way Spotify did better. Most of this literature tests self-consistency, resampling one model. They used cross-model voting, drawing on genuinely independent error distributions rather than one model's sampling noise — strictly stronger, and worth weighing against that 0.744.

The frontier is multi-signal. The 2026 paper argues for combining agreement with document-side evidence: image quality, spatial location, OCR text. That directly addresses the long-text-field problem they left unsolved.

“To calculate the final score, a weighted majority voting approach was implemented. Weight for each model was based on model accuracy and then normalized to sum to one.”

Do we need any kind of train-test split here?

Yes, and in two places.

1. The weights. "Model accuracy" has to be measured on labeled invoices. If you measure it on the same invoices you then report ensemble performance on, the weights are tuned to that set's noise — models that happened to get lucky on the hard cases get upweighted, and reported accuracy is optimistically biased. With only 5–6 weights the overfitting is mild, but it isn't zero, especially if the eval set is small (their Figure 1 buckets suggest it is).

2. Platt scaling — this is the bigger one. Platt scaling fits two parameters mapping raw confidence to calibrated probability, and it must be fit per field. Evaluate calibration on the fitting data and your ECE looks great by construction. Their Figure 3 shows before/after calibration curves; if those are in-sample, the improvement is partly guaranteed rather than measured.

What the setup should be: three splits, or nested cross-validation. One fold to estimate model accuracies for weights, a second to fit Platt parameters, a third untouched fold to report accuracy and calibration.

Why it matters more here than usual. The confidence score is a control — a threshold decides whether an invoice goes to a human. If calibration is optimistic in-sample, the real pass rate at a 95% threshold is worse than the number they'd show an auditor. Under SOX, an overstated control is the failure mode that actually costs you.

“we applied Platt scaling”

Why Platt scaling as opposed to isotonic regression?

Three reasons, in order of weight:

1. Sample size. Platt fits two parameters; isotonic fits a free monotone step function and needs roughly a thousand-plus labeled examples per field to avoid overfitting. They calibrate per field, and their plots suggest far less data than that.
2. Input granularity. With 5–6 models, majority voting yields only about six distinct confidence values. Isotonic would memorize an accuracy per value — effectively a lookup table with no ability to interpolate. A sigmoid smooths across levels.
3. Auditability. They chose linear weights over exponential explicitly for explainability. Two parameters is easier to document to an auditor than a nonparametric fit, and a sigmoid's smooth output avoids the step discontinuities that would make a threshold-based control jumpy.

Where isotonic would win: miscalibration that isn't sigmoid-shaped, such as good calibration mid-range but distortion only at the top end. Platt can't represent that — but at their sample sizes they likely couldn't detect it either.

“This approach requires careful tuning of distance thresholds.”

What would this tuning look like concretely?

Concretely, three separable pieces:

1. What you're tuning. A cosine-distance cutoff τ: two model responses join the same cluster if distance < τ. Too low and genuine paraphrases ("123 Main St." vs "123 Main Street") split into separate clusters, so agreement is understated and confidence collapses. Too high and real disagreements merge — the "0" vs "O" failure they hit — so confidence is overstated on a wrong answer. Related knobs, if you're being thorough: the embedding model, and the linkage rule (single-link chains transitively and will bridge distinct answers).

2. The objective. Not clustering quality. Build a labeled set of response groups where you know which responses are equivalent, then sweep τ and score against the two error types above. Because false merges are the dangerous direction in finance, weight precision over recall — e.g. maximize recall subject to false-merge rate under some ceiling — rather than optimizing F1.

3. Per-field fitting. One τ won't serve addresses, item descriptions, and vendor names — their embedding-distance distributions differ. That means a labeled set and a sweep per field, plus a held-out fold, since τ is a fitted parameter like the Platt coefficients.

Why it still failed for them: embedding distance is dominated by semantics, and "0" vs "O" is semantically invisible. No τ separates that case, which is why this is a threshold problem you can't fully solve by tuning the threshold.

“This approach requires extensive prompt engineering to ensure the selector prioritizes factual accuracy and consistency.”

How does factual accuracy fit into this? From my understanding, the selector model is just looking at the responses of the voter models, right? What facts are being verified?

Unclear from the post.

“we broke down long text fields into smaller, more manageable components (e.g., splitting addresses into street, city, state, and zip code)”

How did they do this break-down? Using a model or heuristic/regex?

Two possible readings, and the wording favors the first:

1. Split at the prompt, not after. Each model is asked for street, city, state, and zip as separate fields, so five separate votes happen on five short strings. Nothing needs parsing — the decomposition lives in the extraction schema. This fits "broke down long text fields into smaller components," fits their stated goal of improving agreement during voting, and fits their whole approach of solving problems by prompt adjustment rather than added machinery.
2. Split after extraction, parsing each model's full address string into components before voting. This is strictly worse for their purpose: if the parser is a regex it breaks on international addresses, which is their stated problem domain; if it's a model, you've added a component whose own errors now contaminate the vote.

Should we be concerned about the intra-model correlation in the prompt permutation phase? A 94% score suggests a high degree of confidence but if the same models are being used, I feel like it’s somewhat exaggerated.

I think this is a reasonable concern but maybe mitigated somewhat by the calibration stage.

How would I convince a business leader that this project was worth doing?

Hard to say without more insight into the use-cases of human-in-the-loop decisions and meeting regulatory requirements.

How would you improve this system if you were building it today?

1. Add document-side signals to the score

The biggest weakness is that every signal they use is model-side. Agreement can't detect a failure all models share, and their unsolved "0" vs "O" case is exactly that. Current work on extraction confidence combines vote agreement with evidence from the document itself: image quality metrics, whether the extracted value is spatially locatable on the page, agreement with an independent OCR pass, and agreement between two structurally different extraction calls. A wrong character is invisible to voting but visible to a string comparison against OCR text.

2. Use business validation as a free, hard signal

Invoices are internally redundant in ways they never exploit: line items should sum to subtotal, subtotal plus tax should equal total, currency should match the header, the invoice number should match the PO's, the vendor should exist in the master file. These are deterministic checks with no false-confidence failure mode. A total that fails arithmetic should be low-confidence regardless of unanimous agreement, and a total that passes deserves a lift. This is cheaper than any additional model run.

3. Restructure the ensemble spend

Instead of running 5–6 frontier models on every invoice: run 2–3 on the first pass, and escalate to the full ensemble plus permutation only when the cheap signals disagree or validation fails. Most invoices are native digital PDFs where accuracy is 98–99%; the cost belongs on the photographed and multilingual tail.

Two related fixes: two-stage voting for the permutation phase (per-model modal answer, then vote across models) so correlated prompts don't inflate the denominator; and mixing architectures deliberately — a specialized engine like Azure alongside frontier VLMs — since diversity is what the vote is actually measuring.

4. Fix the pipeline, not just the aggregation

A 2026 study on scanned financial documents found feeding full PDFs directly to a VLM was the worst configuration in every test; OCR-first multistage pipelines won by a wide margin. If they're still passing whole documents, upstream structure will buy more than another voter.

5. Tighten the evaluation

The post shows no train/test discipline for the weights or the Platt parameters, and the calibration curves may be in-sample. Nested splits, reported AUC and AURC rather than just correlation, and a stated false-merge rate — because under SOX an overstated control is the failure that costs you.

Does the model run on all items the user adds to their cart or just the ones that turn out to be unavailable?

For all ordered items, not just the ones that turn out to be unavailable. Serving is almost certainly a precomputed item→candidate-substitutes table refreshed in batch, not a live forward pass over the whole catalog per cart.

“then proceeded to binary classification, and eventually pursued a deep learning recommendation model”

The way this is phrased makes it seem like the deep learning model isn't doing binary classification but something else instead?

The phrasing is imprecise - the DLRM is still doing binary classification.

Why did they use TF-IDF cosine similarity as opposed to semantic embeddings (like BERT) at the start?

No reasons presented in the blogpost but possible reasons include:

1. Off-the-shelf BERT is a poor fit for product titles specifically. Grocery SKU names are short, keyword-dense strings with brand tokens, sizes, and units — "Coca-Cola Classic 12 fl oz Can 12pk." Generic BERT is pretrained on prose and, without fine-tuning, its sentence embeddings underperform even simple lexical baselines on this kind of text; it also tends to wash out the exact-token matches (brand, pack size) that carry most of the signal here. TF-IDF preserves those.
2. Operational cost. TF-IDF is a sparse matrix and a dot product — cheap to compute over a large catalog, trivial to refresh when SKUs change, and fully inspectable when a recommendation looks wrong. A transformer at phase 1 means GPU inference, an embedding store, and a debugging surface you can't eyeball. For an MVP whose main job was to unblock feedback collection, that overhead buys little.

Why did they only use the item’s name as opposed to descriptions to find similar items?

No reasons presented in the blogpost but possible reasons include:

1. Descriptions largely didn't exist. DoorDash's grocery catalog was assembled from merchant feeds and in-store data collection, not authored by DoorDash. Names are the one field every SKU has; descriptions are missing, truncated, or inconsistent across merchants for a large share of items. A feature you can only compute for part of the catalog directly caps the coverage metric they later tracked.
2. For groceries, the name carries nearly all the discriminative signal. "Bush's Best Cut Green Beans 14.5 oz Can" already encodes brand, product type, form, and size — the exact axes that determine a good substitute. This is unusual: for restaurants or apparel, descriptions matter much more. Grocery SKU names are effectively structured records rendered as strings.
3. Descriptions would have actively hurt a TF-IDF model. They're mostly marketing boilerplate and shared brand copy. Under bag-of-words cosine similarity, two unrelated products from the same manufacturer share long stretches of identical text and score as similar. IDF down-weights terms common across the whole catalog, but not terms common within a brand or category — so the noise survives. Longer documents also dilute the weight of the few tokens that matter (the pack size, the brand).

“our catalog team built out a well-defined taxonomy that let us apply heuristics on top of the text-based similarity score to restrict recommendations to relevant categories”

I didn't fully understand this. What is an illustrative example?

What it means: two filters in sequence. TF-IDF says "these names share words." The taxonomy says "these are the same kind of thing." Rank by similarity within a category node; drop anything outside it regardless of score.

Example. Customer orders Coca-Cola 12-pack cans. TF-IDF alone surfaces Pepsi 12-pack and Diet Coke 12-pack (good), but also Coca-Cola Gummy Candy, Coca-Cola BBQ Sauce, and a Coca-Cola T-Shirt — all scoring high on the brand token. The taxonomy cuts the last three: wrong category node.

“chose to use LightGBM for this phase because of both its relatively high performance with minimal hyperparameter tuning”

High performance as measured on their training data or based on prior literature?

Unclear from the blogpost. It’s worth noting that "performance" here could mean predictive accuracy or computational efficiency. LightGBM's actual differentiator versus XGBoost is speed — histogram-based splitting and leaf-wise tree growth — with accuracy roughly comparable

“this model combines principles from approaches based on collaborative filtering and predictive analytics”

In what ways does this model combine those principles?

Collaborative filtering side = the embeddings plus the explicit interaction step. Learning a vector per item and scoring pairs by dot product is matrix factorization, the canonical CF method. DLRM computes pairwise dot products among all embeddings, so that layer is essentially a bank of MF models. The signal is behavioral: which items co-occur or get approved together, with no notion of what the item is.

Predictive analytics side = the two MLPs over dense features. Continuous features (price, size, popularity, similarity scores) go through the bottom MLP and, after interaction, the top MLP — a standard supervised classifier on engineered features, which is exactly what phase 2's LightGBM was. The signal is attribute-based.

Why combining matters here: pure CF fails on cold-start items with little feedback; pure feature-based models miss preference patterns not encoded in attributes (the Pepsi 12-pack over 2-liter Coke case). DLRM lets both flow into the same sigmoid.

One caveat: DoorDash's item embeddings were pretrained on search behavior, not learned fresh from substitution labels. So the CF component partly comes in as a frozen input rather than being fit end-to-end — the post doesn't say whether those embeddings were fine-tuned.

“categorical features (or in this context, items in our catalog) are processed as embeddings”

How are items represented in this model? A single embedding per item or broken down into different features (some of which would be categorical features represented as embeddings)?

Most likely a single vector per item, not a decomposition into brand/size/category sub-embeddings. The post's own phrasing points this way, and the semantic embeddings it inherited are whole-item vectors by construction. Attribute-level features are named as future work ("organic," "kosher," image embeddings), which implies they weren't decomposed at the time. Standard DLRM would allow additional categorical fields — taxonomy node, brand, merchant/store, each with its own table. Plausible here given the catalog taxonomy already existed from phase 1 but bot mentioned either way.

Note that we have at least two item embeddings per scoring pass. The task is pairwise — score candidate j as a substitute for ordered item i. So both IDs get looked up, and the explicit interaction layer's dot product between them is where most of the signal lives. The post never says this, but the task formulation requires it.

“Next, feature interactions are computed explicitly”

What does this mean?

In DLRM, "explicit interaction" is a specific operation: take every embedding vector plus the bottom MLP's output, and compute the dot product between every pair. Those scalars get concatenated with the dense vector and passed to the top MLP.

"Explicit" contrasts with "implicit." If you just concatenate all inputs and feed an MLP, the network can in principle learn multiplicative relationships between features, but it has to discover them through stacked additive layers — slow and sample-hungry. DLRM instead hard-codes second-order terms, borrowing from factorization machines. Here that's very natural: the dot product between the ordered item's embedding and the candidate's embedding is a similarity score, which is the core signal for substitution.

What is the difference between the bottom MLP and the top MLP?

Why the bottom MLP exists at all (inferred): dimensional compatibility. The interaction layer takes dot products between vectors, so every participant must have the same width. Item embeddings might be 64-d; the raw dense features are some arbitrary count of scalars (price, size, popularity, TF-IDF similarity). The bottom MLP projects them to 64-d so the dense side can participate in the same pairwise dot products as the items. Without it, dense features couldn't interact with embeddings at all — you'd be stuck concatenating them at the end.

Why the top MLP exists: the interaction layer outputs a bag of scalars with no ordering or weighting. Something has to learn which of those pairwise agreements matter and how they combine nonlinearly. That's the top MLP — it's the actual classifier, and it's where higher-order effects beyond second order get modeled.

Rough intuition: bottom MLP is a translator getting dense features into embedding space. Top MLP is the judge deciding the verdict from all the evidence.

“the embeddings are trained on the search behaviors of DoorDash users”

In 2-3 sentences, explain how the search behavior is used to train the embeddings.

Search logs supply pairs of things users treated as equivalent — a query and the item they clicked, or two items clicked under the same query. A twin (Siamese) network encodes each side with shared weights and is trained with a contrastive objective: pull matched pairs together in vector space, push randomly sampled non-matches apart. The result is that items people search for interchangeably end up nearby, which is why green beans land closer to peas than to corn.

“For example, as shown in Figure 4, the LightGBM model recommended canned corn as a substitute for canned green beans. The deep learning model, however, recommended canned green peas, because item embeddings accurately represent that beans are more similar to peas than corn.”

Is there something intrinsic to these models that leads to LightGBM performing worse for this example or is it more random?

It's intrinsic — but the cause is representation, not "trees are worse than neural nets." Two distinct mechanisms:

1. Nothing in LightGBM's feature set separates corn from peas.
Trees split on features you hand them. Under plausible phase-2 features — name token overlap, taxonomy node, price, size, popularity, per-pair feedback rate — canned corn and canned green peas are near-identical for a green-beans query. Same node (canned vegetables), same price band, same "canned" token. With no discriminating feature, the model falls back on the strongest remaining signal: popularity. Canned corn outsells canned peas, so it wins. That's a systematic bias, not noise.

2. Even given embeddings, a tree can't cheaply express pairwise similarity.
Feed LightGBM the 64-d item vectors as columns and it still can't compute a dot product — axis-aligned splits approximate a similarity surface only with an enormous number of splits. DLRM's explicit interaction layer does it in one operation. This is a genuine architectural gap for pairwise tasks.

“We then compared what percentage of the algorithm’s recommendations were matched by human-curated substitutions.”

Is this equivalent to precision, recall, or something else?

Precision@k

“coverage (percent of ordered items with recommendations)”

Shouldn’t this be 100% by construction?

No — and the reason is that the model was deliberately allowed to abstain.

A pairwise scorer over the whole catalog will always return something if you take the argmax. But shipping the argmax regardless of score is bad: a wrong substitution is worse for the customer than no suggestion, since the fallback (chat with the Dasher, or a refund) already exists and is acceptable. So there's a confidence threshold, and items whose best candidate falls below it get nothing.

“We also plan to develop personalized recommendations because we’ve observed that customers have highly individualized substitution preferences”

What would be the best way to incorporate personalization into this system?

Three places personalization can enter, by pipeline stage:

1. Input layer — a user ID embedding as an additional categorical field.
DLRM-native: add a user embedding table, and its dot products with the ordered-item and candidate-item embeddings fall out of the existing interaction layer for free. Architecturally the cleanest. Directly hits the sparsity wall above, so it only works for the head of the user distribution.

2. Feature layer — user-level dense features into the bottom MLP.
Engineered signals that generalize across users rather than identifying them: this customer's historical brand-switch rate, private-label acceptance rate, size-flexibility rate, organic-purchase share, price sensitivity, past thumbs-down rate by category. Each is estimable from a handful of events and degrades gracefully to a population prior when data is thin.

3. Post-scoring layer — constraints and re-ranking on a global model.
Hard filters from stated preferences (dietary restrictions, allergen avoidance, no-substitute flags) plus boosts from the customer's own purchase history — a candidate they've bought before should outrank an equally-scored one they haven't.

How would I convince a business leader that this project was worth doing?

1. Come up with some estimate of the cost to DoorDash of
   1. Users not receiving the item they wanted because out of stock
   2. Friction from manual substitution
2. Calculate how much of that cost has been mitigated through the algorithm

If you’re A/B testing, you can also look at various user engagement and revenue metrics

How would you improve this system if you were building it today?

Ranked by expected value.

1. Condition on real-time store-level availability.
The system as described scores substitutes for an item in the abstract. But a recommendation is worthless if the suggested item is also out of stock in that store, and the Dasher is standing in the aisle with a list. I'd make the candidate set store-and-time-specific: score P(good substitute) × P(in stock at this store now), where the second term comes from an availability model fed by Dasher found/not-found reports, POS feeds where available, and recency of last confirmed pick. This is the largest gap in the original design and it's orthogonal to model quality — a mediocre recommender that only suggests things actually on the shelf beats an excellent one that doesn't.

2. Generate the missing metadata with a VLM instead of waiting for merchants.
The post's own next steps name richer attributes (organic, kosher) and image embeddings as future work, and my earlier answer on why they used names only came down to descriptions being absent. That constraint is largely gone. Run a vision-language model over product images and packaging text to extract a structured attribute record per SKU: brand, form (canned/frozen/fresh/dried), net quantity and unit, flavor, dietary flags, allergens, private-label indicator. Those become explicit features rather than something the model has to infer from a name string, and they make the failure modes debuggable — you can point at why a swap was blocked.

3. Fix the label.
Thumbs-up/down on a shown recommendation is a preference proxy collected before the fact, and it's only observed for pairs the system chose to display — classic selection bias, where the model is trained on its own past decisions. I'd train on realized outcomes instead: was the substitute accepted, kept versus refunded or returned, and how was the order rated. Combine with logging propensities so offline evaluation can use inverse-propensity or doubly-robust estimators rather than accuracy on a biased sample. This is unglamorous and probably worth more than any architecture change.

4. Split retrieval from ranking.
A single pairwise classifier over the catalog forces a precomputed table. I'd use a two-tower encoder plus approximate nearest neighbor search for candidate generation (cheap, refreshes as the catalog changes, handles new SKUs), then a cross-encoder reranker over the top ~100 that can see the ordered item, the candidate, store context, and the customer together. Retrieval gets coverage; reranking gets precision. It also makes the personalization layer from the last turn much cheaper to apply, since it only touches the shortlist.

5. Hard constraint layer, separate from the model.
Allergens, dietary and religious restrictions, alcohol, pharmacy, infant formula, and large price deltas should be enforced by rules that sit outside the scorer and cannot be overridden by a high probability. The errors that damage trust most are categorical, not marginal — and a learned model trained on aggregate approval will occasionally produce them.

6. Evaluate on the coverage/approval frontier, not a point.
The precision-against-golden-set metric from phase 1 is rank-insensitive and head-biased. I'd report the full curve of approval rate versus coverage as the abstention threshold sweeps, broken out by category and by customer history depth. That prevents shipping a model that looks better only because it helps fewer people.

“We compared the Abnormal detection system—a multi-stage pipeline combining behavioral models, downstream rules, and an LLM critic”

How does an LLM critic fit into the Abnormal pipeline? What is its role?

Abnormal says their classification cost is $20/million messages. Based on frontier model cost, LLMs only fire for 0.1% of messages.

This suggests that the critic sits off the default path, invoked on a small minority of messages after cheaper stages have already decided.

The use of the word “critic” could mean that (assuming the prior stages are tuned more for recall than precision) the LLM is being used to suppress false positives but this isn’t stated in the blog post.

A full pipeline could look like

1. Lighter-weight model/model cascade with a low threshold sends all cases above a certain score to the next step
2. LLM reviews likely positives based on specified guidelines

“the efficacy comparison reflects this specific evaluation set, not production performance.”

Are there any obvious gaps in the curation of the data set? Beyond the fact that Abnormal’s models are trained to be aligned with Abnormal’s security analyst decisions

1. Selection — the attack set is conditioned on Abnormal having caught it. Attacks Abnormal never flagged don't reach that queue.
2. Selection — the safe set is likely hard-negative enriched.
3. Label taxonomy — "safe" and "graymail" overlap by construction
4. Temporal — contamination is checked in one direction only
5. What each system was shown

“returns a decision—attack, spam, graymail, or safe”

How does Abnormal typically deal with these four categories of decisions? What is the action taken based on the decision?

“Each model ran as a single-pass classifier—given a message, return one verdict (attack, spam, graymail, or safe), with no surrounding pipeline. Each was run under detection-oriented prompting suited to the scenario.”

1. Are the models given definitions of these verdicts?
2. Are they given any features besides the message in the email?

(1)

Not stated in the blogpost but they probably were

(2)

Partly stated, and only in the negative. The Executive Summary asserts the models lacked "organizational context—behavioral baselines, sender relationship graphs, and campaign-level signals—that a message-level LLM call cannot replicate." Campaign-level dedup also strips the cross-message repetition that makes commodity attacks trivially detectable.

What's never specified is what "a message" contained: headers, SPF/DKIM/DMARC results, envelope sender vs. display name, URLs, attachments, thread history, or body text alone. That single choice plausibly moves the results more than anything else — authentication headers alone separate a large share of commodity phishing.

Using the negative example sourcing they described could still lead to some false negatives if similar queries are included in the search log. Is this a significant enough risk to be concerned about?

This would be pretty rare and neural networks are fairly robust to a small percentage of "noisy labels.”

Standard industry practice is to not treat as negatives other cases where the product ID was identical so this should mitigate the issue somewhat.

The soft loss function also reduces the impact {somehow}.

How exactly does the weighting in the self-adversarial re-weighting influence the training process?

The weights are inside the loss function

“where the in-batch negative examples are given different weights as a function of the model’s prediction”

How are the weights determined in the self-adversarial re-weighting?

The weights aren’t manually determined. The formula above acts as a scale that assigns a specific weight to every single negative in the batch based on its current score.

How do the special tokens work in product embeddings?

{Full explanation unclear}

It’s able to use these to learn that a match in one category might be more impactful than matches in other categories.

Let’s say the tokens in the [PN] segment were most predictive of the final outcome; the Self-Attention mechanism will then pool more information from the [PN] segment into the final [CLS] vector.

For converting products to embeddings, is there any capacity in their current methodology to incorporate images or reviews?

If not, how could they do it?

Images

1. Late Fusion
   1. pre-trained Image Encoder (like a ResNet or a Vision Transformer / ViT) to generate a separate "Visual Embedding"
   2. visual vector would be concatenated with the text-based vector generated by the current Transformer
2. Early Fusion
   1. treat the image as a series of "Visual Tokens" (patches of the image)
   2. These visual tokens would be fed into the Transformer alongside the [PN] and [PBN] tokens. The Self-Attention mechanism would then learn to correlate the word "Milk" in the title with the white jug in the image

Reviews

1. Aspect Extraction
   1. Use a separate LLM to summarize thousands of reviews into "Aspect Tags" (e.g., "Very Fresh," "Good Value," "Easy to Open")
   2. These tags would be added as new tokens inside the [PAS] (Product Attributes/Stats) segment
2. Sentiment Weighting
   1. Calculate a "Global Sentiment Score" for the product based on reviews
   2. This score could be used as a scalar multiplier for the final embedding score.
   3. If two products have nearly identical semantic embedding scores, the one with higher positive review sentiment would receive a slight "boost" in the final ranking.
3. Virtual "Searchable" Reviews
   1. Use the top 5 most helpful reviews as additional text input.
   2. Create a new special token, such as [PRV] (Product Reviews), to append to the product tower input.
   3. This allows the model to match queries that use "consumer language" (e.g., "creamy milk for coffee") which might not appear in the formal manufacturer-provided product name ([PN])

How does their multitasking regime stabilize model training?

User clicks are noisy so adding another objective function mitigates the label noise.

Also helps with Cold Start: when a brand-new milk brand is added to Instacart, it has zero clicks. Without multitasking, the model wouldn't know where to put the new product because it has no behavior data to learn from.

With multitasking, the model sees the brand and category metadata. Because it was trained to cluster those features, it automatically places the new product in the correct neighborhood of the vector space, making it instantly searchable.

How does their multitasking regime ensure that products belonging to the same category or brand are clustered in the vector space?

Embedding vectors sent to brand and category heads. Products with the same brand or category will have their vectors be moved closer together.

How do they determine the weights for the different tasks in the overall loss function?

This isn’t explained in the blogpost.

Industry best-practice would be Grid Search with different weights then look at {evaluation metrics}

Why do they use conversion frequency instead of conversion rate when curating their dataset?

Conversion rate’s utility is impacted by noise from low-incidence products.

Conversion frequency ensures we only use examples where the query and product are very likely to be related

What is meant by “allowing the model to attune to the unique characteristics of the two domains.”

Search queries are usually shorter, informal and more likely to be misspelled. The product strings are cleaner and more organized.

“the few remaining ones not yet cached in FeatureStore are computed on the fly” - how is the caching done?

For the vast majority of the catalog (the "Head" and "Mid-tail" products), embeddings are generated in nightly batch jobs and stored in the FeatureStore.

For each search query coming in, we send a batch of product IDs to the FeatureStore. Any product ID that comes back empty is flagged as a "Cache Miss.". These missing IDs are sent to a dedicated inference service running the Product Tower. The newly computed vector is used for the current search and immediately written back to the FeatureStore so it’s available for the next person who searches for it.

What would be the best way to approach this problem today?

1. Instacart’s model was unimodal (text-only). Today, you would use a Contrastive Vision-Language (CLIP-style) architecture, like SigLIP 2 or Amazon Nova Multimodal.
2. LLM-Driven Query Expansion & Rewriting
3. Two-Stage Retrieval (Dense + Rerank)
   1. Matryoshka Embeddings then Cross-Encoder or ColBERT
4. Synthetic Data Generation
   1. You feed your product catalog to an LLM and ask it to "Imagine 50 different ways a user might search for this item."

Does Abnormal sit in the pre-delivery or post-delivery layer?

Abnormal is a post-delivery architecture. Its API-based architecture connects directly to Microsoft 365 and Google Workspace without MX or DNS changes — the provider delivers the mail, then Abnormal reads it via API and removes anything malicious. Abnormal's own architecture page states the tradeoff: operating post-delivery means messages briefly exist in the mailstore before remediation—typically measured in seconds.

One narrow pre-delivery exception, added Apr 2026: Auto-Forwarding Protection for Microsoft 365 introduces pre-delivery scanning and remediation for emails that are automatically forwarded to third-party tools. That covers one flow, not general inbound.

“where bad actors change a malicious email campaign’s content” - change relative to what? Do they mean that a single campaign can have many different kinds of content, subject etc, potentially using dynamic generation?

Yes, and yes.

“This increases the probability that some of the email variations will bypass traditional email security controls”

What kinds of controls specifically will be bypassed by this?

Content hashes / fuzzy fingerprints — a hash of the body identifies one exact message. Varying "Geeks Squad" vs "Geek_Squad" and randomizing the subscription number changes every hash. Fuzzy fingerprinting tolerates small edits, but the randomized fields are exactly the tokens it keys on.

Subject-line and body regex rules — an analyst-written rule matches a literal string. Rotating between service update and order status falls outside it.

IP and domain reputation, blocklists — reputation accrues per sender IP and domain over time. Fresh domains (geeks-squad51[.]com, geek-squadupdate28[.]com) and rotating IPs have no history to score, so they arrive unclassified rather than bad.

“Abnormal has trained and fine-tuned the BERT models on our own unique data sets”

Are they using BERT embeddings as a feature in a larger model and fine-tuning those embeddings by training on a labeled dataset of final classifications? Or are they training it independently?

Note that ‘trained’ here might mean pre-training on email corpus for domain adaptation.

1. What the sources state. The BERT post says several additional models were built on top of the fine-tuned BERT. "On top" is stacking language. End-to-end training would more naturally be described as one model.

2. What the described architecture requires. The final classifier consumes long-horizon behavioral aggregates — sender-recipient counts, vendor relationships, time since first contact. Aggregations over history aren't differentiable, and a tabular feature set of this shape is almost certainly served to a tree ensemble, which has no gradient path at all. Joint training is therefore not just unattractive but unavailable.

3. What deployment economics favor. A frozen encoder is computed once and cached, serves multiple consumers (campaign similarity, intent scoring, downstream detectors), and doesn't need retraining every time a behavioral feature is added. Joint training would couple a 110M-parameter encoder to a feature set that changes weekly.

Each of these alone is suggestive; (2) is close to dispositive.

Where joint training could still exist: inside the BERT-side models specifically. If an intent classifier is a head sitting directly on the encoder with no behavioral features, that head and the encoder can be trained together. That's joint training within the text subsystem, still separate from the final verdict model. This is a hybrid, not a counterexample.

“BERT can bidirectionally analyze and read text from left to right and right to left. This allows Abnormal to be incredibly accurate and efficient”

What does bi-directionality have to do with efficiency? Are they drawing an implicit contrast with RNNs?

Not super sure but it doesn't matter too much.

“Abnormal is always learning through a combination of supervised and unsupervised machine learning”

What is the role of unsupervised machine learning in Abnormal’s process? Are there any references online?

Inference — three distinct roles, non-overlapping:

1. Behavioral baselining. The per-customer "history systems" in the ML overview build a distribution of normal communication per user and per relationship. No labels exist for "normal"; the model estimates a density and scores deviations. This is the load-bearing use, and it's what makes the company name literal.
2. Campaign clustering. Grouping polymorphic variants by embedding proximity requires no labels. The BERT post's claim about determining whether two emails belong to the same campaign is unsupervised at inference time.
3. Self-supervised pretraining. Continued masked-language-model training on their email corpus adapts BERT's vocabulary to business email. Technically self-supervised rather than unsupervised, though marketing copy rarely distinguishes them.

https://abnormal.ai/learning/anomaly-based-detection

“they exploit trusted relationships between internal and external identities”

Does this mean account compromise or something else?

Path 1: The account is genuinely controlled by the attacker (compromise). Mail comes from the real address with real history. Authentication passes because it is the sender. Covers internal account takeover, which the post names as a target category, and vendor email compromise — a real supplier's mailbox used to send a fraudulent invoice mid-thread.

Path 2: The account is not compromised, only imitated (impersonation). A lookalike domain, a spoofed display name, or a free-mail account with the CEO's name. Nothing is breached; the attacker borrows the identity's credibility. This is where the Geek Squad example in the same post sits — brand impersonation with no compromised asset anywhere.

“It analyzes every email from every identity across thousands of contextual signals”

How can thousands of contextual signals be analyzed within the latency constraints?

First, the premise is looser than it sounds. Post-delivery buys enormous slack. The budget isn't the ~100ms an inline gateway must hit to avoid delaying SMTP; it's seconds, since the message is already in the mailbox and the goal is removal before a click. Abnormal's own architecture page puts remediation at seconds. That's one to two orders of magnitude more headroom than the question assumes.

Then, "thousands of signals" is cheap for the model itself. A gradient-boosted tree scoring a 5,000-dimensional vector is sub-millisecond — traversal depth doesn't scale with feature count, only the features actually used at split points get read. Even a dense neural net over thousands of inputs is a few matrix multiplies. Model inference is not the bottleneck.

The real cost is feature retrieval, and it's mostly precomputed. Inference (not stated in the posts, but standard for this architecture and consistent with their description of layered representations over long time periods):

• Precomputed aggregates. Sender-recipient history, vendor relationship state, per-user baselines — these are updated in batch or streaming and read as a keyed lookup. Thousands of features collapse into a handful of feature-store fetches.
• Cheap per-message extraction. Header parsing, URL and attachment metadata, string comparisons. Microseconds each.
• The genuinely expensive item is BERT. One transformer forward pass dominates everything else combined. Mitigations: batching, caching embeddings for repeated campaign content, distillation or a smaller variant, and cascading — run the encoder only when cheap signals leave the verdict uncertain.

Cascading is the key architectural idea. Not every email pays the full cost. A tiered pipeline scores most mail with cheap features and escalates only ambiguous cases to expensive models. If 95% of traffic resolves at tier one, average latency and cost are dominated by the cheap path while the tail gets full treatment.

What the marketing phrase probably means: thousands of features exist in the feature space, not that thousands of independent computations run per email. Most are precomputed aggregates over shared history, and the count is a proxy for richness rather than per-message work.

“Ability to analyze both North-South and East-West traffic flow”

What does this mean?

North-south — mail crossing the organizational boundary. Inbound from external senders, outbound to external recipients. This is what a gateway sits in the path of.

East-west — mail between identities already inside the environment. Employee to employee, and any lateral movement after an account is compromised.

“Anomaly Detection: Ability to apply the risk models to channels (like email) by aggregating risk signals”

What do they mean by aggregating risk signals?

Three distinct senses it could carry

1. Combining many signals into one verdict — the ensemble reading. Hundreds of individual indicators (unfamiliar sender, unusual send time, urgent language, first-time vendor) get fused into a single risk score by the model. Most likely primary meaning, since it matches the "thousands of contextual signals" claim in the same section.
2. Rolling raw events into features over time — the feature-engineering reading. Individual emails get summarized into per-sender and per-relationship statistics (count of prior messages, days since first contact, typical send hours). This matches the first post's description of layered representations built over long time periods, and it's aggregation in the more literal, SQL sense.
3. Grouping related detections into one incident — the alerting reading. Fifty variants of one campaign surface as a single case rather than fifty alerts.

“automatically remediating attacks”

What does this mean?

The core meaning: removing a delivered message from the mailbox without a human acting. This follows directly from the post-delivery architecture. A gateway blocks — the message never lands. An API-based system can't block on the default inbound path, so its equivalent action is to reach into the mailstore after delivery and pull the message out. "Remediate" is the vocabulary the architecture forces; there's nothing to remediate if you prevented delivery in the first place.

What "automatically" is contrasting with. The manual alternative is an analyst receiving an alert, investigating, running a search across all mailboxes to find every copy of the campaign, and purging each one. One of the customer quotes in the first post names exactly this pain — remediating phishing messages manually consuming hours. Automatic means the platform executes on its own verdict, and the practical claim is that removal happens within seconds of delivery rather than hours.

Actions that plausibly fall under the term, from most to least certain:

• Deleting or quarantining the message across every recipient mailbox in the campaign, not just the one that triggered detection
• Moving to a quarantine folder the security team can review, rather than hard deletion
• Downstream account actions when the trigger is compromise rather than a message — forcing sign-out, revoking sessions, resetting credentials

The first is definitely in scope. The third is a different product surface (account takeover protection) and isn't what this sentence is about.

The tradeoff worth naming: automatic action on a false positive removes legitimate business mail from someone's inbox. That's the operational cost that makes the BERT post's emphasis on reducing false positives load-bearing rather than decorative — a detection system that only alerts can tolerate more noise than one that acts unilaterally.

“Our security analysts now use GPT-4 through the secure Azure API as a tool in their arsenal.”

Who are security analysts? What background do they typically have, and how do they fit into the broader company?

Three distinct functions, MECE by output:

1. Triage / escalation response — work customer-reported or customer-escalated messages, deciding attack vs. spam vs. safe. Output: a verdict, fast.
2. Detection quality / labeling — produce the ground-truth labels that train and evaluate models, and audit FP/FN samples. Output: labeled data and error diagnoses. This is the function post 1's labeling section is scaling.
3. Threat research — characterize new campaigns and attacker TTPs, feed signatures or new detector requirements to engineering. Output: intelligence and detection requirements.

At a detection company, analysts sit between customers and ML engineering, unlike at an enterprise where the SOC is a cost center defending one org. Practical consequence: their labeling throughput is a hard constraint on model iteration speed. That's the actual economic argument in post 1 — GPT-4 doesn't replace analysts, it multiplies the number of messages a fixed analyst headcount can adjudicate, which is why it "helps scale our labeling resources and decrease our response latency for misclassifications".

“Look up suspicious attributes from our vast feature store”

What does it mean to look up suspicious attributes? Why do we need to go to the feature store

I’m not entirely sure. It could mean that they compare the current email’s suspicious attributes to the values of those same attributes for the broader population.

“Summarize an email’s processing logs”

What are email processing logs?

Processing logs are the record of what Abnormal's own detection pipeline did to a message — not mail server delivery logs (SMTP handoffs, bounces), which is the more common meaning of "email logs" elsewhere.

For a single message that would include: which features were computed and their values, which models scored it and what scores came back, which rules or thresholds fired, and the final verdict with the path that led to it. Essentially a trace of one message's journey through the system.

The reason an LLM helps here is volume and format. A single message can generate a large, deeply nested, machine-oriented trace — hundreds of feature values, multiple model outputs, timing data. An analyst asking "why did we let this through?" would otherwise have to read all of it to find the two or three fields that explain the verdict. Summarization turns that into a paragraph. That's also why the next bullet is comparison: the most common diagnostic question is "these two emails look nearly identical, why did we catch one and miss the other?" — which means diffing two traces to find the field that diverged.

“with an internal vector store of labeled messages”

How is this internal vector store of labeled messages being used? Is it being passed in the prompt or some other way?

Not fully clear from the post but one possible interpretation:

The mechanism is retrieval into the prompt. Embed the email under review, run nearest-neighbor search over the store of previously labeled messages, and inject the top matches with their labels as few-shot examples alongside the analyst prompt. GPT-4 then classifies the target with those neighbors as reference points.

“In some instances, we can use GPT-4 at a near-human-trained level of accuracy to determine if a message is malicious.”

A GPT-4 based classification will probably have a few seconds of latency. In what circumstances would this kind of classification be used?

They say “we can reliably scan for messages that our primary system falsely detected or missed” But does that mean that the GPT-4 pipeline is running for every message or just a selected subset?

Answer: a subset, and running after the primary verdict rather than inline.

The source settles the volume question, just in a different section. The model bootstrapping paragraph states that “the costs of running such a large model quickly become infeasible for the billions of messages processed every day“ — that's the stated reason lighter models get trained on GPT-4-generated synthetic data instead. Cost rules out running GPT-4 on everything, and latency rules it out independently.

The stronger clue about when it runs is the phrase "decrease our response latency for misclassifications." You only have a misclassification to respond to after the primary system has already delivered a verdict. So GPT-4 isn't in the blocking path deciding whether to deliver a message — it's auditing decisions already made. Email security has an unusual property that makes this workable: messages can be retracted from a mailbox after delivery. A verdict reversal minutes later still prevents most of the harm, because the window that matters is when the user reads and acts on the email, not when it arrives.

Inference on how the subset gets chosen. The obvious candidates are messages where the primary system's score sits near its decision threshold, since that's where errors concentrate; messages a customer reported; and messages resembling a newly discovered attack, found by searching for neighbors of a confirmed case. Sampling ordinary high-confidence traffic at a low rate would also make sense as a way to measure the false-negative rate you can't otherwise see, but the post doesn't say this.

“for our high-volume detectors”

What do they mean by high volume detector? What would be an example of this kind of detector?

"high volume" describes the traffic the detector sees, not the number of attacks it catches. Post 2 draws the same line explicitly: it says Abnormal traditionally uses lightweight ML models that run on nearly all traffic, and reserves LLMs for specific segments of emails where the added cost is justified. A high-volume detector is one on the "nearly all traffic" side — it scores every message, or a large fraction of them, at a per-message budget of milliseconds and fractions of a cent. Against billions of messages daily, that budget is what forces a small model rather than an LLM.

Example: a BEC / payment-fraud detector. Every inbound message has to be scored for it, because you can't know in advance which emails contain a fraudulent payment request. The base rate is extremely low — a handful of true positives per million messages — so the model runs constantly and fires rarely. That combination is exactly what makes synthetic data attractive: real labeled examples of a specific attack variant are scarce, so GPT-4 generates plausible ones to train the cheap model that does the actual scanning.

The contrast case would be a detector that runs only on messages already flagged as suspicious, or only on those with attachments. Those see a filtered slice of traffic, so a more expensive model per message is affordable.

“we have pipelines to generate synthetic data with GPT-4 that lighter-weight models are trained on”

How do they know that GPT-4 is capable of generating good/useful synthetic data? It doesn't look like they present any evaluation results so it’s not clear to me why data augmentation of this form would be helpful.

The test is downstream, on real data. Train the lightweight model on the synthetic corpus, evaluate it against a held-out set of real emails with human labels. Judging the generations directly — do they look realistic — is a proxy that fails in both directions: plausible emails that teach nothing new, clumsy ones that still fix the right boundary.

The binding constraint is the real evaluation set, not the generation. For a new attack type you might have a dozen confirmed cases. Enough to measure precision, not recall — and at a base rate of a few per million, the false-negative denominator is invisible. The practical move is to ship in shadow mode, where the model scores traffic without acting, and let analysts adjudicate what it surfaces. That measures recall and builds the real labeled set that eventually replaces the synthetic one.

The failure mode to watch is divergence from real attacks. GPT-4's notion of a payroll-diversion email comes from pre-training, not from what attackers send this month. If they've drifted, the pipeline yields a confidently wrong training set and every signal inside the synthetic loop still looks healthy. Only contact with real labeled attacks exposes it.

“Our vast datasets across attack types allow us to train our own large language models”

Do they mean "train" or "fine-tune" here? Training an LLM from scratch seems beyond the capabilities of a company of this size and focus.

Pretty sure they mean fine-tune based on the second post.

“For the task of classifying a given email as a threat, Abnormal traditionally uses lightweight machine learning (ML) models that run on nearly all traffic. However, for certain cases, LLMs have additional benefits that make them worth the cost to run on specific segments of emails.”

How do they determine which emails the LLMs will run on as opposed to the lighter weight models? In other places, they've mentioned that they want millisecond latency, so how does that align with LLM use?

The likely selector is the lightweight model's own score. High-confidence ‘safe’ and high-confidence ‘attack’ get resolved cheaply; the uncertain band in the middle goes to the LLM. That band is small — a fraction of a percent of traffic — which is what makes the cost tolerable, and it's where nearly all errors live, which is what makes it worth paying for. Two other segments plausibly get routed regardless of score: messages matching a newly-launched attack type where no trained detector exists yet, and messages to high-value recipients like finance or executive accounts, where a false negative is expensive enough to justify the spend on every message.

On latency, post 1 resolves the tension. Its labeling section talks about decreasing response latency for misclassifications, which only makes sense after a verdict has been issued — the LLM is auditing decisions, not making them inline. Email tolerates this because messages can be retracted post-delivery, so a reversal seconds or minutes later still lands before the user acts. Post 2 doesn't say whether the LLM classifiers it describes run inline or in that same post-delivery position; my read is post-delivery, for the same cost and latency reasons, but that's inference.

“With larger pre-trained models, we can leverage significantly smaller datasets for newer attacks. In some cases, we can use only a prompt or a single reference email to build a high-precision classifier.”

How did they determine the larger pre-trained models could work reasonably well on smaller data sets? Don't you need a lot of data to make this determination confidently anyway?

Few-shot capability in large pre-trained models was established public knowledge by 2023 — the GPT-3 paper onward. The reasonable read is that Abnormal took it as a known property and tested whether it held on their task, rather than discovering it.

I think they mean that larger pre-trained models need little training data to perform well. They're claiming few labels are needed to build the classifier, not to evaluate it. Those are different pools, and evaluation labels can be produced on demand after the fact.

The specific reason it's cheap here: they claim high precision, not high recall. Precision is measurable by sampling only what the model flagged — run it over a slice of traffic, take the messages it fired on, have analysts adjudicate those few hundred. That's a self-generating evaluation set requiring no pre-existing labeled corpus. Recall would require knowing every attack of that type in the traffic, including the ones nobody found, which is the expensive quantity. The claim is scoped to the half that's cheap to verify, and a high-precision detector that misses cases is still worth shipping when it layers on top of existing ones.

Back-testing covers the rest. Historical traffic is already stored and partly labeled, so a new classifier can be run retroactively against past confirmed attacks of that type — giving a rough recall figure against known cases without collecting anything new.

“Knowing what’s typical gives the large model a powerful “common sense” reasoning ability”

What does this mean?

A lightweight model learns what's typical only from Abnormal's own labeled emails and the features engineered for them. A pre-trained LLM arrives already knowing how business correspondence works — that invoices come from vendors you've transacted with, that a CEO doesn't email a junior employee asking for gift cards, that "urgent wire transfer, don't call me" is a strange thing for a real colleague to write. None of that had to be labeled; it fell out of reading the internet.

Why that reduces both error types. False positives: a message can look statistically odd — new sender, unusual hour, financial language — and still be obviously legitimate to anyone who reads it, like a first invoice from a newly contracted supplier. The LLM can recognize the benign story. False negatives: an attack can be statistically unremarkable on every engineered feature and still be transparently fraudulent in content. The LLM reads the pretext, not just the metadata.

The load-bearing word is "plausible." The model isn't checking facts about the tenant — it's judging whether the situation the email describes hangs together as a normal business interaction. That's a genuinely different signal from behavioral anomaly detection, which is why it composes with it rather than duplicating it. It also fails differently: an attacker who writes a coherent, unremarkable pretext defeats the common-sense check while the behavioral features may still catch the anomaly.

“Using the latest techniques for efficiently fine-tuning these models, like Low-Rank Adaptation (LoRA), we can incorporate our internally labeled email datasets into improving them.”

What do the features and target look like for this kind of fine-tuning?

The input is a text prompt, so everything gets serialized into it: the email itself (headers, subject, body) plus those out-of-band features rendered as text — most plausibly the behavioral history from the feature store, like whether this sender has ever emailed this recipient, prior volume, domain age. The post is explicit that these come from outside the message, which is the interesting part: they're injecting the same signals the lightweight models consume, formatted for a language model to read.

The label is one of attack/spam/safe. Given that post 2 lists explainability as a core reason for using LLMs at all — reasoned justifications, not just "attack" or "safe" — the target is almost certainly label plus written explanation, since a model fine-tuned to emit only a bare label loses that capability. Where the explanation text comes from isn't stated; GPT-4-generated rationales over analyst-labeled emails would be the standard approach.

Let's say fine-tuning is expensive, and I'm not sure yet if it's worth it. How do I determine whether to proceed with it or not? What factors should I consider, and are there smaller-scale evaluations I can use before committing to the full process?

Establish the ceiling before spending anything on training. Run the base model few-shot on a few hundred labeled examples. That number is your floor. Then run the strongest available model — GPT-4-class — on the same set, few-shot. That's roughly your ceiling, since a 7B fine-tune matching it is the good outcome, not the expected one. If the gap between floor and ceiling is small, fine-tuning has little room to buy you anything. If few-shot already clears your production bar, ship that and stop.

Diagnose whether your errors are the kind fine-tuning fixes. Read the failures from the few-shot run. Fine-tuning fixes definitional errors — the model has the reasoning but draws your attack/spam boundary in the wrong place, or won't hold the output format. It doesn't fix missing information: if the model fails because it can't see the sender's history with this recipient, the answer is putting that in the prompt, not adjusting weights. Sorting a hundred errors into those two buckets tells you most of what you need.

Do a scaled-down fine-tune before the real one. LoRA on a few thousand examples for a few hours is cheap relative to the full run. What you're measuring isn't final accuracy — it's slope. Train at 500, 2,000, and 5,000 examples and plot the curve. Still climbing steeply means more data pays; flat by 2,000 means you've extracted what's there and the full run won't help.

The costs that actually decide it are usually not the training run. A fine-tuned model is a hosting commitment, a retraining commitment every time attackers shift, and an eval-set commitment to detect when it's degraded. An API call to a frontier model has none of those. Fine-tuning wins on inference economics at volume — which is Abnormal's stated reason, running on a large share of billions of messages — and on latency. If your volume is low, the arithmetic rarely favors it regardless of accuracy.

How would I convince a business leader that this project was worth doing?

Show Benefit > Cost

1. Benefit
   1. Reduced False Positives and Reduced False Negatives -> $
   2. Saved Analyst Time -> $
   3. Explainability?
   4. Reduced Infrastructure Cost?
2. Cost
   1. X hrs of manpower -> $

If you were building this system today, how would you improve it?

The routing economics have changed enough to redraw the diagram. In 2023 the LLM had to be a narrow slice of traffic because inference was expensive and slow. Small fast models are now cheap enough that the "uncertain band" can widen considerably, and more importantly the LLM can move from post-delivery audit into the inline path for a meaningful share of messages. That changes the product, not just the cost line: catching an attack before delivery beats retracting it afterward, because the retraction only works if the user hasn't already acted.

Retrieval and fine-tuning shouldn't be an either/or. Post 2 frames LoRA as replacing the vector store. I'd keep both, because they carry different things. Fine-tuning encodes stable judgment — where your attack/spam/safe boundaries sit, what your output format is. Retrieval carries what changed this week: the campaign that started Tuesday, the vendor domain a customer just onboarded. Baking current threat intelligence into weights means retraining every time attackers shift, which is the wrong update cadence for the fast-moving half of the problem.

The gap I'd treat as most serious is adversarial robustness, which neither post addresses. You have an attacker-controlled text field going into a model that reasons over it. Prompt injection in the email body is the obvious exposure, but the subtler one is that attackers can iterate against your classifier much faster than they could against a feature-based model — send variants, observe which land, keep the ones that do. I'd want injection-resistant prompting where email content is clearly delimited as untrusted data, adversarial examples in the fine-tuning mix, and the LLM deliberately kept as one vote alongside behavioral signals rather than a sole verdict, so defeating it isn't sufficient.

Evaluation should be continuous infrastructure, not a launch gate. The failure mode for a fine-tuned detector isn't a bad launch, it's silent decay as attacks drift away from the training distribution — and recall degradation is exactly the thing you can't see, since you don't count what you never caught. I'd fund a standing labeled stream from analyst adjudication, a held-out set refreshed monthly with recent confirmed attacks, and automatic comparison against a frontier-model baseline on a sample, so the day the cheap model falls behind is a number on a dashboard rather than something you learn from a customer.

“Research work involves open-ended problems where it’s very difficult to predict the required steps in advance. You can’t hardcode a fixed path for exploring complex topics, as the process is inherently dynamic and path-dependent.”

I believe this, but what's an illustrative example that would help me solidify what this looks like a bit better?

Take: "Should we adopt library X for our inference stack?"

A hardcoded pipeline would be: search docs → search benchmarks → summarize. Now watch what actually happens:

• Step 1 surfaces that X was rewritten in v3, so all benchmarks older than 8 months are irrelevant. The existence of this fork wasn't knowable at query time.
• Step 2 you now need v3-only benchmarks. You find two, and they disagree by 4×.
• Step 3 the disagreement traces to one using batch size 1. Now you need your batch size regime — a question that only exists because of what step 2 returned.
• Step 4 you notice both benchmarks ran on A100s. Now: does the kernel path differ on H100? Another branch spawned by an incidental detail.
• Step 5 a GitHub issue says the maintainer left in March. Now you're researching project health, which was never in the original decomposition.

“A linear, one-shot pipeline cannot handle these tasks.”

Does one shot here mean a single tool call?

No. It means a fixed, predetermined sequence — however many steps it contains.

• Linear = no branching. Step 3 is always step 3, regardless of what step 2 returned.
• One-shot = the plan is committed once, up front, and not revised. Contrast this with the loop the post describes, where the LeadResearcher "synthesizes these results and decides whether more research is needed."

“exploring different aspects of the question simultaneously before condensing the most important tokens for the lead research agent”

How does the sub-agent decide what the most important content to include in the summary is and what the length should be? On the developer's side, is the prompt for the sub-agent just something like “summarize the most important information that you found”?

The lead agent specifies the contract, not the subagent. Principle #2: each subagent needs "an objective, an output format, guidance on the tools and sources to use, and clear task boundaries." So output format is set per-task at spawn time.

The subagent prompt (research_subagent.md, 48 lines) controls compression three ways:

1. What's important — a four-criterion filter: significant, directly relevant, precise (numbers/dates), from high-quality sources. Judged against its task, not the user's full query.
2. Length — no word count. A self-set "research budget" in tool calls (under 5 simple → up to 15 multi-part, hard cap 20 calls / ~100 sources), plus "dense in reporting."
3. Output format — not in this prompt. The lead agent specifies it per task at spawn time.

Two extras: conflicting facts get escalated to the lead rather than resolved, and there's a full section on flagging speculation and low-quality sources.

The full sub-agent prompt can be found here:

https://github.com/anthropics/claude-cookbooks/blob/main/patterns/agents/prompts/research_subagent.md

“Each subagent also provides separation of concerns—distinct tools, prompts, and exploration trajectories.”

I get the different exploration directories part, but why distinct tools? I feel like there are scenarios where two sub-agents are exploring different trajectories, but in the course of those explorations, they require similar tools.

Claude thinks that “Distinct" here doesn't mean disjoint tool sets. distinct trajectories through the same tools are the more common one. Same tool, different call sequence, no shared state.

Unclear what’s actually the case though.

“We found that a multi-agent system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents.”

Is there any reason for the choice of Opus as the lead agent and Sonnet as the sub-agent?

1. The jobs differ in kind. The lead agent's work is decomposition, delegation, and synthesis — one-shot, high-leverage reasoning where a mistake poisons everything downstream. A bad decomposition can't be recovered by good subagents. Subagent work is search-and-filter: bounded task, clear objective, verifiable output. That's the profile where a smaller model loses least.
2. Cost scales with the wrong side. There's one lead and N subagents, and the post says subagents do all the substantial information gathering. So subagent tokens dominate the bill. Putting the expensive model on the singleton and the cheap model on the fan-out is where the money is.
3. Latency is set by the slowest subagent. The post flags this explicitly — synchronous execution means the system blocks on one subagent finishing. Faster subagents shorten the critical path; a faster lead doesn't.

Is there any limit on the number of iterations for the agent system?

Three levels, and they're governed differently.

1. Outer research loop (lead spawns → synthesizes → decides whether to spawn again)
No stated numeric cap, in either the post or the prompts. The post describes it as a decision the LeadResearcher makes each round. Termination is heuristic: stop at diminishing returns, stop if time is running long, then write the report. The lead prompt's example — if you've identified the top 5 startups with high confidence, stop immediately.

2. Subagent count
Soft default 3. Guidelines scale 1 → 2-3 → 3-5 → 5-10. Hard-ish ceiling of 20, phrased as never exceed unless strictly necessary; if you think you need more, restructure.

3. Subagent tool calls
Budget by difficulty: under 5 / ~5 / ~10 / up to 15. Floor of 5 distinct calls. This is the only genuinely hard limit — over 20 tool calls or ~100 sources and the subagent is terminated.

“Multi-agent systems work mainly because they help spend enough tokens to solve the problem.”

Token usage feels like an intermediate causal step, not the actual reason. Why is this framed as token expenditure?

I think their framing is a bit confusing.

1. It's a variance-decomposition claim, not a mechanism claim (source-explicit). The sentence is doing regression-speak: they found token usage explains 80% of performance variance on BrowseComp, with tool calls and model choice bringing it to 95%. "Explains variance" ≠ "is the cause." Tokens are the measured regressor that happens to soak up most of the variation.

2. Tokens are a proxy for the thing you actually can't measure. The real quantity is something like amount of useful serial-plus-parallel reasoning and evidence gathering applied. That has no clean unit. Tokens are its observable shadow — and importantly, the one you can spend deliberately. You can't buy "more reasoning"; you can buy more tokens and hope. That framing is what makes the architecture actionable: the post's next sentence says this validates distributing work across separate context windows to add capacity for parallel reasoning.

“In our data, agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats.”

Is this ceteris paribus or just comparing the average regular chat vs the average agent use?

Most likely observed averages across production traffic, not a controlled comparison.

“Further, some domains that require all agents to share the same context or involve many dependencies between agents are not a good fit for multi-agent systems today.”

1. Could we not maintain a single global context in addition to each sub-agent's specific context?
2. Can't dependencies be handled the same way they are in data pipelines or software engineering, where we have a dependency graph?

1. Shared global context is very token-intensive (N sub-agents x G context length).
2. 1. Edges are discovered — you learn task C depends on B's output only after B returns. You'd be rescheduling continuously. This isn't a deal breaker, but it would be a little tricky to handle..
   2. The lead agent might not be good at coordinating these dependencies well.

“When a user submits a query, the lead agent analyzes it, develops a strategy, and spawns subagents to explore different aspects simultaneously.”

How does the lead agent determine the decomposition of the different aspects to be explored? On the developer side, is this something we only include in the prompt or are there other ways we can make it better at making this determination?

How it decides (source-explicit, from research_lead_agent.md): classify the query into depth-first (multiple perspectives on one question → 3-5 methodological angles), breadth-first (independent sub-questions → enumerate and split), or straightforward (single investigation). Then plan, with an explicit instruction to define crisp boundaries between sub-topics to prevent overlap.

Beyond the prompt — five levers, roughly in order of payoff (inferred):

1. Evals on decomposition itself. Grade the plan, not just the final report. Cheap, fast, and it's where errors originate.
2. Fixed decompositions for recurring query shapes. If "compare N vendors on M criteria" is common, template it rather than re-deriving.
3. A cheap scoping search before planning. Decomposing "compare EU tax systems" without first knowing the member list produces bad boundaries. The lead prompt does this — retrieve the list, then split.
4. Replanning after the first round. The post's synthesize-and-decide loop is a second chance; a strong first decomposition matters less if gaps get caught.
5. Model choice. Their Opus-lead setup is itself a lever here.

“In contrast, our architecture uses a multi-step search that dynamically finds relevant information, adapts to new findings, and analyzes results to formulate high-quality answers.”

This is a little vague. Let's flesh this out with some details and examples.

"Adapts to new findings" is the real differentiator, and it's implemented as an explicit OODA loop the subagent runs after every tool result — observe what's been gathered, reorient on what's still missing, decide on the next tool call, act. The lead prompt's own example shows why this can't be precomputed: for "compare EU country tax systems," first spawn a subagent to retrieve the current member list, then reason about which metrics are worth comparing, then split subagents by region. Step two's content is a function of step one's output, so no fixed plan could have specified it.

"Analyzes results" turns out to mean source skepticism specifically, not generic reasoning. The subagent is told to catch pages that speculate in future tense ("could," "may," financial projections) and report them as predictions rather than as events that happened, to prefer original sources over aggregators, and to escalate conflicts it can't reconcile to the lead agent rather than silently picking a winner.

The other two clauses are smaller. "Multi-step" is a 5–15 tool-call budget with a search-then-fetch-the-full-page loop. "Dynamically finds" means queries are written per step under a five-word guideline, broadening when hits are thin.

“evaluates tool results using interleaved thinking”

Elaborate on how this works and why it’s used.

How it works. Standard extended thinking happens once, before the model's first output. Interleaved thinking lets the model emit a thinking block after each tool result and before the next tool call, so reasoning is inserted at every step of the loop rather than only at the front. In the Claude API this is a beta feature, and the thinking blocks stay in the conversation history so later steps can see the earlier reasoning. (Mechanism is from the docs the post links to, not the post itself.)

Why is it used. Without it, the mapping from tool result to next tool call is close to reflexive — the model sees ten search snippets and immediately fires the next query. The post names three things that the reasoning step is supposed to do: evaluate the quality of what came back, identify what's still missing, and refine the next query accordingly.

Where you can see it load-bearing. Several instructions in the subagent prompt are only enforceable if there's a reasoning step between result and next action:

• The research budget — deciding you've hit diminishing returns requires assessing the results you have.
• The broaden-or-narrow rule — you can't tell whether to widen a query without judging whether hits were thin or abundant.
• The source-skepticism checks — spotting that a page is speculating in future tense, or is an aggregator rather than an original, is an evaluation of content, not a lookup.

“The LeadResearcher synthesizes these results and decides whether more research is needed.”

How does the lead agent determine whether more research is needed? On the developer side, is this something we only include in the prompt or are there other ways we can make it better at making this determination?

How it decides: it compares the facts gathered so far against the plan it wrote at the start, and stops at diminishing returns — the prompt's example is that if you've been asked for the top 5 startups and have five with high confidence, stop immediately rather than confirming further.

Best lever beyond the prompt: have the lead emit its plan as an explicit checklist of sub-questions. Then sufficiency is "which items are unanswered" rather than a judgment call over a pile of text. Second-best: have subagents report what they couldn't find, so the lead can distinguish "no such fact exists" from "nobody looked."

“CitationAgent, which processes the documents and research report to identify specific locations for citations”

How does the CitationAgent find citations?

Mechanism (from citations_agent.md in the same cookbook directory): it's a post-hoc mapping step, not a retrieval step. It receives the already-written report plus the source documents the subagents gathered, and its job is to attach claims to the documents that support them. It is explicitly forbidden from changing the report text — it only inserts citation markers.