WhatsApp PersonalBusiness
Recruitly LogoRecruitly

Every AI model, scored on real recruitment work.

We score every model against the real workloads running inside Recruitly, and publish the results.

29 of 29 models
1Meta: Muse Wonder(via OpenRouter)820$0.0180$4.00 / $20.001,048,5766.1s
2Meta: Muse Spark 1.3(via OpenRouter)810$0.0003$0.1 / $0.21,048,57616.4s
3Meta: Muse Spark 1.2(via OpenRouter)810$0.0002$0.1 / $0.21,048,5765.9s
4StepFun: Step 5 Preview(via OpenRouter)790$0.0051$1.00 / $2.701,000,00013.1s
5Google: Gemini 4 Argon(via OpenRouter)790$0.0028$2.00 / $10.00—4.0s
6Xiaomi: MiMo V2.6 Proopen(via Direct API)780$0.0004$0.435 / $0.871,050,00010.6s
7inclusionAI: Ling 3.1 Flash(via OpenRouter)780$0.0000$0 / $0262,1448.0s
8Google: Gemini 3.8 Flash(via OpenRouter)780$0.0032$0.75 / $3.751,048,5764.5s
9OpenAI: GPT 5.6 Luna(via OpenRouter)770$0.0003$0.2 / $1.201,050,0002.8s
10OpenAI: GPT 6 Luna(via OpenRouter)770$0.0001$0.1 / $0.51,050,0002.2s
11Unbiased: Pareto 26.10 Preview(via OpenRouter)760$0.0014$0.8 / $3.201,048,57610.4s
12Google: Gemini 3.1 Flash Lite(via Direct API)760$0.0003$0.25 / $1.501,048,5761.4s
13Xiaomi: MiMo V2.6 Flashopen(via OpenRouter)750$0.0001$0.14 / $0.281,050,0008.1s
14Cohere: Command A+(via OpenRouter)750$0.0016$0.3 / $1.50192,0006.2s
15Xiaomi: MiMo V2.6 Flashopen(via Direct API)750$0.0001$0.14 / $0.281,050,0004.6s
16Z.ai: GLM 5.3 Flashopen(via Telnyx)750$0.0001$0.075 / $0.251,048,5761.8s
17Xiaomi: MiMo V2.6 Proopen(via OpenRouter)740$0.0003$0.435 / $0.871,050,00011.7s
18DeepSeek V4 Flash Vision Expopen(via OpenRouter)740$0.0002$0.22 / $0.661,048,5762.9s
19Anthropic: Claude Haiku 5.5(via Direct API)740$0.0003$0.1 / $0.51,000,0002.4s
20Anthropic: Claude Haiku 5.5(via OpenRouter)740$0.0003$0.1 / $0.51,000,0003.4s
21Stealth: Space Bunny Alpha(via OpenRouter)730$0.0000$0.03 / $0.131,000,0004.8s
22OpenAI: GPT OSS 120Bopen(via Groq)710$0.0002$0.15 / $0.6131,0720.9s
23Qwen3.8 Omni Flash(via Direct API)700$0.0002$0.15 / $0.471,000,0004.9s
24Liquid: LFM2.5 24B A2B(via OpenRouter)690$0.0019$0.1 / $1.00—19.8s
25Mistral Large 4(via OpenRouter)680$0.0007$0.68 / $2.091,048,5764.7s
26Xiaomi: MiMo V2.5open(via Direct API)680$0.0001$0.14 / $0.281,050,0007.3s
27Qwen3.8 Flashopen(via OpenRouter)650$0.0002$0.16 / $0.471,000,0004.6s
28DeepSeek V4 Flash 0731open(via OpenRouter)640$0.0001$0.14 / $0.281,048,5764.5s
29Upstage: Solar Mini 4(via OpenRouter)640$0.0001$0.05 / $0.2524,2881.8s

Decisions: the AI that is allowed to change your records

Some of the AI in Recruitly writes nothing at all. It reads a reply, a record or a lookup, picks one answer from a fixed list, and says how sure it is. Recruitly acts on that answer only when it is sure enough. When it is not, the path that ran before the AI was added carries on exactly as it did.

These are decision models, and they get their own board because the question that matters is different. A writing model is judged on how good its writing is. A decision model is judged on whether it is ever sure of a wrong answer, because a sure wrong answer is the one that gets written to a customer’s record.

Jev (TypeSafe), what we run
184 / 186
cases right every run
0
confidently wrong

The same score on all three hosts we run it on. When it is wrong it is unsure, so Recruitly falls back instead of writing.

Qwen Decision (preview), our last resort
134 / 186
cases right every run
7
confidently wrong

Right on most cases, but sure of a wrong answer on some of them, which is why it only answers when every Jev host is down.

How we score decisions

  • Real decisions, labelled. Every case is a real decision Recruitly makes, with the right answer written down. There is no judge: the label is the truth.
  • Three runs a case. A case counts as right only when the model is right every time.
  • Sure and unsure both tested. Some cases require the model to be sure. Genuinely ambiguous ones require it to stay unsure, so the doubt reaches the fallback instead of being written as a plausible guess.
  • Confidently wrong, counted. A wrong answer given at 80% confidence or more on most of a case’s runs. 80% is the strictest gate any Recruitly feature uses before it acts.
  • Production code, byte for byte. The board calls each model through the same code Recruitly runs, so a pass here is a pass in the product.

What decisions do in Recruitly

  • A “take me off your list” reply unsubscribes the person from your emails, the same as clicking the unsubscribe link, instead of only stopping the sequence they replied to.
  • A bounce notice that quotes a sequence email is recorded as a bounce, so reply rates count real people.
  • A client’s reply to a shared CV marks the candidate Interested or Rejected only when the model agrees with the reading, so a vague reply never changes a candidate on the client’s behalf.
  • Company enrichment checks the business it found is the same one before it overwrites a record. A lookup for Apex Recruitment can no longer turn the record into Apex Group, a fund administrator in Bermuda.

What we run, in order

1stJevTypeSafeEvery decision starts here, on TypeSafe’s own API.
2ndJevCloudflare Workers AIThe same model on a second host, used when the first does not answer.
3rdJevOpenRouterA third host, so one outage never stops a decision.
Last resortQwen Decision (preview)Alibaba CloudRuns in Singapore. Answers only when all three Jev hosts are down, and every feature’s confidence gate still applies to it. Free while in preview.
Test board onlyTev1-4BTogether AIUsually right but rarely sure, and it cannot choose from more than 20 options, which rules it out of picking from a customer’s own lists.

Tested and dropped

Every model below was run on our own decision cases before it went near a customer. None of them are on the board, because none of them passed. As of 27 September 2026.

Tested three times on three hosts. It hesitated when it was right and was sure when it was wrong: it read a polite “no thank you” as an out-of-office reply at 0.88 confidence. An out-of-office reading keeps the sequence emailing someone who has just said no. We will not test it again on any host.

Passed about 40% of our checks, with or without our own label descriptions. It was sure of wrong answers (a bounce read as a rejection at 0.86, a “let me come back to you” read as an out-of-office at 0.93), and it cannot read an input longer than 4,000 tokens, which a long email thread passes easily.

81 of 93 cases where Jev scored 92 on the same set, 30 rate-limit errors on plain one-at-a-time calls, and sure of a wrong answer (0.93) on a case Jev gets right. Faster than Jev, which is its only advantage.

15 of 22 reply-reading cases where Jev scored 21, and rate-limited on plain one-at-a-time calls.

The same Jev model, but hundreds of rate-limit errors in every test run. A route that fails that often is not a backup. Archived.

The same Jev resold through another host, and slower. It adds nothing the three routes we run do not already give us.

Why this matters for a recruitment agency: can you trust AI to update your CRM?

Then we ran them for a month

Nearly every model above passes nearly every test, so the tests no longer separate them. These do. Measured on real work through one gateway, on models with at least 800 calls behind them.

What happens when your prompt gets longer

average response by input size

The same model at different context sizes. A flat model does not care how much you put in front of it; a steep one doubles or triples. If you are putting a CV or a transcript in the prompt, this is the number that picks your model, and no provider publishes it.

Jev 1.13.01.5× slower with more context
under 1k0.6s
1k–4k0.9s
over 4k0.8s
TypeSafe: Jev 1.13 202609171.4× faster with more context
under 1k2.4s
1k–4k2.4s
over 4k1.7s
OpenAI: GPT oss 20b1.4× slower with more context
under 1k0.5s
1k–4k0.7s
OpenAI: GPT oss 120b2.0× slower with more context
under 1k1.1s
1k–4k2.3s
Google: Gemini 3.8 Flash1.5× slower with more context
1k–4k7.4s
over 4k11s
Z.ai: GLM 5.3 Flash1.2× faster with more context
under 1k3.9s
1k–4k3.2s
over 4k3.3s
Meta: Muse Spark 1.2 Contributor2.0× slower with more context
under 1k8.7s
1k–4k18s
over 4k17s
Google: Gemini 3.1 Flash Lite2.1× slower with more context
under 1k1.3s
1k–4k4.3s
over 4k2.7s
Meta: Muse Spark 1.3 Contributor2.9× slower with more context
under 1k12s
1k–4k36s
Qwen3.7 Flash2.8× slower with more context
under 1k3.7s
1k–4k9.3s
over 4k10s
OpenAI: GPT 5.6 Luna1.9× slower with more context
under 1k3.4s
1k–4k7.3s
over 4k6.4s
Xiaomi: MiMo V2.53.1× slower with more context
under 1k6.5s
1k–4k17s
over 4k20s
Xiaomi: MiMo V2.6 Flash5.4× slower with more context
under 1k6.3s
1k–4k14s
over 4k34s
Z.ai: GLM 5.3 Flash3.9× faster with more context
under 1k11s
1k–4k2.4s
over 4k2.8s

Generation speed

output tokens per second

Tokens produced per second of wall clock, measured on real jobs rather than a synthetic prompt. This is what decides whether you can stream a response to somebody who is waiting.

Jev 1.13.0726.3 tok/s
TypeSafe: Jev 1.13 20260917539 tok/s
OpenAI: GPT oss 20b232.3 tok/s
Mimo V2.5 Pro Ultraspeed230.2 tok/s
OpenAI: GPT oss 120b194.8 tok/s
Google: Gemini 3.8 Flash182.6 tok/s
Z.ai: GLM 5.3 Flash144.5 tok/s
Meta: Muse Spark 1.2 Contributor100.3 tok/s
Google: Gemini 3.1 Flash Lite89.2 tok/s
Meta: Muse Spark 1.3 Contributor88 tok/s
Qwen3.7 Flash67 tok/s
OpenAI: GPT 5.6 Luna66 tok/s
Xiaomi: MiMo V2.538.8 tok/s
Xiaomi: MiMo V2.6 Flash34.8 tok/s
jev: Pplx Decider V1.1 27b18.3 tok/s
Z.ai: GLM 5.3 Flash0.5 tok/s

Where to set your timeout

average → slowest call recorded

Set a timeout from an average and you will cut off good responses. The bar runs from each model's average to the slowest single call we have recorded. Budget from the right-hand end.

Z.ai: GLM 5.3 Flash3.3s → 242s
Z.ai: GLM 5.3 Flash10s → 604s
OpenAI: GPT oss 20b0.7s → 34s
Qwen3.7 Flash5.1s → 215s
OpenAI: GPT 5.6 Luna4.7s → 188s
Google: Gemini 3.1 Flash Lite3.3s → 104s
OpenAI: GPT oss 120b1.2s → 34s
Google: Gemini 3.8 Flash7.7s → 186s
Meta: Muse Spark 1.2 Contributor16s → 247s
Xiaomi: MiMo V2.517s → 226s
Jev 1.13.00.9s → 9.9s
Xiaomi: MiMo V2.6 Flash14s → 154s
OpenAI: GPT oss 20b0.7s → 4.3s
TypeSafe: Jev 1.13 202609171.8s → 10.0s
Mimo V2.5 Pro Ultraspeed4.5s → 23s
Meta: Muse Spark 1.3 Contributor21s → 111s
jev: Pplx Decider V1.1 27b1.0s → 4.0s

Day to day drift

daily average, best day to worst

Providers retune and reroute without announcing it. A narrow bar is a latency budget you can hold. A wide one means the average you designed against was wrong on most days.

OpenAI: GPT oss 20b0.2s – 1.4s
OpenAI: GPT oss 20b0.6s – 1.0s
Jev 1.13.00.5s – 1.4s
OpenAI: GPT oss 120b1.0s – 1.5s
jev: Pplx Decider V1.1 27b0.9s – 2.5s
Google: Gemini 3.1 Flash Lite1.3s – 16s
Z.ai: GLM 5.3 Flash2.2s – 5.5s
Qwen3.7 Flash2.1s – 9.9s
OpenAI: GPT 5.6 Luna3.6s – 27s
Google: Gemini 3.8 Flash4.3s – 16s
Z.ai: GLM 5.3 Flash1.5s – 132s
Xiaomi: MiMo V2.6 Flash6.7s – 22s
Xiaomi: MiMo V2.56.8s – 44s
Meta: Muse Spark 1.2 Contributor6.6s – 52s
Meta: Muse Spark 1.3 Contributor13s – 60s

Rates, ranges and averages per model. No customer data, no call volumes, no per-account figures.

What we test

Sixteen jobs our software does every day. Each one has to pass, not impress.

TC1HTML profile → JSON

pure JSON, exact schema, no fences, all 4 skills

TC2Intent classification

single label == schedule_interview, no extra tokens

TC3Tool / function calling

tool_call search_candidates {skills:[React],location:London}, no prose

TC4Recruiter outreach (short gen)

<60w, mentions GraphQL, no em dash, role attributed to the fintech client (NOT Acme)

TC5Job description from raw notes

JSON {html} using only p/strong/ul/li, the six sections as <p><strong> headings, covering the notes — and not publishing the salary stretch Alex said to keep out of writing

TC6CV → 3-bullet pitch

exactly 3 bullets, each <25w, no fluff

TC7Call notes → action items JSON

JSON array, 3-4 items, dates resolved in the call's week (shortlist 05-28 Thu, spec 05-29 Fri)

TC8Discriminatory request refusal

declines the discriminatory targeting and does not write the advert

TC9KB RAG grounded answer

grounded in the docs, preserves the image markdown link, no invented facts, not [[KB_NO_ANSWER]]

TC10Rewrite existing text (support reply)

preserves intent + facts, first-name Title Case, no internal-note leakage, JSON {text}

TC11CV-submission cover note + mismatch detection

client-clean message to Sarah + subject <60 chars; objections flags the junior-vs-senior mismatch; no banned phrases/sign-off

TC12Campaign metrics → narrative

2-4 sentences <80w, names campaigns + real numbers, flags the worst anomaly, no invented data

TC13Multi-turn conversation + tool call (Simi)

acknowledges then calls search_knowledge_base with a self-contained English query; stays in persona

TC14Candidate↔job match rationale

JSON {ranked:[2]}, job 1 first, honest gap named for job 2, no fabricated backend experience

TC15Long-document CV parse

complete JSON, phone digits exact, work_history descending (Pied Piper first), skills/languages as strings

TC17KB RAG — nothing answers the question

outputs exactly [[KB_NO_ANSWER]] and nothing else — no apology, no explanation, no guess at a refund policy

TC18CV-submission cover note — three candidates

client note to Sarah gives the NUMBER submitted and a one-line overview, does not list all three; objections flags Daniel's location and Mei Lin's salary

TC16Multi-variant suggested replies

1-3 distinct send-as-is replies to Tom, grounded in the KB fix, no invented links/prices

DC01Out-of-office auto-reply
DC02Client wants an interview and more candidates
DC03Mail-system bounce
DC04Sarcastic decline
DC05Polite role-filled decline
DC06Holding reply with nothing decided
DC07Ticket-received auto-responder is not a person
DC08Reschedule request on an agreed interview
DC09Explicit opt-out from the sequence
DC147Bare acceptance of a time proposed earlier is a decision
DC148Acknowledgement that decides nothing stays NEUTRAL
DC149Declining this role while inviting future contact is not an opt-out
DC150A human saying they are the wrong contact is not a bounce
DC151Personally answering while away is a reply, not an auto-reply
DC152Genuinely ambiguous one-liner must stay unsure
DC48WhatsApp: asks to stop being messaged
DC49WhatsApp: declines this role but wants to stay in touch — NOT an opt-out
DC50WhatsApp: genuine interested reply
DC51WhatsApp Business away message
DC52SMS: bare STOP
DC53SMS: free-text opt-out with no keyword
DC54SMS: who is this — not an opt-out
DC10Contract InMail: rates and tone apply, prep pack does not
DC11Client brief task: only the brief skill applies
DC12Nothing in the catalog applies
DC13Plain thank-you closes the chat
DC14Thank-you while an investigation is still owed
DC15Still broken after a fix
DC16Agreeing to an offer needs the agent to act
DC17Payment failure is urgent
DC74Fixed and acknowledged
DC75Answered in full, waiting on the customer to try it
DC76We promised an engineer and have not come back
DC77Repeating themselves, nothing fixed
DC78Customer asks for a person
DC79Question never answered
DC80How-to answered on the spot leaves nothing behind
DC81A broken behaviour is ticket-worthy
DC82Passed to the team is an open promise
DC83Billing settled in the chat leaves nothing behind
DC84A chaser is not the topic
DC85The chat moved on to a new problem
DC86A greeting is never the topic
DC87The passage with the steps beats the one that shares words
DC88Same feature, no answer in it
DC89The documents carry the steps
DC90Right feature, wrong question
DC91A partial answer still counts
DC92A reply that sticks to the passage
DC93An invented price is not supported
DC94Asked twice, told there is nothing
DC95Answered and moving along
DC96Just asked, still working on it
DC43Another customer, same Gmail disconnect → same incident, RT-101
DC44Unrelated problem → new, no ticket
DC45Same words, different root cause (Outlook, expired password) is NOT RT-101
DC18Delete a pipeline is destructive
DC19Listing the team is a read
DC20Removing a user is destructive
DC21Renaming a stage is benign
DC22Adding a dropdown value is benign
DC35Closing a job is consequential
DC36Rewriting the description is content
DC37Retainer and fee stages are commercial
DC38Salary and skills edits are content
DC39Reassigning the owner is consequential
DC46Switching the pipeline workflow is consequential
DC23A bug report is an issue of kind bug
DC24How-do-I is a question, not an issue
DC25Pricing enquiry is commercial
DC26Angry churn threat is an escalation
DC27Our own promise is a commitment
DC40Client agrees terms and asks for CVs
DC41Client pushes back on fees and pauses the search
DC42Routine weekly catch-up, nothing decided
DC60value-map: 'Perm' is the existing Permanent, not a new employment type
DC61value-map: the typo 'Permanant' is Permanent
DC62value-map: 'Apprenticeship' is NOT Internship — none, so a new value is created
DC63value-map: 'Info Tech' is the existing IT sector
DC64value-map: a genuinely new sector maps to none
DC65value-map: import header 'Mob No.' is the Mobile field
DC66value-map: import header 'Current Employer' is Company Name
DC67value-map: a junk internal column maps to none (stays Ignore)
DC68value-map: CV language 'Castellano' is Spanish
DC69value-map: CV education 'BSc (Hons) Computer Science' is a bachelor's degree
DC176sequence import: placeholder 'last name' is the surname field
DC177sequence import: placeholder 'company' is the company name field
DC178sequence import: placeholder 'city' is the city field
DC179sequence import: 'job title' has no field — none, never the company or a name
DC180sequence import: 'job salary' has no field — none
DC181sequence import: 'your name' is the sender, not the recipient — none
DC182sequence import: placeholder 'zip' is the postcode field
DC183sequence import: tag 'Hot leads' is the account's 'Hot lead'
DC184sequence import: tag 'VIP' is not in the account — none
DC185sequence import: stage '1st interview' is 'First Interview', not Second
DC70call pulse: candidate keen, interview agreed
DC71call pulse: client cancels the role and questions the invoice
DC72call pulse: voicemail is neutral, not negative
DC73call pulse: a polite decline with the door left open is not a bad call
DC97video topic: How To Schedule Interviews
DC98video topic: Drip Campaigns
DC99video topic: AI Matches Vs LinkedIn Sourcing For A Job
DC100video topic: AI PHONE SYSTEM
DC101video topic: Submit A Candidate To A Client For A Particular Job
DC102video topic: Home Page How To Configure
DC103video topic: Convert Lead To Deal & Contact
DC104video topic: BD Hub & Its Uses For CV Specing
DC28Strong fit needing sponsorship
DC29Wrong discipline is a reject
DC30Right skills, too junior, location unstated
DC31vocab-map: job type 'Full-time' → permanent
DC32vocab-map: salary period 'Annual' → Per Year
DC33vocab-map: industry 'Dentistry' → Medical/Pharmaceutical/Scientific (32 options)
DC34vocab-map: two fields in one call — 'Software Developer' industry + 'Manchester' region
DC110agent-pick: everyday synonym lands on the tenant's type
DC111agent-pick: abbreviation lands (perm)
DC112agent-pick: a second change in the same message is ignored
DC113agent-pick: a different field entirely is none, not a guess
DC114agent-pick: naming the field without a value is none
DC115agent-pick: reject reason from the recruiter's own words
DC116agent-pick: reject reason — withdrew
DC117agent-pick: no reason given is none, never the first option
DC118agent-pick: office by its city
DC119agent-pick: office by a street in its label
DC120agent-pick: a city the client has no office in is none
DC121agent-pick: seniority from an everyday word
DC122agent-pick: seniority — lead reads as senior
DC123agent-pick: owner by full name
DC124agent-pick: owner by first name only
DC125agent-pick: a first name two colleagues share is not written
DC131agent-pick: the full name of that same colleague IS picked
DC126agent-pick: a colleague who does not work here is none
DC127agent-pick: an existing client is Repeat business (or Client referral)
DC128agent-pick: source named outright
DC129agent-pick: workflow by what the job is
DC130agent-pick: a workflow this tenant does not have is none, held loosely
DC132match-screen: meets every essential and more
DC133match-screen: the profile says not suitable
DC134match-screen: strong on skills, but the job will not sponsor
DC135match-screen: half the essentials
DC136match-screen: judged against the job, not against each other
DC55quick-actions: CV with the client nine days, no reply → chase the client
DC56quick-actions: client replied asking to meet → schedule the interview
DC57quick-actions: interview was three days ago, nothing recorded → get feedback
DC58quick-actions: shortlisted, CV never sent → submit the CV
DC59quick-actions: two jobs in one call — offer → place, interview stage unbooked → schedule
DC137eval-judge: a good answer to the outreach rates above 1.0
DC138eval-judge: a bad answer to the outreach rates below 1.0
DC139eval-judge: a good answer to the campaign summary rates above 2.3
DC140eval-judge: a bad answer to the campaign summary rates below 2.3
DC141eval-judge: a good answer to the support rewrite rates above 1.13
DC142eval-judge: a bad answer to the support rewrite rates below 1.13
DC145eval-judge: a good answer to the refusal rates above 1.28
DC146eval-judge: a bad answer to the refusal rates below 1.28
DC153offboarding: double charge raised in chat and on a call
DC154offboarding: only the cancellation form, nothing else
DC155offboarding: customer asked us to stop emailing
DC156offboarding: business is closing
DC157offboarding: our team already discussed a downgrade on a call
DC158offboarding: customer asks for someone to call
DC159offboarding: the problem they raised was fixed
DC160offboarding: only automated follow-ups mention leaving
DC161auto-wrap: fix delivered, waiting on the customer to confirm
DC162auto-wrap: we promised to come back and have not
DC163auto-wrap: customer confirmed it works
DC164auto-wrap: one of two questions left unanswered
DC165auto-wrap: frustrated customer
DC166auto-wrap: follow-up question answered, nothing open
DC192auto-wrap: known bug acknowledged, no ticket, nothing promised
DC193auto-wrap: feature request noted
DC194auto-wrap: problem fixed by support in the chat is not outstanding
DC167share reply: one candidate wanted, one turned down
DC168share reply: an unnamed candidate is not mentioned
DC169share reply: rejects the whole list without naming anyone
DC170share reply: a hesitation about rates is not a rejection
DC171company identity: same business, legal suffix differs
DC172company identity: different business sharing one word
DC173company identity: websites and industries contradict
DC174company identity: same brand on another country domain
DC175company identity: CRM has only the name, exact match
DC186person pick: John Smith the Product Designer among ten John Smiths
DC187person pick: Brad Smith the engineer, not the famous Brad Smith ranked first
DC188person pick: Brad Smith the President, already ranked first
DC189person pick: the person is second in a list led by a colleague
DC190person pick: location separates two namesakes at the same employer
DC191person pick: a namesake at another company and a relative are not the person
DC210priority: a client ready to offer
DC211priority: a client asking for interview slots
DC212priority: a new vacancy from a client
DC213priority: feedback on CVs sent
DC214priority: a candidate's start date before resigning
DC215priority: an invoice query
DC216priority: a routine admin question
DC217priority: a second chase, annoyed
DC218priority: a candidate about to take another offer
DC219priority: a promise of our own with no pressure
DC220priority: small talk that needs nothing commercial
DC221priority: a rate question holding up a contractor start

How we score

  • Machine checks first. Valid JSON, right schema, right function, nothing made up. One failure fails the case.
  • Then a blind judge. Another model rates the answer without knowing who wrote it.
  • Client-facing work counts more. A cover note a client reads outweighs an internal field extraction.
  • Cost is measured. Real token usage times list price, not an estimate.

Same prompts, same number of runs, caching off. We publish the results, not our prompts.