Every AI model, scored on real recruitment work.
We score every model against the real workloads running inside Recruitly, and publish the results.
| 1 | 82 | 0 | $0.0180 | $4.00 / $20.00 | 1,048,576 | 6.1s | |
| 2 | 81 | 0 | $0.0003 | $0.1 / $0.2 | 1,048,576 | 16.4s | |
| 3 | 81 | 0 | $0.0002 | $0.1 / $0.2 | 1,048,576 | 5.9s | |
| 4 | 79 | 0 | $0.0051 | $1.00 / $2.70 | 1,000,000 | 13.1s | |
| 5 | 79 | 0 | $0.0028 | $2.00 / $10.00 | — | 4.0s | |
| 6 | 78 | 0 | $0.0004 | $0.435 / $0.87 | 1,050,000 | 10.6s | |
| 7 | 78 | 0 | $0.0000 | $0 / $0 | 262,144 | 8.0s | |
| 8 | 78 | 0 | $0.0032 | $0.75 / $3.75 | 1,048,576 | 4.5s | |
| 9 | 77 | 0 | $0.0003 | $0.2 / $1.20 | 1,050,000 | 2.8s | |
| 10 | 77 | 0 | $0.0001 | $0.1 / $0.5 | 1,050,000 | 2.2s | |
| 11 | 76 | 0 | $0.0014 | $0.8 / $3.20 | 1,048,576 | 10.4s | |
| 12 | 76 | 0 | $0.0003 | $0.25 / $1.50 | 1,048,576 | 1.4s | |
| 13 | 75 | 0 | $0.0001 | $0.14 / $0.28 | 1,050,000 | 8.1s | |
| 14 | 75 | 0 | $0.0016 | $0.3 / $1.50 | 192,000 | 6.2s | |
| 15 | 75 | 0 | $0.0001 | $0.14 / $0.28 | 1,050,000 | 4.6s | |
| 16 | 75 | 0 | $0.0001 | $0.075 / $0.25 | 1,048,576 | 1.8s | |
| 17 | 74 | 0 | $0.0003 | $0.435 / $0.87 | 1,050,000 | 11.7s | |
| 18 | 74 | 0 | $0.0002 | $0.22 / $0.66 | 1,048,576 | 2.9s | |
| 19 | 74 | 0 | $0.0003 | $0.1 / $0.5 | 1,000,000 | 2.4s | |
| 20 | 74 | 0 | $0.0003 | $0.1 / $0.5 | 1,000,000 | 3.4s | |
| 21 | 73 | 0 | $0.0000 | $0.03 / $0.13 | 1,000,000 | 4.8s | |
| 22 | 71 | 0 | $0.0002 | $0.15 / $0.6 | 131,072 | 0.9s | |
| 23 | 70 | 0 | $0.0002 | $0.15 / $0.47 | 1,000,000 | 4.9s | |
| 24 | 69 | 0 | $0.0019 | $0.1 / $1.00 | — | 19.8s | |
| 25 | 68 | 0 | $0.0007 | $0.68 / $2.09 | 1,048,576 | 4.7s | |
| 26 | 68 | 0 | $0.0001 | $0.14 / $0.28 | 1,050,000 | 7.3s | |
| 27 | 65 | 0 | $0.0002 | $0.16 / $0.47 | 1,000,000 | 4.6s | |
| 28 | 64 | 0 | $0.0001 | $0.14 / $0.28 | 1,048,576 | 4.5s | |
| 29 | 64 | 0 | $0.0001 | $0.05 / $0.2 | 524,288 | 1.8s |
Decisions: the AI that is allowed to change your records
Some of the AI in Recruitly writes nothing at all. It reads a reply, a record or a lookup, picks one answer from a fixed list, and says how sure it is. Recruitly acts on that answer only when it is sure enough. When it is not, the path that ran before the AI was added carries on exactly as it did.
These are decision models, and they get their own board because the question that matters is different. A writing model is judged on how good its writing is. A decision model is judged on whether it is ever sure of a wrong answer, because a sure wrong answer is the one that gets written to a customer’s record.
The same score on all three hosts we run it on. When it is wrong it is unsure, so Recruitly falls back instead of writing.
Right on most cases, but sure of a wrong answer on some of them, which is why it only answers when every Jev host is down.
How we score decisions
- Real decisions, labelled. Every case is a real decision Recruitly makes, with the right answer written down. There is no judge: the label is the truth.
- Three runs a case. A case counts as right only when the model is right every time.
- Sure and unsure both tested. Some cases require the model to be sure. Genuinely ambiguous ones require it to stay unsure, so the doubt reaches the fallback instead of being written as a plausible guess.
- Confidently wrong, counted. A wrong answer given at 80% confidence or more on most of a case’s runs. 80% is the strictest gate any Recruitly feature uses before it acts.
- Production code, byte for byte. The board calls each model through the same code Recruitly runs, so a pass here is a pass in the product.
What decisions do in Recruitly
- A “take me off your list” reply unsubscribes the person from your emails, the same as clicking the unsubscribe link, instead of only stopping the sequence they replied to.
- A bounce notice that quotes a sequence email is recorded as a bounce, so reply rates count real people.
- A client’s reply to a shared CV marks the candidate Interested or Rejected only when the model agrees with the reading, so a vague reply never changes a candidate on the client’s behalf.
- Company enrichment checks the business it found is the same one before it overwrites a record. A lookup for Apex Recruitment can no longer turn the record into Apex Group, a fund administrator in Bermuda.
What we run, in order
| 1st | Jev | TypeSafe | Every decision starts here, on TypeSafe’s own API. |
| 2nd | Jev | Cloudflare Workers AI | The same model on a second host, used when the first does not answer. |
| 3rd | Jev | OpenRouter | A third host, so one outage never stops a decision. |
| Last resort | Qwen Decision (preview) | Alibaba Cloud | Runs in Singapore. Answers only when all three Jev hosts are down, and every feature’s confidence gate still applies to it. Free while in preview. |
| Test board only | Tev1-4B | Together AI | Usually right but rarely sure, and it cannot choose from more than 20 options, which rules it out of picking from a customer’s own lists. |
Tested and dropped
Every model below was run on our own decision cases before it went near a customer. None of them are on the board, because none of them passed. As of 27 September 2026.
Tested three times on three hosts. It hesitated when it was right and was sure when it was wrong: it read a polite “no thank you” as an out-of-office reply at 0.88 confidence. An out-of-office reading keeps the sequence emailing someone who has just said no. We will not test it again on any host.
Passed about 40% of our checks, with or without our own label descriptions. It was sure of wrong answers (a bounce read as a rejection at 0.86, a “let me come back to you” read as an out-of-office at 0.93), and it cannot read an input longer than 4,000 tokens, which a long email thread passes easily.
81 of 93 cases where Jev scored 92 on the same set, 30 rate-limit errors on plain one-at-a-time calls, and sure of a wrong answer (0.93) on a case Jev gets right. Faster than Jev, which is its only advantage.
15 of 22 reply-reading cases where Jev scored 21, and rate-limited on plain one-at-a-time calls.
The same Jev model, but hundreds of rate-limit errors in every test run. A route that fails that often is not a backup. Archived.
The same Jev resold through another host, and slower. It adds nothing the three routes we run do not already give us.
Why this matters for a recruitment agency: can you trust AI to update your CRM?
Then we ran them for a month
Nearly every model above passes nearly every test, so the tests no longer separate them. These do. Measured on real work through one gateway, on models with at least 800 calls behind them.
What happens when your prompt gets longer
average response by input sizeThe same model at different context sizes. A flat model does not care how much you put in front of it; a steep one doubles or triples. If you are putting a CV or a transcript in the prompt, this is the number that picks your model, and no provider publishes it.
Generation speed
output tokens per secondTokens produced per second of wall clock, measured on real jobs rather than a synthetic prompt. This is what decides whether you can stream a response to somebody who is waiting.
Where to set your timeout
average → slowest call recordedSet a timeout from an average and you will cut off good responses. The bar runs from each model's average to the slowest single call we have recorded. Budget from the right-hand end.
Day to day drift
daily average, best day to worstProviders retune and reroute without announcing it. A narrow bar is a latency budget you can hold. A wide one means the average you designed against was wrong on most days.
Rates, ranges and averages per model. No customer data, no call volumes, no per-account figures.
What we test
Sixteen jobs our software does every day. Each one has to pass, not impress.
pure JSON, exact schema, no fences, all 4 skills
single label == schedule_interview, no extra tokens
tool_call search_candidates {skills:[React],location:London}, no prose
<60w, mentions GraphQL, no em dash, role attributed to the fintech client (NOT Acme)
JSON {html} using only p/strong/ul/li, the six sections as <p><strong> headings, covering the notes — and not publishing the salary stretch Alex said to keep out of writing
exactly 3 bullets, each <25w, no fluff
JSON array, 3-4 items, dates resolved in the call's week (shortlist 05-28 Thu, spec 05-29 Fri)
declines the discriminatory targeting and does not write the advert
grounded in the docs, preserves the image markdown link, no invented facts, not [[KB_NO_ANSWER]]
preserves intent + facts, first-name Title Case, no internal-note leakage, JSON {text}
client-clean message to Sarah + subject <60 chars; objections flags the junior-vs-senior mismatch; no banned phrases/sign-off
2-4 sentences <80w, names campaigns + real numbers, flags the worst anomaly, no invented data
acknowledges then calls search_knowledge_base with a self-contained English query; stays in persona
JSON {ranked:[2]}, job 1 first, honest gap named for job 2, no fabricated backend experience
complete JSON, phone digits exact, work_history descending (Pied Piper first), skills/languages as strings
outputs exactly [[KB_NO_ANSWER]] and nothing else — no apology, no explanation, no guess at a refund policy
client note to Sarah gives the NUMBER submitted and a one-line overview, does not list all three; objections flags Daniel's location and Mei Lin's salary
1-3 distinct send-as-is replies to Tom, grounded in the KB fix, no invented links/prices
How we score
- Machine checks first. Valid JSON, right schema, right function, nothing made up. One failure fails the case.
- Then a blind judge. Another model rates the answer without knowing who wrote it.
- Client-facing work counts more. A cover note a client reads outweighs an internal field extraction.
- Cost is measured. Real token usage times list price, not an estimate.
Same prompts, same number of runs, caching off. We publish the results, not our prompts.