Is AI accurate enough to change your CRM?

The number that tells you whether AI is safe to change your CRM is how often it is wrong while being sure of itself, because that is the only kind of wrong answer that ever reaches your records. Every vendor will quote you the first number. Almost none of them measure the second, and it is the one that decides whether your consultants can trust what they open on a Monday morning.

We test every model that is allowed to change a Recruitly record on exactly that, publish the results, and drop any model that fails, however cheap or fast it is. This is how that testing works, what the models actually scored, and how you can put the same question to the AI in whichever CRM you use.

What does "accurate" mean for AI that updates a CRM?

For AI that updates a CRM, accurate means two separate things, and a single percentage hides the difference between them. A model can be wrong and unsure, or wrong and sure. In a product built properly, the first kind is harmless: the software sees the doubt, writes nothing, and the job goes the way it went before the AI existed. The second kind is the dangerous one, because nothing stops it. A confident wrong answer looks exactly like a confident right one, so it gets written.

That is why two models with the same headline accuracy can be completely different to live with. One that is right 95% of the time and unsure every time it is wrong never damages a record. One that is right 97% of the time and sure of its mistakes will quietly put wrong values into your database, and nobody will know which ones they are. When a consultant finds the first bad record, they stop trusting all of them.

What is a decision model?

A decision model is AI that writes no text at all. It reads something (a reply, a record, the result of a lookup), picks one answer from a fixed list, and returns how sure it is. The software acts on that answer only when it is sure enough, and otherwise carries on the old way. I wrote about the first one we put live in we put a decision model in production the day it was released, TypeSafe's Jev, which is still the model we run.

Decisions sit behind a lot of ordinary work in Recruitly that nobody thinks of as AI. When a candidate replies "please take me off your list" to a sequence, a decision model reads the reply, and the person is now unsubscribed from your emails exactly as if they had clicked the unsubscribe link, instead of only being removed from the one sequence they answered. When a mail server bounces a sequence email back with the original quoted underneath, it is recorded as a bounce, so your campaign reply rates count real people. When a client answers a shared CV, the candidate is marked Interested or Rejected only when the model agrees with the reading, so a vague reply never changes a candidate on the client's behalf. And when company enrichment finds a match, it checks the business it found is the same one before it overwrites anything. A lookup for Apex Recruitment can no longer turn the record into Apex Group, a fund administrator in Bermuda.

Every one of those is a small moment where a wrong answer costs something real: an email sent to someone who said stop, a candidate marked as rejected who was not, a client record overwritten with a stranger's details.

How do we test an AI model before it can change a record?

We test every decision model against a fixed set of real Recruitly decisions, each with the right answer written down, before it goes anywhere near a customer. As of 27 September 2026 that set holds 167 cases, and it grows every time a new feature starts making decisions or a real reply catches a model out. The rules are strict on purpose.

• Every case is real and labelled. The inputs are the kinds of emails, records and lookups Recruitly actually handles, and the correct answer is written down in advance. There is no second AI marking the work: the label is the truth.
• Every case runs three times. A case only counts as right when the model gets it right every time, because a model that is right two times out of three is wrong one day in three in production.
• Some cases require doubt. A genuinely ambiguous reply has no safe answer, so the test demands that the model stays below the confidence the feature needs before it acts. A model that is sure about something nobody could be sure about fails, even if its guess happens to be right.
• Confidently wrong is counted separately. A case is confidently wrong when the model gives a wrong answer at 80% confidence or more on most of its runs. 80% is the strictest bar any Recruitly feature sets before it acts, so a wrong answer above it is one that would have been written.
• The test runs the production code. The model is called through exactly the same code Recruitly uses to make the decision for real, so a pass on the test is a pass in the product.

What did the models actually score?

Jev got 164 of 167 cases right on every run and was never confidently wrong, on all three hosts we run it on. Everything else we tried either scored lower, was sure of wrong answers, or could not stay up under ordinary use. These are the results as of 27 September 2026.

Model	Where it runs	Cases right	Confidently wrong	Where it stands
Jev (TypeSafe)	TypeSafe direct, Cloudflare Workers AI, OpenRouter	164 of 167 on every host	None	What we run, in that order
Qwen Decision (preview)	Alibaba Cloud, Singapore	123 of 167	5 cases	Last resort, only when every Jev host is down
Tev1-4B	Together AI	127 of 144 (the earlier, smaller set)	None	Test board only; cannot choose from more than 20 options
Drex	Nace	81 of 93, where Jev scored 92	Sure of a wrong answer at 0.93	Dropped; 30 rate-limit errors on plain calls
Kev 4B	OpenRouter	15 of 22 reply cases, where Jev scored 21	Not measured	Dropped; rate-limited on plain calls
Fastino GLiNER 2.5	Fastino	About 40% of checks	Read a bounce as a rejection at 0.86	Dropped; cannot read inputs over 4,000 tokens
Laya and Laya Multilingual	laya.studio, OpenMayhem, Impossibl	Hesitant when right	Read a polite decline as an out-of-office at 0.88	Dropped after three tests on three hosts
Jev via Vercel AI Gateway	Vercel	The same Jev model	n/a	Archived; hundreds of rate-limit errors every run

Two results in that table matter more than the rest. Qwen's decision model is free while it is in preview, and on price alone it would win. It was also sure of a wrong answer on five cases where Jev was not sure of any. Drex is faster than Jev and scored respectably, and on one case Jev gets right it gave the wrong answer at 0.93 confidence. Both results are exactly the failure a decision model exists to avoid.

The Laya result shows what that failure looks like in practice. It read a polite "no thank you" as an out-of-office reply, at 0.88 confidence. In a real agency that means the sequence carries on emailing someone who has just asked it to stop, which is a bad look for the consultant whose name is on the email and a consent problem for the agency behind it. A model that makes that mistake once in a test will make it hundreds of times a month across a customer base.

Why did we choose the model that is neither the cheapest nor the fastest?

We chose Jev because it is the only model we tested that knows when it does not know. On the three cases it gets wrong, it is unsure, so Recruitly falls back instead of writing. Price and speed only matter once that is true, and for decisions Jev is already remarkably cheap: at its listed price, a thousand decisions cost about three US cents.

The same principle decides where the other models sit. Qwen still has a place as the last resort, answering only when all three Jev hosts are down, and every feature's confidence bar still applies to its answers, so its occasional certainty cannot reach a record unchecked. Tev1 from Together AI is usually right but rarely sure, and it cannot choose from more than 20 options, which rules it out of decisions like mapping a value onto one of a customer's own lists. It stays on the test board, where we can watch it improve.

What happens when the AI is not sure?

When the AI is not sure, it writes nothing and the old way runs, exactly as it did before the AI was added. A reply the model cannot read confidently goes through the path it always went through. A company match it cannot confirm leaves the record as it was. A CV-share reply it is unsure of leaves the candidate where they are, for the consultant to look at. The recruiter loses nothing by the AI declining, and that is the whole reason declining is safe.

The same thinking applies to outages. Jev runs from three hosts in order (TypeSafe directly, Cloudflare Workers AI, then OpenRouter), with Qwen behind them, so one provider having a bad afternoon never stops a decision. If every route failed at once, the old path would carry on and most recruiters would notice nothing. Lokesh set out why this matters in can you trust AI to update your CRM records: the AI has to be able to decline, you have to be able to tell afterwards what it did, and the old way still has to work. The testing above is how we hold ourselves to the first of those.

What does this change for a recruiter?

For a recruiter, it means the records you open are records you can believe, and a growing share of the admin behind them is done before you get to it. The unsubscribe is handled, the bounce is filed, the reply is sorted, the company record is right, and none of it needs you to check whether the software guessed. Your time goes on the conversations only you can have: the candidate deciding whether to move, the client who needs to hear the salary is wrong, the offer that needs closing.

AI will never replace the recruiter, and this is what it looks like when it earns its place on the desk instead: it takes the small, constant decisions that used to eat the afternoon, does them carefully, and hands back anything it is not sure of. A consultant who no longer spends an hour a day tidying records can run far more jobs than one who does, and that is the change worth having.

How can you test the AI in your own CRM?

You can test the AI in any CRM with four questions and a handful of your own emails, and a vendor who has done the work will answer all four without a pause.

1. Ask how often it is confidently wrong, not how often it is right. If the vendor has only an accuracy figure, they have not measured the thing that damages your data.
2. Bring your own ambiguous emails. A reply that could be a no or a maybe, a bounce with the original quoted underneath, an out-of-office from someone who has actually left. Nothing in a demo database is ambiguous, so the demo will always look perfect.
3. Ask what happens below the confidence bar. The right answer is that nothing is written and the old way runs. "It still makes its best guess" is the wrong answer.
4. Ask to see the results. Which models, tested on what, scored how. Our guide to telling whether a vendor's AI is real has more questions in the same spirit.

Where can you see the results?

Every result in this article is live on our AI model benchmarks page, on the Decisions board, and it updates each time we re-run the tests. The board shows every model's cases right and cases confidently wrong side by side, lists every model we tested and dropped with its result, and sits next to the boards for the text, speech and agent models we use, scored on the same real recruitment work.

We publish it because we would rather be judged on evidence than on adjectives. No other recruitment CRM shows you how the AI that touches your records was chosen, and the fact that we can is the clearest sign of how seriously Recruitly takes your data.

Written for agency owners and recruiters deciding how much to let AI change in their CRM.