Comparing AI models for the work your business does

Compare a new AI model with the one your business already uses on the same task, using the same source material and the same standard for a usable result. Include the work a person has to do before the result can go to a colleague or customer.

Ryan Doser describes testing new models on two pieces of his own work: improving a landing page and reviewing his AI SEO audit instructions. His preference for those tasks over launch leaderboards is useful, although calling benchmarks useless goes too far. A published benchmark can help identify a model worth trying; it cannot settle whether replacing your current one improves the work in your company.

What should stay the same in an AI model comparison?

Keep the assignment, source material and acceptance rules unchanged for the first comparison. If you improve the instructions while changing the model, you have tested the combination and cannot tell which change helped.

Take a hypothetical business that uses AI to turn a weekly operations update into a briefing for its owner. The update contains completed jobs, overdue work and decisions waiting for the owner. The briefing is useful when the owner can see what needs a decision and trace each point to the update. More fluent prose is a secondary benefit if it leaves out an overdue job.

Save a copy of an update already suitable for the approved tool and keep the current instructions with it. Give both models that material. Decide beforehand which facts the briefing must retain, which decisions it must surface and what it should do when the update leaves something unclear. Use fictional or appropriately approved material when testing would otherwise expose confidential information.

The old model needs a fresh run too. Comparing a new model's best attempt with an old answer produced under different instructions gives the newcomer an advantage you have not measured. Record the model and settings the tool lets you identify, along with the instructions and output, so the comparison has a basis you can revisit.

How much correction does each answer need?

Judge the answer by whether someone can use it and what remains to be fixed. The person who normally relies on the briefing should check it against the original update before choosing which output they prefer.

In the hypothetical example, one model might write a shorter, smoother briefing but turn an unresolved supplier response into an agreed delivery date. The other might preserve the uncertainty in awkward prose. Those are different defects: editing a sentence and finding an unsupported commitment require different work. A preference for the smoother answer can hide the more expensive correction.

Have the reviewer mark omissions and unsupported statements, then record the time actually spent checking and repairing each version. Keep that separate from the time the model took to produce its first answer. If the reviewer cannot resolve a gap from the update, the finished briefing should leave the question open for the person who knows.

Repeat the comparison on other representative updates, including one with a changed priority or an unfinished item carried over from the previous week. A small set can reveal a reason to reject a model or investigate further. It cannot establish a general error rate. Once both models have seen the test material, retain another suitable update for checking any revised instructions.

When is changing AI models worth the work?

Change when the improvement you observed matters enough to cover the work of adopting the replacement. The result may favour the current model, a revised instruction or a new model for only part of the task.

For the weekly briefing, the owner could decide that fewer missed decisions justify a trial of the replacement while a person still checks the output. That conclusion would apply to preparing the briefing. It would give the system no new permission to change delivery dates, assign work or contact suppliers.

If the new model needs different instructions, give the current model a chance with the revision as well. This second comparison asks whether the new setup is better. Keep its results separate from the initial comparison, which asked what changed when only the model changed. Otherwise a useful improvement in how you describe the work can become a misleading reason to switch suppliers.

The trial also has to fit the company's approved tools and access. Bring unanswered questions about the provider and the information it handles into the AI vendor assessment. A good briefing does not answer those questions. Where the proposed benefit is lower cost, distinguish measured review time from cash the business can actually avoid spending.

Doser's recurring tasks provide a useful starting point: work familiar enough to judge. Keeping the comparison stable adds a way to explain the decision afterwards. The owner should be able to point to the update, the two outputs and the corrections that made changing the model worthwhile, or showed that it was better to stay.