Your Eval Set Is Your Product Spec

Up until now, I wrote mostly about how AI models work. We talked about embeddings, MLPs, backpropagation, why raw MLPs were not enough, and transformers. This post is a bit different. It is not about architecture diagrams, advanced math, or a new model trick. It is about something less glamorous, but just as important if you actually want to build useful AI systems: evaluations.

Most people who used statistical models, machine learning, or deep learning have heard terms like accuracy, precision, recall, and F1. These are important terms, and if you are building a classifier, they can tell you a lot. For example, imagine you build a model that needs to classify bananas and apples. If 99 out of 100 examples in the dataset are bananas, the model can learn to always say “banana” and still get 99% accuracy. The number looks amazing, but practically the model learned a useless shortcut, because it completely fails on apples.

This is why people use precision, recall, F1, confusion matrices, and other metrics. They help us avoid being fooled by one nice-looking number. But with language models, things get more complicated, because a lot of the time there is not one exact “right” answer. There are many acceptable answers, and the quality depends not only on whether the answer is factually correct, but also on how the model answered, what it missed, whether it hallucinated, whether it followed the format, and whether the output is actually useful in the product.

Take a simple question: “Who was the first president of the United States?” The model can answer “George Washington,” or “The first president of the United States was George Washington,” or “George Washington became the first president of the United States in 1789.” All of these are acceptable answers. But if the model says, “George Washington became the first president in 1777,” the answer sounds similar, but it is wrong.

This is what makes evaluation of LLMs and SLMs harder than evaluation of simple classifiers. You are not only checking if the model picked class A or class B. You are checking whether the answer is correct, useful, complete, grounded, formatted properly, and safe for the workflow.

And there is no perfect eval.

I think this is going to be one of the most important subjects in applied AI over the next few years. Everyone talks about better models, better prompts, better agents, better RAG, and better fine-tuning. But if you cannot evaluate the system properly, you do not really know if you improved it. You are just vibes-testing.

Usually, I think about evals in four broad categories: benchmarks, deterministic evals, human or expert evals, and LLM-as-judge evals.

Benchmarks are public or semi-public tests that let people compare models on the same task. This is how people compare OpenAI, Anthropic, Google, open-source models, and so on. They are useful because they create a common language. The disadvantage is benchmark-maxxing. Once a benchmark becomes important, everyone starts optimizing for it. A model can become better at the benchmark without becoming meaningfully better for your specific product.

Deterministic evals are the cleanest evals when you can use them. If the correct answer is “banana” and the model says “apple,” it is wrong. If the correct output is valid JSON and the model returns broken JSON, it failed. If the model needs to extract a number from a document and it extracts the wrong number, it failed. These evals are cheap, repeatable, and easy to understand. The problem is that not every AI task can be evaluated this way. Summaries, explanations, recommendations, legal drafts, clinical notes, sales-call analysis, and support responses usually require judgment.

Human or expert evals are sometimes the only way to get high-quality feedback. If you are generating legal documents, you probably need a lawyer. If you are analyzing medical notes, you probably need a clinician. If you are analyzing sales calls, you probably need someone who understands the sales process. The advantage is quality. The disadvantage is cost, speed, and consistency. Humans are not automatically aligned with each other, and if the criteria are not clear, human evals can become subjective and noisy.

LLM-as-judge is where you use another language model to evaluate the output. This is useful because it is much cheaper and faster than human review, so it can help you scale qualitative evaluation. But it also has risks. The judge model can be biased, prefer longer answers, miss domain-specific errors, or be fooled by confident writing. So I do not think LLM-as-judge should replace human evals completely. I think it should be calibrated against human judgment and used carefully.

But after saying all of this, I think the most important question is not which eval method you choose. The most important question is: what does “good” mean?

This sounds obvious, but I think a lot of AI products fail here. People say things like “the answer was good,” “the summary was bad,” or “the analysis was 6 out of 10.” But what can an AI engineer actually do with that?

If someone tells me the model’s call analysis was 6 out of 10, I still do not know what failed. Did it miss the objection? Did it hallucinate a detail? Was the summary too long? Did it tag irrelevant sentences? Did it fail to detect whether the sales rep handled the objection? Did it use the wrong tone? Did it ignore a key moment in the call?

A general score is not enough. This is where evals become product work, not only machine-learning work.

Let’s take the call-analysis use case. Suppose we have an LLM or SLM that analyzes a sales call. A bad eval would be: “Rate the analysis from 1 to 10.” A better eval starts by defining the exact criteria. Did the model identify the main objection? Did it avoid tagging sentences that were not objections? Did it distinguish between a current objection and a historical objection? Did it capture whether the rep handled the objection? Did it identify missing information? Did it hallucinate anything that was not in the call? Did it mark “nothing” when nothing meaningful happened?

This is much more useful, because now, if the model fails, we know where it failed. This is why I think evals should start with a failure taxonomy, not with a metric. Before asking whether the model scored 82% or 91%, you need to define the types of mistakes that matter.

In a call-analysis system, a missed real objection is one kind of mistake. Over-tagging is another. Tagging a historical objection as a current objection is another. Producing a correct label with a bad explanation is another. Giving a nice summary but missing the actual business signal is another. These mistakes should not all have the same weight, because in real products, not all mistakes cost the same. Some mistakes are annoying, some destroy trust, and some make the product unusable.

For example, in my conversational-analysis work, one of the biggest problems was over-tagging. The model would tag sentences that should not have been tagged at all. At first, this might look less serious than missing an important objection. After all, the model is “finding more things.” But in reality, over-tagging destroys trust very quickly. A model that tags too much looks active, but feels stupid. If every sentence becomes a signal, then no sentence is a signal. Users stop trusting the tags, and the system becomes noise.

This is why the “nothing” class is so important. A lot of AI systems are bad at saying “nothing happened here.” They try to be helpful, so they find something. They tag something. They summarize something. They produce an answer even when the right answer is that there is nothing meaningful here. But in real products, knowing when not to act is part of intelligence.

This is true in many domains. In sales-call analysis, the model should not tag every sentence as an objection. In support automation, the model should not invent an answer when it should escalate. In legal AI, the model should not confidently draft something when the facts are missing. In medical AI, the model should know when the information is insufficient. So if your eval does not test when the model should say “I don’t know,” “nothing,” or “not enough information,” it is probably incomplete.

Another important idea is slice-based evaluation. Global numbers can hide the exact cases that matter. A model can be 90% accurate overall and still fail on the most important 10%. It can perform well on obvious cases and fail on edge cases. It can work well in English and fail in Hebrew slang. It can work well on long, clear sentences and fail on short utterances. It can work well when the objection is explicit and fail when the meaning is implied.

For a call-analysis model, slices might include short utterances, negations, past-tense statements, ambiguous objections, Hebrew slang, sentences that sound like objections but are not, sentences that should be tagged as nothing, rep handling moments, and missing-data cases. If the model improves overall but gets worse on the slice that matters most, that is not really an improvement. It is a tradeoff, and the team needs to decide if it is acceptable.

This is why I like the sentence: your eval set is your product spec.

In normal software, the product spec usually comes before the implementation. You define how the feature should behave, engineers build it, and QA tests whether it works. In AI systems, the eval set often becomes the real product spec. It defines the behavior the model is supposed to learn. It defines what counts as good. It defines what counts as failure. It defines which edge cases matter and which mistakes are unacceptable.

If your eval set is weak, your product spec is weak. If your eval set ignores edge cases, your model will ignore edge cases. If your eval set rewards confident answers, your model will become confident. If your eval set punishes “I don’t know,” your model will stop saying “I don’t know.” And if your eval set does not reflect the real workflow, your model can look great in testing and still fail in production.

This is also why criteria must be defined before looking at the results. This is an honesty test. You cannot run the model, look at the outputs, and then quietly change the eval criteria to make the result look better. Of course, evals should evolve over time as you learn more. Real products change, users find new failure modes, and production exposes edge cases that your first eval set did not include. But changing the criteria after seeing the result is not evaluation. It is storytelling.

The better process is to version the evals. You can have Eval v1, Eval v2, Eval v3. Every time you add new criteria, new failure modes, or new slices, you preserve the history. This way you can still compare progress honestly while improving the quality of the evaluation over time.

This is how evals become a compounding asset. Every weird production failure can become a future test case. Every user complaint can become part of the eval set. Every hallucination can become a regression test. Every edge case can become a slice. Over time, the eval set becomes the memory of what the product learned the hard way.

This is also why I think the eval loop is one of the biggest moats in applied AI. Not because the metric itself is complicated, but because the eval set contains domain judgment. It contains product judgment. It contains the painful edge cases that were discovered through real usage.

The model can change, the prompt can change, the architecture can change, and the fine-tuning method can change. But the eval loop tells you whether the system is actually getting better.

So my view is that AI evals should usually be hybrid: use deterministic evals where possible, use human experts where judgment matters, use LLM-as-judge carefully and calibrate it against humans, and use benchmarks to understand general model capability without confusing them with your product reality. Most importantly, define the failure modes before choosing the metric.

Because the hard part is not calculating F1. The hard part is deciding what “good” means. And in AI products, that decision is not just a technical decision. It is the product.

That is why evals are not only measurement. They are how you encode product judgment into the system.