What a 1997 chess engine taught me about LLM inference

TL;DR

You don't need an LLM agent to optimize inference. Across six tuning tracks on an NVIDIA L4, Tensward's expert rules and heuristics matched frontier LLM agents running at maximum effort - Astra 6 and Opus 5.5 - and achieved the highest aggregate score: 2.29 versus 2.24 for Opus 5.5 and 1.70 for Astra 6. Tensward won or tied on four of the six tracks, with no LLM in the optimization loop. you can try it here

Where this started

The first time I deployed an LLM (a fine-tuned Gemma) into an enterprise environment, I had to optimize inference on someone else's hardware, and like everyone today, I asked an LLM to help.

I kept getting the feeling it was being lazy, and once I started profiling I confirmed it: it suggested the usual suspects, tried one or two of them, and stopped, without ever systematically exploring the optimization paths.

That raised a question, why does this need to be agentic at all? The search space is finite, the hardware constraints are known, and the measurements are deterministic, which is a shape of problem we've solved before. That question became Tensward, an open-source tool that takes a model, a workload and a target environment, and figures out how the model should run there, from runtime and quantization to KV-cache and batching settings, without an LLM in the loop.

So how does it relate to chess?

As a chess fan (my peak Elo is 1950, btw), the first thing that came to mind was Deep Blue, the IBM engine that beat Garry Kasparov in their 1997 rematch. It had no neural networks and learned almost nothing in the modern sense, and instead combined four ideas:

1. Rules, the legal moves of the game.
2. Positional understanding, a hand-designed evaluation function with roughly 8,000 features such as king safety, pawn structure and piece activity, shaped together with a small team of grandmaster advisors.
3. Heuristics, accumulated chess knowledge like a curated opening book and endgame databases, which tell the engine what is usually right without having to calculate it.
4. Brute force, alpha-beta search running on 480 custom chess chips and evaluating about 200 million positions per second.

The first three encode what experts know, and the fourth makes up for everything they don't.

Why brute force doesn't transfer

Deep Blue could afford to be wrong a lot, since a bad move cost nanoseconds to evaluate and discard. In inference optimization, every "position" is an experiment where you reload the model with a new configuration, warm it up, run a representative workload, and measure latency, throughput and quality, which takes minutes to hours on hardware someone is paying for, often in a customer environment you only get limited access to.

So the objective flips: Deep Blue maximized the positions it searched, while we minimize the experiments we run, and the other three ideas are what make that possible.

1. Rules: hard constraints, and how they drift

Every environment has hard limits, for example a model with 40 GB of weights won't fit as-is into a GPU with 24 GB of memory, or onto a CPU server without enough RAM. Checking that once is easy, but the subtle part is that constraints move as you change the configuration.

Take KV-cache quantization: say an int4 KV cache freed enough memory to raise the batch size, and then you switch to int8 to recover quality. The cache now takes twice the memory, so the batch size that ran a minute ago may no longer fit, which is why Tensward tracks configuration state across the whole session and re-checks constraints after every change, catching this drift before it surfaces as an out-of-memory error or a quiet throughput drop.

2. Situational awareness: the bottleneck and the goal

The most repeated word in inference is "bottleneck", meaning whatever is stopping the model from being faster or serving more users, whether that's compute, memory bandwidth, KV-cache capacity or scheduling. A change only helps if it targets the actual bottleneck and moves the metric you care about.

If you're not an inference engineer, it's hard to even explain to an LLM what you want at each step, so in Tensward you state the goal once, for example TTFT (time to first token) or TPOT (time per output token), and the system holds that goal for the whole session, prioritizing the bottlenecks that matter for it while still improving other metrics, as long as those improvements don't hurt the main goal. This matters because optimizing for latency and optimizing for throughput often pull in opposite directions, so the main goal decides which side wins when they conflict.

3. Heuristics: situations we've already seen

This is the interesting part. Inference tuning is a balancing act where you make a change, measure its effect on performance and quality, and then decide whether to keep it or roll it back. An experienced engineer usually knows what to try for a given bottleneck, but without that experience the knowledge is hard to come by.

So we encoded it, first giving the system a picture of what healthy inference looks like and then running experiments across many models and hardware setups. For us, a heuristic isn't a generic rule like "quantize the KV cache", it's a situation defined by:

• Hardware, the GPU or CPU and its memory
• Model, its architecture, size and precision
• Runtime metrics, such as KV-cache utilization and GPU or CPU utilization
• Previous steps, meaning what was already tried, so failed paths are never retried
• Workload, the user's representative traffic

Each heuristic is validated on real scenarios, so we know whether it works and how much it actually moves the numbers. When Tensward analyzes a new environment, it matches the current situation against the ones it has already seen and ranks the candidate next steps, which in practice narrows each step down to 3–4 options.

That's the core contrast with Deep Blue, which searched hundreds of millions of positions per second, while we run a handful of experiments per step and choose the next move the way an expert would.

How we ran the benchmarks

We tested two workloads, each with three separate tuning goals: throughput, p95 TTFT and p95 TPOT. That gives six tracks. Each competitor in a track ran sequentially on the same machine, starting from the stock vLLM configuration.

Item	Setup
Hardware	One NVIDIA L4 per track, on GCP
Runtime	vLLM 0.30.0, pinned
Chat workload	Gemma 4 E4B
Agent workload	granite-3.3-8b-instruct, 16 clients, roughly 6,000-token tool-calling prompts
Tensward	Version 0.3.16
LLM agents	Codex CLI with gpt-6-astra; Claude Code with Opus 5.5; both at maximum effort
Tuning budget	60 minutes of counted session time per competitor; engine startup and background answer judging excluded
Final evaluation	Selected configurations re-measured on the track's machine after warm-up, with 1–3 scored runs per configuration

We also included vLLM auto_tune as a non-agent reference. The results below focus on Tensward and the two LLM agents.

For the combined score, the target metric gets half the weight and the other two get a quarter each, using a weighted geometric mean of improvements over the default. Answer-quality regressions can reduce the score. Configurations that break the benchmark's quality or service guardrails receive a score of 1.00, the default reference value.

This means a score of 2.29 is a composite result, not a claim that inference is universally 2.29 times faster. We also report goal-only results, because a combined score can favor a configuration that sacrifices some target-metric performance for gains elsewhere.

Answer quality was checked using approximately 200 paired answers per configuration, compared with the default and evaluated by two LLM judges in both presentation orders. The check applied no quality penalty to the reported Tensward or agent configurations. That is evidence from this check, not proof that quality is unchanged on every task.

Results

Tensward finished with the highest aggregate combined score, 2.29 versus Claude Code's 2.24 and Codex's 1.70, and won or tied on four of the six tracks under the benchmark's noise-band rule. The lead over Claude Code is small; the interesting result is that a fixed heuristic system was competitive with both agents.

Codex returned no configuration in the chat TPOT track and received the fallback score. †Its agent-throughput result remained at the default. Ties follow the benchmark's per-track noise bands; they are not claims of statistically established equivalence.

Three results are worth looking at beyond the aggregate.

On chat token latency, Tensward improved both speed and capacity. Its p95 TPOT fell from 106 ms to 60 ms, compared with Claude Code's 70 ms. At the same time, throughput rose from 2.79 to 4.87 requests per second, versus Claude Code's 4.23. That's roughly 14% lower TPOT and 15% higher throughput than Claude Code in this track.

The fastest first token wasn't the best combined result. On the chat TTFT track, Codex reached 401 ms, ahead of Tensward's 442 ms. But Tensward served 4.73 requests per second versus Codex's 3.85, about 23% more, and had lower TPOT at 62 ms versus 77 ms. Tensward won the combined score; Codex won on TTFT alone. Which configuration you would deploy depends on what you actually need.

The biggest improvement over defaults was shared by all three tuners. On the agent TTFT track, the default took 41.81 seconds at p95 to produce the first token. Tensward brought that down to 2.10 seconds, Claude Code to 2.08 seconds, and Codex to 1.96 seconds. All three also more than doubled throughput. That's a large opportunity in this particular default configuration, and all three approaches found it.

The agents also had clear wins. Claude Code won the chat-throughput track, reaching 4.79 requests per second against Tensward's 4.14. Codex won the combined score on agent TPOT. These results support using a specialized heuristic optimizer for this problem; they don't establish that LLM agents cannot tune inference well.

What this does and doesn't show

This is an initial result covering two workloads, one GPU type and six tracks from a broader planned benchmark. Different tracks ran on different L4 machines and in different regions, although competitors within each track shared the same machine. Repeated final measurements help check a selected configuration, but they don't establish how consistently an agent would find it across independent tuning sessions.

Try Tensward

Tensward is completely open source under Apache 2.0, so you can run it on your own models and hardware today: github.com/Tensward/tensward - Currently support vLLM on NVIDIA GPUs, support for llama.cpp and Apple Silicon - coming soon!