optlens

Debugging LP and MILP models with a coding agent: a 400-case benchmark

Infeasible, wrong and working models of up to 837,000 constraints. What a debugging engine adds to Claude, and how six setups compare.

TL;DR

The six setups

All six run Claude Sonnet 5.5. All but the plugin run through our own minimal agent loop (a short system prompt and the tools), not Claude Code; section 8 measures the plugin in Claude Code itself. Every setup starts from the same first message. It holds the model's context and the question, and the agent can ask for the model's document. The setups differ only in what the agent can call.

The model context lists each constraint and variable family with its business meaning (in the benchmark, also the documented result). optlens generates it once from the model's document or code. An LLM writes the meanings, and optlens checks them against the model's actual families. optlens keeps the context as JSON you can review and commit. It shows the context the same way in every session, so every agent and every person describes the model in the same words. In Claude Code, the plugin creates the context the first time you open a model.

setup what the agent can call what it is
Bare Python with HiGHS, SCIP and gurobipy a coding agent on its own: the baseline
Tools the optlens tools, one call per action the MCP server, in any MCP client
Code ("optlens" in the tables) optlens preloaded in its Python, called from its own code what the plugin's run_python offers
Claude Code + plugin tools and run_python with the optlens skill (it also offered an analyst subagent, since removed: 2.5x the cost, no gain) what an OR user installs
optchat the code setup, plus a planner prompt, an approval step and a decision log the agent for business planners
optchat + pack optchat plus a domain pack: a skill (Markdown) and helpers (Python) one pack per business domain

What the context is worth. We gave the bare agent the context in every comparison, so the tables measure optlens alone. Without the context, the bare agent lost one case on small models and one on the largest. It also cost 6-8 % more:

bare agent with the model context without it
27 small infeasible models: first fix right 26/27 25/27
cost per case $0.119 $0.129
8 hotel models of 279,000-837,000 constraints: first fix right 8/8 7/8
cost per case $0.174 $0.185

Both misses happened where meanings matter. One small anonymous model has no meanings for its constraints, and the agent proposed deleting a block of them. One 279,000-constraint hotel model has a demand forecast typed ten times too high. Without the context, the agent did not know which constraints are demand. It searched for the wrong value and ran out of turns. With the context, it found the value.

What we measured

400 cases, each with a solver-verified answer key:

case type cases the question
Infeasible, one injected error (a tightened limit, an added target, a flipped sense, a tightened bound) 97 "Why is it infeasible, and how do I fix it?"
Infeasible, two independent errors or a cascading conflict (from OR-Debug-Bench) 124 the same, when one fix is not enough
Infeasible as published (Netlib, MIPLIB) 45 diagnosis only
Solves to a wrong optimum (sign, coefficient, dropped constraint, big-M; documented values off by 20-25 %) 70 "This plan looks wrong. Why?"
What-if, sensitivity, goal and why-not questions on working models 32 "What if...?"
Planner conversations (5-6 turns, edits kept between turns, a three-scenario comparison) 7 a planner's working session
Scenario comparisons 5 "How do these plans differ?"
Hotel revenue-management models of 279,000 and 837,000 constraints, infeasible or wrong 16 the same questions, on models too large to read
Planner conversations on those hotel models 2 a working session on a model too large to read
Planner conversations on a 14-night hotel model: a messy week, and a rate plan that feeds the booking plan 2 a session across two linked models

285 cases use business models with a model document. These are Gurobi's modeling examples, synthetic ad-allocation and hotel revenue-management models, and OR-Debug-Bench's models. The other 115 use anonymous Netlib and MIPLIB models.

Scoring. The scorer applies every recommended fix to the model and re-solves it. It checks four things. Does the model solve? Is the change small and within policy, with no business rule deleted? Does it restore the correct optimum? Does it touch the actual error? What-if and conversation answers end in a JSON block, which the scorer checks against solver-computed keys.

The tools behind each kind of question, with examples, are on What you can ask.

Results

1. Correctness: level

First fix right, code against bare, on the same cases:

suite optlens bare
Infeasible, 27 mixed models (business and anonymous) 26/27 26/27
Infeasible, 14 business models, 2 runs 28/28 28/28
What-if, sensitivity and goal questions, 2 runs 64/64 64/64
Wrong optimum, reference objective given 19/19 19/19
Wrong optimum, no reference objective 19/19 18/19
Plausible wrong values (documented value at 80 % or 125 %) 12/12 12/12
Largest ad and yield models, wrong optimum, no reference 5/5 5/5
Two errors and cascading conflicts 40/40 40/40
Planner conversations, every turn right 7/7 7/7
Hotel models of 279,000 and 837,000 constraints 10/10 10/10
Hotel group linked by group-wide constraints, 279,000 and 837,000 constraints 6/6 6/6
Planner conversations on the linked hotel group, every turn right (9 turns) 2/2 2/2

The two are never more than one case apart. A frontier coding agent computes its own IIS, relaxes constraints and re-solves. On a model too large to read, it reads the file as text and solves a piece of it. If you evaluate LLM tools for optimization, compare them with a bare agent and a solver, not with a chat model.

These cases are not easy for a weaker model. The 27 mixed infeasible models in the first row above ran first on Claude Sonnet 5, then on Sonnet 5.5, with the same questions and scoring:

27 infeasible models, first fix right Sonnet 5: optlens (tools) Sonnet 5: bare Sonnet 5.5: optlens (tools) Sonnet 5.5: bare
right 24/27 21/27 25/27 26/27
left unanswered (30-turn limit or crash) 1 5 0 0
median turns 7 21 4 9
cost per case $0.100 $0.245 $0.064 $0.119

Sonnet 5 needed the tools. The bare agent ran out of turns on 5 cases and fixed 21. optlens answered 26 and fixed 24. On Sonnet 5.5 the agents answer every case. The bare agent's median turns fall from 21 to 9, its cost per case falls by half, and its correctness catches up. The Sonnet 5 runs also predate the model context, which is worth about one case (see above). So the newer model explains most of the jump, but not all of it.

2. Cost is a third to a half less, and the saving grows with model size

optlensbare (solver only)
optlens, small: $0.056 per case, 5.1 turns, 80/81 right$0.056bare (solver only), small: $0.085 per case, 6.2 turns, 79/81 right$0.085−35%smalloptlens, mid: $0.066 per case, 5.7 turns, 56/56 right$0.066bare (solver only), mid: $0.111 per case, 8.3 turns, 56/56 right$0.111−40%midoptlens, large: $0.050 per case, 3.7 turns, 13/13 right$0.050bare (solver only), large: $0.087 per case, 6.7 turns, 13/13 right$0.087−42%largeoptlens, too large to read: $0.069 per case, 6.1 turns, 16/16 right$0.069bare (solver only), too large to read: $0.133 per case, 10.0 turns, 16/16 right$0.133−48%too large to read
Mean cost per case and run, same cases; the percentage is optlens's saving. Hover a bar for turns and correctness.
model size right (optlens / bare) optlens: cost, turns bare: cost, turns optlens saves
small (constraints × variables < 10⁴) 80/81 / 79/81 $0.056, 5.1 $0.085, 6.2 35 %
mid (< 10⁶) 56/56 / 56/56 $0.066, 5.7 $0.111, 8.3 40 %
large (≥ 10⁶) 13/13 / 13/13 $0.050, 3.7 $0.087, 6.7 42 %
too large to read (50,000+ constraints) 16/16 / 16/16 $0.069, 6.1 $0.133, 10.0 48 %

The figures are means per case and run, on the same cases. The bare agent gets the answers, but it writes the diagnostic code again in a scratch script every session.

3. Time on models too large to read

These are hotel revenue-management models with injected errors. One size has 4 hotels × 365 nights (279,000 constraints, a 176 MB file). The other has 12 hotels × 365 nights (837,000 constraints, 530 MB). A second set links the hotels with group-wide constraints, so the model no longer splits into one hotel at a time.

optlensbare (solver only)
hotel models (10)optlens, hotel models (10): 23 min of tool time23 minbare (solver only), hotel models (10): 40 min of tool time40 min−43%linked hotel group (6)optlens, linked hotel group (6): 19 min of tool time19 minbare (solver only), linked hotel group (6): 64 min of tool time64 min−71%conversations, linked group (2)optlens, conversations, linked group (2): 22 min of tool time22 minbare (solver only), conversations, linked group (2): 46 min of tool time46 min−52%
Total tool time per suite (models of 279,000 and 837,000 constraints); the percentage is optlens's saving.
optlens: tool time, cost bare: tool time, cost right (optlens / bare)
hotel models (10 cases) 23 min, $0.62 40 min, $1.40 10/10 / 10/10
linked hotel group (6 cases) 19 min, $0.36 64 min, $0.86 6/6 / 6/6
planner conversations on the linked group (2, 9 turns) 22 min, $0.41 46 min, $0.60 9/9 / 9/9

optlens's time and cost come from a rerun on the release engine, one pass each. Every case and turn was right again.

optlens finds the conflict from the solver's own proof of infeasibility, so it never needs the whole model. It found 13 to 37 conflicting constraints out of 279,000-837,000, in 4 to 58 seconds.

4. Tools, code or plugin

14 business infeasible models and 32 what-if questions right cost per case
bare 46/46 $0.061
tools (one call per action) 45/46 $0.052
code (optlens in the agent's Python) 46/46 $0.043

Calling optlens from code is the cheapest setup, because one call runs several steps on one model load. Claude Code with the plugin answered 5/5 questions that compare three ad-allocation scenarios. The scenarios are three 9,680-variable MIPs opened side by side. The questions asked for status and objectives, a what-if on one scenario, and the smallest feasible budget of another. Each question cost $0.14. This was a small test; section 8 measures the plugin properly.

5. optchat and domain packs

Each cell shows the turns answered right and the mean cost per conversation:

conversation code optchat optchat + hotel pack
Everyday planner conversations (7) 40/40, $0.22 40/40, $0.28 no pack for these models
Linked hotel group, 279,000-837,000 constraints (2) 9/9, $0.20 9/9, $0.38 9/9, $0.41
Messy week on the hotel model (1) 5/5, $0.23 5/5, $0.41 15/15 over 3 runs, $0.36
Two-stage rate plan feeding the booking plan (1) 1/4, $0.39 2/4, $0.68 4/4, $0.19

On a single model, optchat answers as correctly as the code setup and costs a little more. Its planner prompt asks for checks, plain language and the planner's approval before a change becomes the plan. Where models are linked, the pack makes the difference. When the planner changes the rate plan, the pack's helper carries the new prices and demand into the booking plan exactly. It takes 1.5 seconds of tool time. Without the pack, both agents rebuilt or estimated the link, in 110 and 277 seconds, and got turns wrong.

A new business domain needs no code change to optlens or the agents. You add these:

you add format what it gives
the model and its document LP, MPS or Python; Markdown every question above (all the Gurobi example models ran this way)
a pack's skill Markdown the domain's levers and words in every answer
a pack's helpers (optional) Python exact steps across linked models, data checks against the document

6. Plan comparison on large MIPs

A MIP with many equally good plans can re-solve to any of them. So a plain diff mostly shows the solver's choice among ties. On the ad min-cost what-ifs, ties made up over 99 % of the reported changes. optlens compares against the optimal plan closest to the old one. That takes one extra solve, at most 1.5 s here:

ad min-cost what-if variables reported as changed
plain diff 755
optlens (closest optimal plan) 9

In a planner conversation, a planner said "campaign 3 now needs 48 impressions". optlens answered with the four slot changes the edit forces, which add up to the +32.4 cost. A plain diff would have shown a 779-variable reshuffle.

7. Checked fixes for suspicious values

optlens flags inputs that break their own pattern. Examples are a limit unlike its siblings', a flipped sense, an out-of-scale coefficient and a missing constraint. It solves each flag's undo, alone and together. On 221 cases with a known error, it gave a verified correct fix for 47 and a wrong one for 2.

8. The plugin in Claude Code

The numbers above come from our own minimal agent loop. A user who installs the plugin gets Claude Code instead, which brings its own context to every step. So we measured the plugin there too, with Claude Code's claude plugin eval: the same question asked in a fresh Claude Code session with the plugin and without it (Claude Code with Bash and Python, so it can write its own solver code). Three runs each. The questions are about the hotel pack's model: seven on a 14-night week (2,700 rows) and four on a hotel group over a year (279,000 rows), covering a mistyped input, a plan that solves but earns too much, a linking-row typo, rule costs, what-ifs and three scenarios side by side.

model questions with the plugin without time saved cost saved
Sonnet 5.5 14-night hotel week (2,700 rows), 7 questions 20/21 right, 36 s, $0.15 18/21 right, 59 s, $0.16 38% 3%
Sonnet 5.5 Hotel group over a year (279,000 rows), 4 questions 12/12 right, 91 s, $0.16 12/12 right, 332 s, $0.25 73% 38%
Sonnet 5.5 All 11 questions 32/33 right, 56 s, $0.16 30/33 right, 158 s, $0.19 65% 19%
Opus 5.5 14-night hotel week (2,700 rows), 7 questions 20/21 right, 47 s, $0.31 18/21 right, 60 s, $0.32 22% 5%
Opus 5.5 Hotel group over a year (279,000 rows), 4 questions 12/12 right, 112 s, $0.33 12/12 right, 171 s, $0.39 35% 15%
Opus 5.5 All 11 questions 32/33 right, 70 s, $0.32 30/33 right, 100 s, $0.35 30% 9%

Sessions right out of three runs per question; time and cost are means per session.

On the year model, Claude without the plugin parses the 121 MB file again in every script; the plugin keeps the model loaded. Without the plugin it missed most on questions that need a method (what a closure is worth, which rule costs most). On the small model the plugin's answers cost about the same. Opus 5.5 writes good solver code itself, so the plugin helps it less; Sonnet 5.5 with the plugin was as accurate as Opus with it, faster and half the cost.

The questions, graders and scripts are public in the optlens repository (plugin/evals): run claude plugin eval plugin there to check them yourself. One more question, on an anonymous Netlib model, is in the suite but left out above: its grader passed and failed the same answer from run to run.

What optlens adds when the agent is already right

Conflicts, flags, fixes and plan diffs come from tested code, not from a script the agent wrote in this session. optlens also re-solves every recommended fix before it offers it.

Limits

Try it

pip install "optlens[scip,mcp]"

Then connect it once to Claude Code, Codex, Cursor, VS Code or Claude Desktop. The README has the line for each.

The benchmark itself stays private, so that it keeps measuring new versions of optlens fairly. A script generates the tables above from its recorded runs. The Claude Code questions of section 8 are public, so you can rerun them. Problems with your own models are the most useful feedback we can get. Open a GitHub issue or write to hello@optlens.dev.

Credits

Business example models from Gurobi's modeling-examples (Apache-2.0). Two-error and cascading cases from OR-Debug-Bench / ORLoopBench (Ao, Simchi-Levi and Wang, arXiv 2601.21008; data CC BY 4.0), with constraint names changed and cases filtered. Netlib LP (infeasible set collected by J. Chinneck) and MIPLIB 2017 instances.

← Back to optlens