Debugging LP and MILP models with a coding agent: a 400-case benchmark
Infeasible, wrong and working models of up to 837,000 constraints. What a debugging engine adds to Claude, and how six setups compare.
TL;DR
- optlens is an open-source Python library that debugs LP and MILP models. It finds why a model is infeasible, the smallest fix, what a change costs and how two plans differ. An agent uses it through tools (MCP) or from its own Python code.
- A coding agent with a solver already gets the answers right. On Claude Sonnet 5.5 it scores 95-100 % in every suite, with or without optlens. The cases are not easy. On Sonnet 5 the bare agent left 5 of the first 27 unanswered.
- In our benchmark loop, optlens's edge is cost and speed. Called from the agent's own Python in a minimal agent loop, it gives the same correct answers at 35-48 % less cost. On models too large to read, it takes 43-71 % less time.
- In Claude Code, the plugin is right more often and faster. On 11 questions about a hotel model, three runs each, Sonnet 5.5 with the plugin answered 32 of 33 right against 30 without it, in 65 % less time (73 % on a 279,000-row model). Cost was about the same on the small model and 38 % lower on the large one. On Opus 5.5 the plugin saved 30 % of the time. Details.
- On linked models, a domain pack also wins on correctness. In one case a rate-plan change must flow into the booking plan. optchat with the hotel pack answered 4/4 turns right. The same agent without the pack got 2/4, and the coding agent with optlens got 1/4.
- Plan comparisons show what an edit forces. After one edit to an ad-allocation MIP, a plain diff reports 755 changed variables. Most of them are the solver switching between equally good plans. optlens reports the 9 that the edit forces.
- optlens, its MCP server and the Claude Code plugin are open source: github.com/jjd-lab/optlens. optchat, the planner agent, is available on request: hello@optlens.dev.
The six setups
All six run Claude Sonnet 5.5. All but the plugin run through our own minimal agent loop (a short system prompt and the tools), not Claude Code; section 8 measures the plugin in Claude Code itself. Every setup starts from the same first message. It holds the model's context and the question, and the agent can ask for the model's document. The setups differ only in what the agent can call.
The model context lists each constraint and variable family with its business meaning (in the benchmark, also the documented result). optlens generates it once from the model's document or code. An LLM writes the meanings, and optlens checks them against the model's actual families. optlens keeps the context as JSON you can review and commit. It shows the context the same way in every session, so every agent and every person describes the model in the same words. In Claude Code, the plugin creates the context the first time you open a model.
| setup | what the agent can call | what it is |
|---|---|---|
| Bare | Python with HiGHS, SCIP and gurobipy | a coding agent on its own: the baseline |
| Tools | the optlens tools, one call per action | the MCP server, in any MCP client |
| Code ("optlens" in the tables) | optlens preloaded in its Python, called from its own code | what the plugin's run_python offers |
| Claude Code + plugin | tools and run_python with the optlens skill (it also offered an analyst subagent, since removed: 2.5x the cost, no gain) |
what an OR user installs |
| optchat | the code setup, plus a planner prompt, an approval step and a decision log | the agent for business planners |
| optchat + pack | optchat plus a domain pack: a skill (Markdown) and helpers (Python) | one pack per business domain |
What the context is worth. We gave the bare agent the context in every comparison, so the tables measure optlens alone. Without the context, the bare agent lost one case on small models and one on the largest. It also cost 6-8 % more:
| bare agent | with the model context | without it |
|---|---|---|
| 27 small infeasible models: first fix right | 26/27 | 25/27 |
| cost per case | $0.119 | $0.129 |
| 8 hotel models of 279,000-837,000 constraints: first fix right | 8/8 | 7/8 |
| cost per case | $0.174 | $0.185 |
Both misses happened where meanings matter. One small anonymous model has no meanings for its constraints, and the agent proposed deleting a block of them. One 279,000-constraint hotel model has a demand forecast typed ten times too high. Without the context, the agent did not know which constraints are demand. It searched for the wrong value and ran out of turns. With the context, it found the value.
What we measured
400 cases, each with a solver-verified answer key:
| case type | cases | the question |
|---|---|---|
| Infeasible, one injected error (a tightened limit, an added target, a flipped sense, a tightened bound) | 97 | "Why is it infeasible, and how do I fix it?" |
| Infeasible, two independent errors or a cascading conflict (from OR-Debug-Bench) | 124 | the same, when one fix is not enough |
| Infeasible as published (Netlib, MIPLIB) | 45 | diagnosis only |
| Solves to a wrong optimum (sign, coefficient, dropped constraint, big-M; documented values off by 20-25 %) | 70 | "This plan looks wrong. Why?" |
| What-if, sensitivity, goal and why-not questions on working models | 32 | "What if...?" |
| Planner conversations (5-6 turns, edits kept between turns, a three-scenario comparison) | 7 | a planner's working session |
| Scenario comparisons | 5 | "How do these plans differ?" |
| Hotel revenue-management models of 279,000 and 837,000 constraints, infeasible or wrong | 16 | the same questions, on models too large to read |
| Planner conversations on those hotel models | 2 | a working session on a model too large to read |
| Planner conversations on a 14-night hotel model: a messy week, and a rate plan that feeds the booking plan | 2 | a session across two linked models |
285 cases use business models with a model document. These are Gurobi's modeling examples, synthetic ad-allocation and hotel revenue-management models, and OR-Debug-Bench's models. The other 115 use anonymous Netlib and MIPLIB models.
Scoring. The scorer applies every recommended fix to the model and re-solves it. It checks four things. Does the model solve? Is the change small and within policy, with no business rule deleted? Does it restore the correct optimum? Does it touch the actual error? What-if and conversation answers end in a JSON block, which the scorer checks against solver-computed keys.
The tools behind each kind of question, with examples, are on What you can ask.
Results
1. Correctness: level
First fix right, code against bare, on the same cases:
| suite | optlens | bare |
|---|---|---|
| Infeasible, 27 mixed models (business and anonymous) | 26/27 | 26/27 |
| Infeasible, 14 business models, 2 runs | 28/28 | 28/28 |
| What-if, sensitivity and goal questions, 2 runs | 64/64 | 64/64 |
| Wrong optimum, reference objective given | 19/19 | 19/19 |
| Wrong optimum, no reference objective | 19/19 | 18/19 |
| Plausible wrong values (documented value at 80 % or 125 %) | 12/12 | 12/12 |
| Largest ad and yield models, wrong optimum, no reference | 5/5 | 5/5 |
| Two errors and cascading conflicts | 40/40 | 40/40 |
| Planner conversations, every turn right | 7/7 | 7/7 |
| Hotel models of 279,000 and 837,000 constraints | 10/10 | 10/10 |
| Hotel group linked by group-wide constraints, 279,000 and 837,000 constraints | 6/6 | 6/6 |
| Planner conversations on the linked hotel group, every turn right (9 turns) | 2/2 | 2/2 |
The two are never more than one case apart. A frontier coding agent computes its own IIS, relaxes constraints and re-solves. On a model too large to read, it reads the file as text and solves a piece of it. If you evaluate LLM tools for optimization, compare them with a bare agent and a solver, not with a chat model.
These cases are not easy for a weaker model. The 27 mixed infeasible models in the first row above ran first on Claude Sonnet 5, then on Sonnet 5.5, with the same questions and scoring:
| 27 infeasible models, first fix right | Sonnet 5: optlens (tools) | Sonnet 5: bare | Sonnet 5.5: optlens (tools) | Sonnet 5.5: bare |
|---|---|---|---|---|
| right | 24/27 | 21/27 | 25/27 | 26/27 |
| left unanswered (30-turn limit or crash) | 1 | 5 | 0 | 0 |
| median turns | 7 | 21 | 4 | 9 |
| cost per case | $0.100 | $0.245 | $0.064 | $0.119 |
Sonnet 5 needed the tools. The bare agent ran out of turns on 5 cases and fixed 21. optlens answered 26 and fixed 24. On Sonnet 5.5 the agents answer every case. The bare agent's median turns fall from 21 to 9, its cost per case falls by half, and its correctness catches up. The Sonnet 5 runs also predate the model context, which is worth about one case (see above). So the newer model explains most of the jump, but not all of it.
2. Cost is a third to a half less, and the saving grows with model size
| model size | right (optlens / bare) | optlens: cost, turns | bare: cost, turns | optlens saves |
|---|---|---|---|---|
| small (constraints × variables < 10⁴) | 80/81 / 79/81 | $0.056, 5.1 | $0.085, 6.2 | 35 % |
| mid (< 10⁶) | 56/56 / 56/56 | $0.066, 5.7 | $0.111, 8.3 | 40 % |
| large (≥ 10⁶) | 13/13 / 13/13 | $0.050, 3.7 | $0.087, 6.7 | 42 % |
| too large to read (50,000+ constraints) | 16/16 / 16/16 | $0.069, 6.1 | $0.133, 10.0 | 48 % |
The figures are means per case and run, on the same cases. The bare agent gets the answers, but it writes the diagnostic code again in a scratch script every session.
3. Time on models too large to read
These are hotel revenue-management models with injected errors. One size has 4 hotels × 365 nights (279,000 constraints, a 176 MB file). The other has 12 hotels × 365 nights (837,000 constraints, 530 MB). A second set links the hotels with group-wide constraints, so the model no longer splits into one hotel at a time.
| optlens: tool time, cost | bare: tool time, cost | right (optlens / bare) | |
|---|---|---|---|
| hotel models (10 cases) | 23 min, $0.62 | 40 min, $1.40 | 10/10 / 10/10 |
| linked hotel group (6 cases) | 19 min, $0.36 | 64 min, $0.86 | 6/6 / 6/6 |
| planner conversations on the linked group (2, 9 turns) | 22 min, $0.41 | 46 min, $0.60 | 9/9 / 9/9 |
optlens's time and cost come from a rerun on the release engine, one pass each. Every case and turn was right again.
optlens finds the conflict from the solver's own proof of infeasibility, so it never needs the whole model. It found 13 to 37 conflicting constraints out of 279,000-837,000, in 4 to 58 seconds.
4. Tools, code or plugin
| 14 business infeasible models and 32 what-if questions | right | cost per case |
|---|---|---|
| bare | 46/46 | $0.061 |
| tools (one call per action) | 45/46 | $0.052 |
| code (optlens in the agent's Python) | 46/46 | $0.043 |
Calling optlens from code is the cheapest setup, because one call runs several steps on one model load. Claude Code with the plugin answered 5/5 questions that compare three ad-allocation scenarios. The scenarios are three 9,680-variable MIPs opened side by side. The questions asked for status and objectives, a what-if on one scenario, and the smallest feasible budget of another. Each question cost $0.14. This was a small test; section 8 measures the plugin properly.
5. optchat and domain packs
Each cell shows the turns answered right and the mean cost per conversation:
| conversation | code | optchat | optchat + hotel pack |
|---|---|---|---|
| Everyday planner conversations (7) | 40/40, $0.22 | 40/40, $0.28 | no pack for these models |
| Linked hotel group, 279,000-837,000 constraints (2) | 9/9, $0.20 | 9/9, $0.38 | 9/9, $0.41 |
| Messy week on the hotel model (1) | 5/5, $0.23 | 5/5, $0.41 | 15/15 over 3 runs, $0.36 |
| Two-stage rate plan feeding the booking plan (1) | 1/4, $0.39 | 2/4, $0.68 | 4/4, $0.19 |
On a single model, optchat answers as correctly as the code setup and costs a little more. Its planner prompt asks for checks, plain language and the planner's approval before a change becomes the plan. Where models are linked, the pack makes the difference. When the planner changes the rate plan, the pack's helper carries the new prices and demand into the booking plan exactly. It takes 1.5 seconds of tool time. Without the pack, both agents rebuilt or estimated the link, in 110 and 277 seconds, and got turns wrong.
A new business domain needs no code change to optlens or the agents. You add these:
| you add | format | what it gives |
|---|---|---|
| the model and its document | LP, MPS or Python; Markdown | every question above (all the Gurobi example models ran this way) |
| a pack's skill | Markdown | the domain's levers and words in every answer |
| a pack's helpers (optional) | Python | exact steps across linked models, data checks against the document |
6. Plan comparison on large MIPs
A MIP with many equally good plans can re-solve to any of them. So a plain diff mostly shows the solver's choice among ties. On the ad min-cost what-ifs, ties made up over 99 % of the reported changes. optlens compares against the optimal plan closest to the old one. That takes one extra solve, at most 1.5 s here:
| ad min-cost what-if | variables reported as changed |
|---|---|
| plain diff | 755 |
| optlens (closest optimal plan) | 9 |
In a planner conversation, a planner said "campaign 3 now needs 48 impressions". optlens answered with the four slot changes the edit forces, which add up to the +32.4 cost. A plain diff would have shown a 779-variable reshuffle.
7. Checked fixes for suspicious values
optlens flags inputs that break their own pattern. Examples are a limit unlike its siblings', a flipped sense, an out-of-scale coefficient and a missing constraint. It solves each flag's undo, alone and together. On 221 cases with a known error, it gave a verified correct fix for 47 and a wrong one for 2.
8. The plugin in Claude Code
The numbers above come from our own minimal agent loop. A user who installs the plugin gets Claude Code instead, which
brings its own context to every step. So we measured the plugin there too, with Claude Code's claude plugin eval:
the same question asked in a fresh Claude Code session with the plugin and without it (Claude Code with Bash and
Python, so it can write its own solver code). Three runs each. The questions are about the hotel pack's model: seven
on a 14-night week (2,700 rows) and four on a hotel group over a year (279,000 rows), covering a mistyped input, a plan
that solves but earns too much, a linking-row typo, rule costs, what-ifs and three scenarios side by side.
| model | questions | with the plugin | without | time saved | cost saved |
|---|---|---|---|---|---|
| Sonnet 5.5 | 14-night hotel week (2,700 rows), 7 questions | 20/21 right, 36 s, $0.15 | 18/21 right, 59 s, $0.16 | 38% | 3% |
| Sonnet 5.5 | Hotel group over a year (279,000 rows), 4 questions | 12/12 right, 91 s, $0.16 | 12/12 right, 332 s, $0.25 | 73% | 38% |
| Sonnet 5.5 | All 11 questions | 32/33 right, 56 s, $0.16 | 30/33 right, 158 s, $0.19 | 65% | 19% |
| Opus 5.5 | 14-night hotel week (2,700 rows), 7 questions | 20/21 right, 47 s, $0.31 | 18/21 right, 60 s, $0.32 | 22% | 5% |
| Opus 5.5 | Hotel group over a year (279,000 rows), 4 questions | 12/12 right, 112 s, $0.33 | 12/12 right, 171 s, $0.39 | 35% | 15% |
| Opus 5.5 | All 11 questions | 32/33 right, 70 s, $0.32 | 30/33 right, 100 s, $0.35 | 30% | 9% |
Sessions right out of three runs per question; time and cost are means per session.
On the year model, Claude without the plugin parses the 121 MB file again in every script; the plugin keeps the model loaded. Without the plugin it missed most on questions that need a method (what a closure is worth, which rule costs most). On the small model the plugin's answers cost about the same. Opus 5.5 writes good solver code itself, so the plugin helps it less; Sonnet 5.5 with the plugin was as accurate as Opus with it, faster and half the cost.
The questions, graders and scripts are public in the optlens repository (plugin/evals): run
claude plugin eval plugin there to check them yourself. One more question, on an anonymous Netlib model, is in the
suite but left out above: its grader passed and failed the same answer from run to run.
What optlens adds when the agent is already right
Conflicts, flags, fixes and plan diffs come from tested code, not from a script the agent wrote in this session. optlens also re-solves every recommended fix before it offers it.
Limits
- Sections 1-7 ran Claude Sonnet 5.5. Sonnet 5 ran only on the first 27 cases, as shown above.
- Each setting had one or two runs. The messy week with the pack had three.
- Every correctness suite is saturated, so these cases cannot show further gains in correctness.
- The largest models are one synthetic family, hotel revenue management. They give 16 cases and 2 conversations.
- The plugin row in section 4 is 5 questions. The optchat rows are one run each, and the two-stage rate plan is one conversation.
- The Claude Code results (section 8) are 11 questions on one model family, three runs per setup, on Sonnet 5.5 and Opus 5.5.
- The benchmark ran Gurobi only on its size-limited license.
Try it
pip install "optlens[scip,mcp]"
Then connect it once to Claude Code, Codex, Cursor, VS Code or Claude Desktop. The README has the line for each.
The benchmark itself stays private, so that it keeps measuring new versions of optlens fairly. A script generates the tables above from its recorded runs. The Claude Code questions of section 8 are public, so you can rerun them. Problems with your own models are the most useful feedback we can get. Open a GitHub issue or write to hello@optlens.dev.
Credits
Business example models from Gurobi's modeling-examples (Apache-2.0). Two-error and cascading cases from OR-Debug-Bench / ORLoopBench (Ao, Simchi-Levi and Wang, arXiv 2601.21008; data CC BY 4.0), with constraint names changed and cases filtered. Netlib LP (infeasible set collected by J. Chinneck) and MIPLIB 2017 instances.