The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral
Pamela Fox runs a guided workshop comparing GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 on Microsoft Foundry, showing that benchmarks cannot predict how a model will behave in a specific application. The talk covers counting, RAG groundedness, vision, multilingual failures, tool calling, parallel tool calls in agents, string-check and LLM-as-a-judge evals, Microsoft's ASSERT tool for generating test sets, and DSPy prompt optimization that lets a cheap model satisfy a spec.
Pamela Fox opens by framing the session as a guided workshop, and the core argument is blunt: benchmarks tell you about a model in general, but only swapping it into your own application and testing it tells you how it behaves for your scenario. She stresses that model choice has to stay flexible, since new models appear constantly and access can be cut off at any time, so AI engineers need the habit of pointing an application at a different model and either having it still work or knowing how to adjust it. The four models compared are GPT-5.5 as the familiar baseline, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3, all deployed on Microsoft Foundry. Setup runs through a repo with a .env file that grants access to the deployments, with the caveat that token capacity is limited and rate-limit (429) errors may appear.
Where the models diverge on simple and grounded tasks
The first test is a deliberately basic letter-counting prompt (how many times does the letter E appear in “The elderly gentleman eagerly entered the elevator”). GPT-3.5 and Kimi both get the correct count of 13, DeepSeek V4 Flash returns 6, and Mistral returns 8. Fox then shows that prompt engineering can close the gap: asking the model to split the sentence into words, count per word, sum, and show its reasoning brings Mistral to the correct answer. The point is that the same model can fail or succeed depending on how the task is phrased, and that differences between models are visible within minutes of testing.
The RAG section moves to groundedness, which she calls the main thing she works on. The dataset covers only carpenter bees, and the question asks how honey bees and carpenter bees build nests. A well-behaved answer should cover carpenter bees and say it cannot answer about honey bees. GPT-5.5 and DeepSeek both decline to speculate, while Mistral draws on its weights and fills in honey bee details that were never in the sources. Prompt engineering improves things (asking each claim to be tied to a source) but the answer still leaks outside information. Her takeaway is that smaller or less capable models are more likely to present their own knowledge as if it came from the context, so willingness to admit what is not known should be an explicit evaluation criterion for any RAG system.
Vision and multilingual pitfalls
With image input, the test images include a mistaken unicorn (actually an aurochs), crocodiles photographed in Costa Rica, and a small-font plant price table where the cheapest entry costs 58 cents. GPT-5.5 handles all three correctly, and Kimi also succeeds. Mistral fails on the aurochs, calls the crocodiles alligators, and returns a $2.30 plant as the cheapest. Fox argues that prompt engineering has little leverage over visual understanding: the model either perceives the image correctly or it does not, and the real work is checking whether a model handles the specific kinds of images your application will receive. She also notes that DeepSeek V4 Flash is text-only and so was excluded. Switching the question into Spanish (caimanes versus cocodrilos) makes even strong models stumble, because the translation adds ambiguity on top of the visual distinction, which matters in bilingual settings such as California.
Tool calling, loops, and agents
Fox positions tool calling as the foundation of all agents: the model receives JSON-schema function definitions, decides whether to call one, and returns a function name plus arguments, while the application code is what actually executes the call. Her calendar-event test shows that the schema itself is prompt engineering, since names, types, and descriptions shape how the model fills in arguments. GPT-5.5 adds an unwanted attendee, DeepSeek includes the wrong attendee and writes “virtual (Microsoft Teams)” as the location, and the lesson is that ambiguous descriptions cause argument errors that should be fixed by clarifying the schema rather than blaming the model.
A calculator-in-a-loop test measures efficiency rather than just correctness. With the system prompt “do not do math in your head,” GPT-5.5 batches the whole shopping-cart calculation into one expression and needs a single tool call, DeepSeek takes four, and Kimi, though generally strong, sometimes loops unnecessarily to double-check itself. She argues that in loops the differentiator is often how many round trips a model needs to reach the goal, since each one is a network request.
The LangChain v1 travel planner extends this into a real agent with flight, hotel, budget, and activity tools. GPT-5.5 issues parallel tool calls on the first turn, searching flights and hotels together because the two are independent, then checks the budget and searches activities with the remainder. Kimi does not parallelize and takes more turns. DeepSeek is reasonably efficient at four calls but does not parallelize, and Mistral stops after the flight and hotel searches and skips the remaining steps, a failure mode she links to context filling up and instructions being forgotten. She mentions that the same model can behave differently across harnesses (agent frameworks), so comparisons should include both model and framework, and she notes that the three frameworks in the repo look increasingly similar.
Evaluation: from string checks to judges and generated test sets
For bulk evaluation she starts with plain string checks on the tool call name and arguments, covering date math (recommending giving the model a date tool), ambiguous durations, multilingual inputs, and slang. She stresses that eval suites must grow as real user inputs arrive in production so they do not drift from actual usage, and that evals are too slow and expensive for ordinary unit-test runs. In response to a question about frameworks, she mentions that observability platforms such as LangFuse, LangSmith, Arize, and Foundry-related SDKs include eval tooling, along with DeepEval and promptfoo, though string checks often need no framework at all.
LLM-as-a-judge handles fuzzier outputs. Each evaluation becomes two model calls, one to generate and one to judge, so judges must be run deliberately and calibrated against human labels. She advises pre-running the application’s answers so the judge can be changed independently of the app, and notes that the hard question is often whether the bug lies in the app or in the judge. Judges need pass/fail verdicts, a reason, the query, the response, and the context, and for agents they need the full trace of tool calls. Her results show most models scoring about two out of three, with Mistral lower on grounding. She points to Hamel Husain’s resources on evals and urges calibrating judges.
Microsoft Research’s new ASSERT tool addresses the difficulty of writing good judges. Given a YAML spec describing the agent, its users, and its dimensions (for example, budget pressure), ASSERT generates behavior categories, scenarios, and test sets that simulate users, and it can stratify those scenarios. Run across DeepSeek, Kimi, and GPT-5.5, GPT-5.5 has the fewest violations, which surprised her given Kimi’s strong showing elsewhere. She is candid that she has only used ASSERT for about a week. Her broader point is that teams often run only default metrics such as coherence and fluency, which LLMs rarely fail, when they need custom evaluators tailored to their own agent behavior.
DSPy, prompt optimization, and provider differences
The closing section addresses the cost of manually retuning prompts for each model. DSPy inverts the workflow: the developer declares the program’s spec and constraints in code, and an optimizer (GEPA, a genetic-style reflective method, which she loosely describes as a “Jeppa”-type approach) iteratively searches for a prompt that satisfies those constraints, using a student model and a reflection model. Her example asks for a numbered list where each line has an exact word count and the first letters spell a target word. The run took about six minutes, and the resulting system prompt was optimized for Mistral, the model that struggled most, so the cheapest model can meet the requirements. She cites Dropbox and Shopify as teams that used this approach to move to smaller models, and argues that treating the prompt as a compiled output of a spec offers more model independence.
She also notes that Anthropic uses a different Messages API and offers built-in citations: documents are supplied as structured sources, and the model returns text blocks linked to them, which is useful for frontend citation display. The repo includes a local llama server option, and the speaker closes by thanking attendees and mentioning she will rotate the exposed keys soon.