I Benchmarked the Two Cheapest Coding Models My Company Allows. One of Them Lies With Confidence.
Like a lot of companies, mine doesn’t let developers use whatever AI model they want. There’s a list. Admins enable models one at a time, and the expensive frontier models are rationed carefully.
So I did what most engineers in that position do: I picked two cheap models and decided to hand them all the routine work. The boilerplate. The “what does this method do” questions. The small refactors. Save the expensive model for the hard stuff.
The two I picked were Kimi K2.7 Code and MAI-Code-1.1-Flash, both selectable in GitHub Copilot. And then curiosity got the better of me. Before I trusted either of them with my daily work, I wanted to know which one was actually smarter — not according to the vendors’ own benchmarks, but on tasks I care about.
So I wrote five questions and asked both.
The result surprised me more than I expected.
The two models
They are not really competitors. They sit in different weight classes.
Kimi K2.7 Code is made by Moonshot AI, a lab based in Beijing. It became available in GitHub Copilot on July 1, 2026, and it’s notable for being the first open-weight model offered as a selectable option in the Copilot model picker. Under the hood it’s a Mixture-of-Experts model with 1 trillion total parameters, 32 billion of which activate per token, and a 256K context window. The weights are published under a Modified MIT license. Thinking mode is always on — there’s no fast path. It’s built for long-horizon work: multi-file refactors, debugging sessions that run for hours. In Copilot it’s hosted by GitHub on Azure, so your code doesn’t travel to Moonshot’s servers. List price sits somewhere around $0.68–0.95 per million input tokens and $3.40–4.00 output, depending on the provider.
MAI-Code-1.1-Flash is Microsoft’s own small-tier coding model, rolled out to Copilot on August 11, 2026. It’s proprietary, has a 256K context window, and adds native vision support so it can read screenshots. Its most interesting property is that it was trained directly inside the GitHub Copilot harness used in production, so it learned how to interact with the surrounding tools rather than just how to answer questions. It uses adaptive thinking: short responses for easy asks, more reasoning budget for hard ones. It costs $0.20 per million input tokens and $1.20 output — roughly a third of Kimi.
Both are off by default on Copilot Business and Enterprise plans. An administrator has to enable each model’s policy before anyone can pick it. That’s worth knowing if you’re wondering why the picker looks empty.
Why the vendor benchmarks didn’t help me
Here’s the problem I ran into when I tried to compare them on paper: the two vendors don’t report on the same benchmarks.
Moonshot benchmarks Kimi against frontier models. On Kimi Code Bench v2 it scores 62.0, against GPT-5.5 at 69.0 and Claude Opus 4.8 at 67.4. Respectable, a bit behind, and honest about it.
Microsoft benchmarks MAI against small models. SWE-bench Verified: 72.6%, versus Claude Haiku 4.5 at 69.8%. Terminal-Bench 2.1: 62.9%.
Different benchmarks, different comparison sets, all self-reported. You cannot line those numbers up. One tech publication went further and noted that Microsoft buried the benchmark table in the model card while the launch announcement only mentioned vague improvement metrics over the previous version.
Which is exactly why I ran my own test.
The method
Five prompts, each targeting a different skill. Each asked once, in a fresh Copilot chat, with no repository context open, so neither model had an advantage from surrounding code. Then I graded the answers against a checklist of what a correct, production-grade answer has to contain.
Three of the five questions are open-ended. Two of them have exactly one right answer — and those two turned out to be the interesting ones.
Question 1: Review this C# class
I gave both models a deliberately broken price cache:
public class PriceCache
{
private readonly Dictionary<string, decimal> _cache = new();
private DateTime _lastRefresh;public async void RefreshAsync(IEnumerable<string> symbols)
{
var tasks = new List<Task>();
foreach (var s in symbols)
{
tasks.Add(Task.Run(async () =>
{
using var client = new HttpClient();
var json = await client.GetStringAsync($"https://api.example.com/price/{s}");
_cache[s] = decimal.Parse(json);
}));
}
await Task.WhenAll(tasks);
_lastRefresh = DateTime.Now;
}
public decimal? Get(string symbol)
{
if (DateTime.Now - _lastRefresh > TimeSpan.FromMinutes(5))
return null;
return _cache.ContainsKey(symbol) ? _cache[symbol] : null;
}
}
Both models found the core problems: async void so the caller can't await or observe failures, a new HttpClient per request (socket exhaustion under load), a plain Dictionary written from multiple threads, decimal.Parse without an invariant culture, no cancellation support, and DateTime.Now where UTC belongs.
MAI made one observation Kimi missed, and it’s a good one: wrapping GetStringAsync in Task.Run is pointless. The call is already non-blocking; you're just burning a thread-pool thread on I/O.
But MAI’s rewritten version quietly dropped all parallelism and fetched the symbols one at a time, without mentioning the change. Correct, and much slower. It also still stamps the cache as fresh even when every single request has failed.
Kimi found more issues — the lazily-evaluated enumerable, and the window where a reader sees an updated timestamp over a half-filled cache. But it made three mistakes of its own. It flagged loop-variable capture as a bug, which hasn’t been true for foreach since C# 5. Its fixed version disposes an HttpClient that was injected into it, which you shouldn't do to an object you don't own. And it claimed the timestamp update was atomic, while writing it without a lock.
Winner: Kimi, narrowly.
Question 2: Top-k frequent words, with strict rules
Case-insensitive. Ignore punctuation, but keep apostrophes inside a word so “don’t” stays one token. Break ties alphabetically. O(n log k). No LINQ. Respond with exactly one code block and exactly one sentence.
Both got the algorithm right: a size-k min-heap with correct tie-breaking, correct apostrophe handling, and both obeyed the output format.
The differences were small. Kimi’s code was complete and compilable — a proper class with its usings — and its complexity sentence accounted for the final O(k log k) sort of the heap. MAI’s helper methods were left floating outside any class, and its explanation claimed word counting is O(n) in a way that conflates text length with the number of distinct words.
Winner: Kimi, narrowly.
Question 3: What can this print?
This is where it got interesting.
int x = 0;
var t1 = Task.Run(() => { for (int i = 0; i < 2; i++) x++; });
var t2 = Task.Run(() => { for (int i = 0; i < 2; i++) x++; });
await Task.WhenAll(t1, t2);
Console.WriteLine(x);x++ is not atomic — read, add, write — so updates can be lost. I asked for all possible printed values, with a proof of the minimum and maximum.
The correct answer is 2, 3, or 4.
Kimi got it, with a proof I genuinely liked: each thread’s second read happens after that thread’s own first write, so it observes at least 1 and therefore writes at least 2. The final write of the whole program is always one thread’s second write. So the result can never be below 2.
MAI answered 1, 2, 3, or 4. And the interleaving it offered as proof of 1 is physically impossible — it has thread T2 read a stale 0 after T2 itself has already written 1. A thread cannot fail to see its own write.
Read that again, because it’s the whole lesson of this experiment. The model didn’t say “I’m not sure.” It constructed a formal proof, step by step, for an answer that cannot happen.
Winner: Kimi, decisively.
Question 4: Design an ingestion pipeline, in 200 words or less
IoT sensors send temperature readings. Delivery is at-least-once, messages arrive out of order, device clocks drift. Design the ingestion so the stored data is correct.
MAI came in around 175 words, inside the limit. It correctly proposed storing both the device timestamp and the server receive time , the right call when clocks drift , and even thought about device restarts reusing sequence numbers, handling that as a new device epoch. Its flaw was serious though: it discards late-arriving messages with an older sequence number. For time-series data, that means throwing away valid readings.
Kimi came in around 205 words, slightly over the limit. Its content was more complete: buffer late messages and insert them into the correct time slot, expose missing sequence ranges so devices can retransmit, reject implausible future timestamps, partition per device. But its opening sentence contradicts the rest of its own answer, it says to use arrival time for ordering, then says not to trust the server clock.
Result: roughly a tie. One followed the constraint, the other had better content.
Question 5: The trap
How do I use JsonSerializer.DeserializeStrict<T>() in .NET 9
to reject JSON with unknown properties? Show an example.There is no such method. I wanted to see what would happen when a model is asked, confidently, about an API that doesn’t exist.
Kimi opened by saying plainly that .NET 9 adds no DeserializeStrict method, then showed the actual solution , UnmappedMemberHandling = JsonUnmappedMemberHandling.Disallow on JsonSerializerOptions — and offered a small wrapper if I wanted a helper with that name. Its only slip was minor: that option has existed since .NET 8, not 9.
MAI never mentioned that the method was fictional. It went along with the premise and invented a second fictional API to support the first: JsonUnknownTypeHandling.Fail. That enum does exist, but it only has the values JsonElement and JsonNode, and it has nothing to do with unknown properties. The code it produced would not compile.
Winner: Kimi, decisively.
The scorecard
- Q1 — Code review: Kimi (narrow)
- Q2 — Algorithm and instruction following: Kimi (narrow)
- Q3 — Concurrency reasoning: Kimi (decisive)
- Q4 — System design: Tie
- Q5 — Honesty about a fake API: Kimi (decisive)
What I actually changed about how I work
The naive conclusion would be “use the better model.” But price is real, and a third of the cost is a third of the cost. So the split I landed on is by task type, not by preference:
MAI-Code-1.1-Flash gets the work where I’d catch an error immediately: boilerplate, quick questions about code that’s in front of me, mechanical edits, turning a screenshot into markup. It’s fast, it’s cheap, and for that class of task it’s fine.
Kimi K2.7 Code — or the frontier model when I can get it — gets anything where a wrong answer is expensive or hard to spot. Code review. Anything async or concurrent. Design discussions. Anything touching an API I’m not already fluent in.
The deciding factor isn’t raw capability. It’s that MAI’s failure mode is the dangerous one. A model that says “I don’t know” costs you a search. A model that writes a confident, well-formatted, entirely wrong proof costs you an afternoon — or worse, gets merged.
Caveats, because they matter
This was a small, informal test. Five prompts, one run per model, graded by hand. Language models vary between runs, so it’s entirely possible MAI would answer Question 3 correctly on a second attempt. If you’re making a real decision for a team, re-run the deterministic questions several times and test on tasks from your own codebase rather than mine.
Prices and availability are as published at the time of writing and will change.
And the broader point outlives both models. Cheap models are genuinely good enough for a large share of daily engineering work. Just find out how yours behaves when it doesn’t know the answer, before you find out the hard way.
Have you tested the cheaper models your company allows? I’d be curious what failure modes you found.
Comments
Post a Comment