The latest AI coding models all perform well on standard tests. Improvements between releases are getting smaller, suggesting we’re approaching a point where they’re all pretty similar.
But real-world coding isn’t a standardized test. It’s about working within a specific codebase, with its own unique conventions and quirks. That’s where these models start to show real differences.
Many developers have strong opinions on which model works best for them. Some swear by one model, while others find it frustrating. This often comes down to personal preferences and how well the model adapts to a specific coding style.
The Best Model Is the One That Works for You
The idea of a universally “best” coding model might be a myth. The optimal model depends on individual needs and the specific project. That’s why finding the right model is so important.
To investigate this, a dataset of real engineering tasks was created. These were typical coding tickets like “optimize database connections.” A set of criteria was developed to assess model performance, focusing on common developer pain points:
- Pattern Adherence: Does the model follow existing code architecture?
- Scope Discipline: Does it stay focused on the task or make unnecessary changes?
- Comment Quality: Does it write helpful documentation, not just repeat the code in words?
Fourteen leading large language models (LLMs) were evaluated using these metrics. Each response received a score from -1.0 to 1.0.
The Results: A Tiered Approach
The results showed a clear winner: o3 medium scored highest (0.53 ± 0.05), closely followed by o4-mini (0.48 ± 0.05). Surprisingly, a popular model, Sonnet 4, placed only 7th (0.41 ± 0.07).
These findings led to a tiered approach to using LLMs for coding:
- o3: Complex, important tasks where quality is paramount.
- o3-mini: Demanding tasks at scale, offering similar quality but faster and cheaper.
- Gemini 2.5 Flash: Documentation, excelling in comment quality at a low cost.
The study also revealed some interesting insights:
- Scope discipline isn’t strongly linked to other coding skills.
- All models struggle with writing good comments.
- More “thinking” doesn’t always mean better code.
It’s important to remember these results reflect a specific codebase and team priorities. The best model for you will depend on your own coding style, project needs, and what you value most in an AI assistant.
If you’re looking for the right coding LLM, consider evaluating different models on your own tasks and with your own criteria.
