Introducing the Kotlin Benchmark for AI coding agents (JetBrains)

What it claims

JetBrains' Kotlin Benchmark contains 105 repository-level tasks taken from active open-source Kotlin projects. An agent must turn an issue description into a patch. A task counts as resolved only if the patch passes the required tests in a containerized environment. The datasets and test harnesses are public on GitHub, and the suite is built on Multi-SWE-bench. JetBrains calls the scores "a signal, not a guarantee": results on your own codebase will depend on its architecture, internal APIs, coding standards, tooling and validation process.

Evidence

The headline results depend on the configuration used, and the summary omits it:

Points for reading the table:

• The gap is small. With 105 tasks, first and second place differ by about 4 tasks. Treat the top three as roughly tied rather than as a firm ranking.
• The rankings are for specific reasoning-effort settings. Running the same agent and model at default effort may give a different result.
• One ranked agent belongs to the publisher. Junie is JetBrains' own product, so you may want to verify results using the public harness.
• The top configurations solve most tasks. This suggests the current task set may not separate strong agents well. JetBrains' plan to add a "wider range of task difficulty levels" is consistent with that reading.

The full per-configuration comparison is at kotlinlang.org/benchmark.

How to use it

• Because the tasks and harnesses are public, you can rerun the suite against your own agent and model setup instead of relying on vendor numbers.
• Environment setup and data-collection details are on a separate Methodology page. The post itself does not reproduce them.
• The post gives no install commands and does not state a licence.

Limits

• Results are already dated. This is the first iteration, and the page says it "do[es] not yet include the most recent model releases." A second iteration is in progress, so the leaderboard will change.
• Pass/fail is the only metric. Cost, performance, maintainability and code quality are planned but not yet measured. Two agents with equal scores may therefore differ a lot in token spend or patch quality.
• Ecosystem coverage is narrow. Android and Kotlin Multiplatform tasks are planned, not included. Teams doing mainly Android or KMP work should not assume these scores transfer.
• Few setups are evaluated. Open-weight models and other commercial agents are listed as future additions, so open-weight options cannot be compared yet.
• Data sources may overlap with training data. The tasks come from public open-source repositories. The post does not discuss contamination.