Kyoto’s AI agents solve all 50 cases of TaxCalcBench v2 with 100% line-by-line accuracy. The best-performing model scores 60%. TaxCalcBench was created by Column Tax to evaluate frontier models on the tax calculation task.
We use the benchmark as a yardstick, but test our system under a harder setup. We don’t give our agents the complete taxpayer file. TaxCalcBench asks an LLM to calculate a return given all of the taxpayer’s information as context. We make our agents discover that information themselves by conducting a conversation with a simulated taxpayer.
We do this with a two-layer architecture that separates tax preparation from tax calculation: our information collector and preparer agent, “Prep”, and an in-house tax calculation engine, “Taxer”.
Background
TaxCalcBench is a benchmark created by Column Tax for evaluating frontier models’ ability to compute a tax return given a taxpayer’s complete information as context.
The first edition was published in July 2025 with 51 federal-only cases. All situational facts and documents (W-2s, 1099s) were supplied as a structured JSON file and given as input to the LLM to file a tax return. The benchmark was evaluated against 4 frontier LLMs of the time with scores ranging from 23% to 32%. In January 2026, one startup reported a score of 100% on that first edition.
In June 2026, Column Tax released the second edition. It introduced two major improvements: 50 cases covering federal taxes and 4 different states, and the addition of real PDFs — W-2s, 1099s, 1098s, prior-year returns. Situational facts were still provided as structured JSON but the cases were more complex. 17 models were evaluated in this edition. The best score was 60% achieved by GPT-5.6 Sol with web search enabled, versus 26% without web search.
Note that in both editions, the LLMs were given all of the taxpayer’s information as structured data. But that is not how a real preparer actually gets information from a taxpayer. The information is gathered through rigorous Q&A and by getting to understand their case. People do not know their situational facts verbatim.
TaxCalcBench was never designed to test information collection, and that is reasonable: its goal is to evaluate tax calculation. We wanted to also test the next layer of the problem, so we made it harder.
The Harder Test
Instead of giving our agent the entire case as context, we put a taxpayer simulator between the agent and the case to play the role of the taxpayer. Our information collector agent, “Prep”, discovers the information and fetches the required documents by asking the taxpayer simulator questions.
Prep interviews the taxpayer simulator as it would a real client. It asks for documents whenever required and asks about the filer’s tax-specific situations when necessary. Once it has everything it needs, it prepares the return.
With our tax calculation engine, “Taxer”, available to it, Prep can calculate the return whenever it needs to. It can even compare and present different scenarios to the taxpayer and work with them to plan for maximum savings.
What We Did Differently
AI excels at things that are verifiable, with clear right or wrong answers. This is why LLMs have become so good at code.
Coding with large language models works so well because of the compiler. A compiler tells the LLM whether a piece of code executes correctly, and if it doesn’t, it explains why through errors and stack traces. We applied the same analogy to another verifiable system — taxes — and built a compiler for taxes that can be used efficiently by an LLM.
Our “compiler for taxes”, a tax calculation engine, is called Taxer. This is not a novel idea. Almost a dozen tax engines have been built, with only a few in the last two decades. Building a tax calculation engine traditionally used to be a mammoth task, involving hundreds of engineers, years of work, and millions of dollars. This is why there are so few of them. We built an AI-native tax engine using coding agents and two human supervisors, translating thousands of pages of IRS tax rules into code in roughly 2 months.
Taxer reproduces the expected calculations of all 50 cases from TaxCalcBench and correctly calculates tax returns from our internal database. Taxer is designed to improve with changes in tax law. Coding agents write tests, implement updates to the engine when tax rules change, and validate their changes against those tests. A human reviews every update before it is published.
The second layer of our architecture is our information collector and preparer agent called “Prep”. We think of it as a tax expert guiding a taxpayer through their filing journey. Prep works from a checklist to ask for only the information required to prepare a return, inferring what it can from supporting documents and prior-year returns, and uses Taxer throughout the process to validate its understanding and calculate the return. Prep discovers. Taxer calculates. The information exchange between Taxer and Prep is what enables the system to score 100% on TaxCalcBench v2.
What’s next
Solving TaxCalcBench was a good challenge and is now a useful milestone, but it is not the end goal. We are building toward a financial planning assistant that considers tax liability at its core and works with taxpayers throughout the year, understands their situations, models different scenarios, and helps them make better tax decisions. We will also build more integrations so that required information flows into the system automatically with as little friction as possible for the taxpayer.
We will continue to push the system beyond TaxCalcBench, tackling increasingly complex tax situations and testing how far we can take this architecture.
We believe AI should do your taxes so you can work on more important and creative endeavors.
If this is interesting to you, talk to us. We would love to show you what we have built and share the research behind it.