Somewhere in your inbox is a proposal that says an AI agent can rewrite your fifteen-year-old system in a fraction of the usual time. The demo looked convincing, the price is attractive, and the people who built the original system left years ago.
Treat the claim with care. Gartner predicts that more than 70% of mainframe exit projects started in 2026 will fail to deliver the intended benefits because the capabilities of generative AI tooling were overestimated. The tools are real, but so is the gap between a demo and a production cutover in legacy code modernization.
Below: what AI agents do well with old code, which tasks still belong to engineers, how to make a vendor show its review steps, and how to pilot on one module with numbers you can trust.
What an AI agent can do with old code today
An AI coding agent is a language model with tools. It can open files, search a codebase, run builds and tests, read the results and repeat until it reaches the goal you set. Point it at a module and it will summarize the code, trace how data moves through it, propose changes and check them against the tests that exist.
At scale this already works. Google reported in 2024 that more than a quarter of its new code is generated by AI, then reviewed and accepted by engineers.
Amazon says that upgrading tens of thousands of Java applications with Amazon Q Developer saved over 4,500 years of developer work. That is a vendor's claim about its own product, and it describes a narrow, repeatable job: moving code from one Java version to another.
That is the shape of the wins so far: a mechanical task, a well documented target, and existing tests to confirm the result. Rewriting pricing logic nobody has touched since 2011 is a different job.
Tasks worth handing to an agent
Documentation. Agents are good at reading code nobody understands and writing a readable account of it. When Meta pointed agents at a data pipeline of more than 4,100 files, about 5% of its modules had context AI tools could use; after an agent-driven mapping run, coverage reached 100%. The output still needs a human read, but the first draft of the map arrives in hours.
Characterization tests. Before you change old code you need tests that pin its current behavior. Michael Feathers defined characterization tests as checks of what the code actually and currently does, not what it is supposed to do. Agents can write them at a volume no team would attempt by hand, and each one is a safety net for later changes.
Simple translation. Version upgrades, framework migrations and the translation of self-contained modules with clear inputs and outputs. The Java upgrade above sits in this category.
Tasks that still need an experienced engineer
Business rules. The code says what happens; it rarely says why. A surcharge for one customer group, a cutoff time that dates from an old dispute, a status code that means different things in two departments: an agent will translate these faithfully or drop them silently, and it cannot tell you which. Someone who knows the business has to read the rules before they are rewritten.
Data. Old systems carry decades of decisions about numbers, dates and encodings. In COBOL, an arithmetic result that overflows its storage is truncated unless the ROUNDED phrase is specified, so a line-by-line translation into a language that rounds by default changes every total.
Migration failures look nothing like a demo. In TSB's 2018 platform migration, the data moved successfully, the new platform failed immediately, and the regulators later fined the bank £48.65 million after months of disruption for a large share of its 5.2 million customers.
Edge cases. Month-end batch jobs, the report only finance runs in March, the one integration that sends dates in a different format. The tests an agent relies on rarely cover these paths, so they pass until the day they run.
Hand to an agent | Keep with engineers |
|---|---|
Summaries and documentation of existing code | Recovering the business rules behind it |
Characterization tests that pin current behavior | Deciding which current behavior is a bug |
Version upgrades and translation of self-contained modules | Numeric, date and currency handling; data migration |
First-pass refactors, reviewed before merge | Batch jobs, integrations, rare paths and sign-off |
How to make a vendor show the review steps
Every vendor will say a person reviews the agent's output. Ask to see how. Four questions separate a process from a slogan.
Who reviews each change, and what do they check? You want a named senior engineer reading every change before it merges, and review comments from a recent project you can inspect.
How is equivalence proved? AWS Transform's Compare tool checks that a modernized application produces the same results as captured reference data, bit by bit, and Google's Dual Run can replay live events from the production mainframe and compare the outputs of the old and new systems. Then ask who chooses the cases, because reference data only covers what someone captured.
What did the last pilot measure? Ask for defect counts, review hours and time to merge, not a demo. METR's randomized trial found that experienced developers using AI tools took 19% longer to complete their issues while believing they had been sped up by 20%. Self-reported speed is not evidence.
What does the agent log? Which files it read, what it changed and who approved it, kept where the agent cannot edit it. Our governance checklist for AI agents covers that log in detail.
Risks: silent errors and code nobody understands
The failures that matter are quiet. In the 2025 Stack Overflow survey, 66% of developers named AI solutions that are almost right, but not quite, as their biggest frustration, and 46% said they distrust the accuracy of AI output. Veracode's tests across more than 100 models found that AI-written code introduced security vulnerabilities in 45% of cases. A translated module can compile, pass its tests and still misprice an invoice.
The second risk shows later. If an agent rewrites a system and your team only skims the result, you have replaced code nobody understands with new code nobody understands.
Thoughtworks has kept complacency with AI-generated code on hold in its Technology Radar, noting that agents now produce larger change sets that are harder to review. The 2025 DORA report found that AI adoption continues to have a negative relationship with software delivery stability. Speed without review turns into incidents.
A pilot plan: one small module, measured
Do not start with the core. Pick one module and treat it as an experiment.
Choose the module. Self-contained, with a business owner, an output someone checks, and changes that finish and verify within a week, the limit DORA sets for working in small batches.
Record the baseline. Test coverage, open defects, and how long a typical change takes today.
Documentation and tests first. Let the agent write the summary and the characterization tests. Have the business owner read the summary and mark what is wrong or missing.
Then the change. Translation or refactor, every change reviewed by a named engineer before it merges.
Prove equivalence. Run captured inputs through the old and new code and compare the outputs, including month-end and the awkward customer.
Measure and decide. Defects found in review, defects that escaped, review hours per change, time to merge, and whether your team can explain the new code without the agent. Decide on those numbers.
This is the same discipline as any AI pilot that proves its value, and it fits the strangler approach: replace one piece at a time while the old system keeps running.
Where this leaves legacy code modernization
AI agents have turned documentation, test writing and mechanical translation from months of effort into days. They have not touched the hard parts: knowing why the old code behaves as it does, proving the new code behaves the same, and owning the result when it does not.
A vendor who shows you review steps, equivalence tests and pilot numbers is worth talking to. One who shows you a demo is not there yet.
Our legacy system modernization team uses agents for the mechanical work and keeps senior engineers on the rules, the data and the review. Our AI agent development team builds the agents that belong in the finished system. Book a call to pick the first module.
