Star ScarfBench on GitHub
Modernizing enterprise applications is one of the largest and most expensive software engineering activities an organization undertakes. Teams migrate applications between frameworks to improve maintainability, cloud readiness, developer productivity, and access to the latest features.
Recent advances in coding agents are fueling the rise of AI-powered modernization. But important questions remain.
Can AI agents reliably modernize real-world enterprise applications?
While existing software engineering benchmarks have shown impressive progress in bug fixing and code generation, framework migration presents fundamentally different challenges. Success requires not only transforming your code, but also preserving behavior, adapting your build system, and avoiding runtime dependencies.
To address this gap, we introduce ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java.
ScarfBench focuses on migrations across three major Java ecosystems:
Spring Jakarta EE Quokas
Unlike traditional benchmarks that compare generated code to a reference implementation, ScarfBench assesses whether migrated applications actually retain their build, deployment, and behavior.
Why migration is difficult
Framework migration is more than just replacing annotations.
A simple repository migration may require changes across dependency injection, persistence configuration, queries, and framework descriptors. Small mistakes in any of these parts can prevent a successful deployment.
Figure: Example of transition from spring to Jakarta
Framework migration requires not only the source code but also the translation of the framework’s semantics.
Introducing ScarfBench
ScarfBench provides a systematic way to evaluate AI agents on enterprise Java framework migration tasks.
The application requires the following:
The build was successful. Please expand it correctly. Passes operation verification.
This allows you to more realistically measure the quality of your modernization.
Benchmark overview
Metric Value Application 34 Framework Implementation 102 Migration Tasks 204 Lines of Code ~ 151,000 Source and Test Files ~ 2,000 Expert-Written Tests 1,331
ScarfBench includes both intensive migration tasks and full application migrations.
Diagram: ScarfBench construction pipeline
Starting with a JSR-based enterprise Java taxonomy, expert migration creates validated implementations across Spring, Jakarta EE, and Quarkus.
How does Frontier Agent work?
We evaluated several cutting-edge coding agents on ScarfBench.
Despite good performance on traditional software engineering benchmarks, framework migration remains difficult. Success rates vary widely between pairs of frameworks, and migration of entire applications remains particularly difficult.
Figure: Current leaderboard
Even today’s most powerful agents have operational success rates of less than 10%, demonstrating the gap between producing compilable code and maintaining application behavior.
Diagram: Compile → Deploy → Test progress
Compilation success consistently exceeds deployment success, which in turn exceeds operational success. Build success alone greatly overestimates migration quality.
Figure: Migration results by target framework
The difficulty of migration is highly dependent on the target framework, and Jakarta EE has proven to be particularly difficult.
What I learned about the AI agent for Java modernization
In addition to measuring success rates, ScarfBench helps you understand how your agents perform during modernization.
Can the agent know for sure when the migration is complete?
Migrated applications are only useful if you actually build and run them.
Therefore, we compared the results reported by the agent to independent build validation.
Findings: Agents are overconfident
Claude Code reported successful builds for 29 out of 30 applications overall.
Only 22 of these applications actually built successfully.
On the other hand, one application that was classified as failed by the agent eventually built successfully.
This suggests that an agent’s self-evaluation should not be treated as a reliable signal of transition completion.
Independent build and test validation remains essential.
How does the agent navigate application dependencies?
Framework migrations rarely affect a single file or layer.
Changes to configurations, services, databases, and web components often ripple throughout the application.
Findings: Migration is iterative, not linear
The most frequently accessed layers are:
Configuration Web Database Service
Common migrations include:
Configuration ↔ Web Service ↔ Database
This suggests that migration is an iterative dependency resolution process rather than a simple source-to-source conversion.
Where do agents spend the most effort?
We used layer revisit frequency as a proxy for migration effort. Layers that need to be accessed repeatedly typically include debugging, dependency resolution, or framework adaptation.
Findings: Configuration dominates migration efforts
Rather than proceeding linearly, the agent returned iteratively to configuration-related artifacts while resolving framework differences and dependency issues.
What are the challenges other than code conversion?
Not all migration problems are caused by source code.
Findings: Environment and tools matter
Agents frequently suffered from environmental problems such as:
Docker cache inconsistencies Port connectivity issues Maven wrapper and build tool issues
These operational concerns often delayed validation even when the source code migration itself was nearly complete.
Figure: Distribution of failure modes
Modernization failures span build systems, deployment environments, dependency injection, databases, endpoints, assertions, and infrastructure.
Important points
The biggest challenge in modernizing a framework is not converting Java code.
It manages a web of dependencies across configuration, infrastructure, and runtime environments.
Although Frontier Agents can automate a significant portion of the migration process, reliable validation and architectural reasoning remain critical to achieving a successful outcome.
ScarfBench helps illuminate these challenges and provides a standardized way to measure progress towards truly autonomous application modernization.
Explore the scarf bench
ScarfBench is designed as an open resource for researchers and practitioners.
Resources include:
Benchmark Datasets Evaluation Infrastructure Public Leaderboards Documentation Open Source Code
Researchers can compare agent architectures and techniques. Practitioners can use ScarfBench to evaluate modernization solutions before deploying them in production.
Website
https://scarfbench.info
dataset
https://huggingface.co/datasets/ibm-research/ScarfBench
space
https://huggingface.co/spaces/ibm-research/ScarfBench
GitHub repository
https://github.com/scarfbench/scarfbench
leader board
https://scarfbench.info/leaderboard
paper
https://arxiv.org/abs/2605.06754
Framework migration remains one of the biggest unsolved problems in AI-assisted software engineering. We hope ScarfBench will help the community measure its progress and accelerate the modernization of the next generation of AI-powered applications.
We invite researchers, practitioners, and the framework community to evaluate our agents, provide new migration scenarios, and contribute to advancing the state of the art.
