Automated Performance Improvement With Large Language Models

Funding Agency: Wallenberg Autonomous Systems, AI, and Software Program (WASP)

Call: Academic Doctoral Student Projects 2024

Principle Investigator: Philipp Leitner

Involved Lab Members: Philipp Leitner, Gregory Gay, Lirong Yi, Larissa Salerno

Publications: (Yi et al., 2026)

Large Language Models (LLMs) can generate code, but can they generate fast code for complex, real-world software systems? In this project, we investigate this question and propose agentic approaches that can diagnose and patch complex performance bottlenecks.

Unlike prior studies, which focused on algorithmic puzzles, we have conducted experiments on actual performance-sensitive production code and employed developer-written JMH benchmarks to rigorously validate performance gains against human baselines. Our initial results reveal a nuanced reality - although LLMs demonstrate a surprisingly high capability to solve these complex engineering problems, their solutions suffer from extreme volatility and still lag behind human developers on average. Consequently, we find that the current benchmarks based on algorithmic tasks yields an overly optimistic assessment of LLM capabilities. We trace this real-world performance gap to two primary limitations: first, LLMs struggle to autonomously pinpoint performance hotspots, and second, even with explicit guidance, they often fall short of synthesizing optimal algorithmic improvements. Our results highlight the need to move beyond static code generation towards more complex agent-based systems that are able to profile and observe runtime behavior for performance improvement. In our current work, we are exploring solutions to these shortcomings.