We are open sourcing (https://github.com/xeol-io/bumpgen/), a tool that uses an LLM to upgrade your dependencies to their newer versions. We are building this for our customers (security teams) who need to upgrade their dependencies for security and compliance but lack engineering resources. Most patching tend to be painless but major version upgrades will cause breaking changes that require engineers to fix. We want bumpgen to reduce this engineering cost to security.

It has been an interesting experience using AI to identify breaking changes, fix them, then propagate the fixes across a codebase. We thought for sure the biggest challenge would be overcoming a LLM’s coding shortcomings but turns out navigating the codebase correctly was the much bigger issue.

When bumpgen fixes a breaking change, the fix itself needs to be applied to the rest of the codebase. This requires bumpgen to understand how every function in a codebase interacts with each other and how a change to one would affect the others. To address this, we drew inspiration from CodePlan (https://arxiv.org/abs/2309.12499) that describes a theoretical(?) way to apply change consistently for an entire repository. In short, it combines a dependency graph and an oracle to understand the codebase then verify a change.

If I were to summarize my learnings, it would be that applying codegen to this problem is much more a “refactoring” problem than it is a “coding” problem (if that makes sense). So to answer my own question, I’d say AI CAN keep your code up-to-date but ONLY if it is guided carefully through the codebase.

We benchmarked bumpgen against our test suite (https://github.com/xeol-io/swe-bump-bench) and it currently stands at around 50% accuracy. The bench suite is a set of repos with human commits for version bumps. We run bumpgen on the prior commit then compare bumpgen’s code diffs to that of the human commit along with passing builds to determine success.

There are a lot of improvements we want to test out in May to increase accuracy. Starting with improving and tightening our core CodePlan logic. We think getting this right will unlock automating far more complex breaking changes and codebases like upgrading a vue 2 to vue 3.

We have been building bumpgen for a month and we would love to hear people’s thoughts, experiences, and suggestions!