I've posted this a few weeks back without much HN traction - today we've added a free community tier, so anyone can try it out.
TL;DR: We’ve launched StableBuild, a new tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
As the CTO at an ML startup w/ 75 people (https://edgeimpulse.com/) I’ve grown incredibly frustrated with non-deterministic builds. Last year basically every week one of our containers (we have 40+ unique ones in prod) would stop working properly because some dependency was updated or removed. This ranges from Nvidia deleting cuda base images from Docker Hub, to Chromium being removed from the Ubuntu package registry in favor of the snap version, to pandas 2 being published with breaking APIs - while everyone just depends on e.g. pandas>=1.4.
This has been super disruptive because builds break for no apparent reason: someone pushes some unrelated code change, a container needs to be rebuilt, now it gets the latest dependencies => boom, either a compile error or an integration test fails. Many times this even blocks deployment. If the build system has decided that a container on master needs to be rebuilt, we can’t deploy the complete system if a dependency has shifted. And, fixing this naturally falls on the most senior engineers.
Anyway, to fix this I’ve funded (together w/ my Edge Impulse cofounder) StableBuild. It’s a set of mirrors and registries that let you easily freeze and pin Docker images, apt/apk packages, Python packages, and arbitrary files and URLs from the internet. It currently consists of:
* A custom pull-through cache for Docker Hub, that makes any image pulled immutable. Protects against updated or removed images; and as a nice byproduct also bypasses pull-rate limits in Docker Hub.
* Full daily copies of the Ubuntu, Debian and Alpine package registries + the most popular PPAs; so you can pin to a specific date (give me the package registry as it was on 2023-12-15). Essentially what snapshot.debian.org does, but fast and highly available (and for more repos).
* Full daily copy of the PyPI registry, so you can also pin to a specific date. This has been super useful for resurrecting old Python code. Any Python example w/ dependencies is bitrotted the moment it gets published - StableBuild’s historic registry helps tremendously (see https://docs.stablebuild.com/mirrors-and-caches/pypi-mirror#...)
* A generic file / URL cache for arbitrary things you need to pull from the internet during builds.
This has all been in production with SB’s first customers and has basically eliminated random build failures due to changed dependencies for them. Naturally you still want to upgrade dependencies (security patches are nice!) - but you can do it at their own pace, rather than whenever a container rebuilds.
StableBuild is now available for everyone. There's a free Community tier (since today) that gives free access to all services and mirrors (although with a hard 15GB/month traffic limit), and commercial pricing starting at $199 (cheaper than running a high-available apt mirror on AWS - which we used to do at Edge Impulse).
Would love to hear people's thoughts <3
Sign up: https://dashboard.stablebuild.com