At the start of the pandemic I was migrating our platform to Kubernetes from virtual machines managed by Puppet. My primary goal was to build an observability system similar to what we had when we managed Puppet at Twitter prior to the acquisition. I started building the observability system with the official prometheus community charts [1], but quickly ran into issues where the individual charts didn’t work with each other. I was frustrated with how complicated and difficult to configure these charts were. They weren’t well integrated, so I switched to the kube-prometheus-stack [2] umbrella chart which attempts to solve this integration problem.

The umbrella chart got us further, as long as we didn’t stray too far from the default values, but we quickly ran into operational challenges. Upgrading the chart introduced breaking changes we couldn’t see until they were applied, causing incidents. We needed to manage secrets securely so we mixed in ExternalSecrets with many of the charts. We decided to handle these customizations by implementing the rendered manifests pattern [3] using scripts in our CI pipeline.

These CI scripts got us further, but we found them costly to maintain. Teammates needed to be careful to execute them with the same context they were executed in CI. We realized we were reinventing Hiera to manage a hierarchy of helm values.yaml files to inject into multiple charts.

At this point I started looking for a more holistic solution to this problem of integrating multiple charts together. We saw the value in the rendered manifests pattern, but we couldn’t find an agreed upon implementation. We built a Go command line tool to implement the pattern as a data pipeline. I’d been thinking about the comments from the Why are we templating YAML? [4][5] posts and wondering what an answer to this question would look like.

The Go command line tool was an incremental improvement over the CI scripts, but we still didn’t have a good way to handle the data values. We were still templating YAML which didn’t catch errors early enough. It was too easy to render invalid resources Kubernetes rejected, causing deployment problems. I searched for a solution to manage helm values, something like Hiera which we knew well from Puppet, but not hierarchical because we knew it was important to trace where config values came from in an outage. A few HN comments mentioned CUE [6], and an engineer we worked with at Twitter used CUE to configure Envoy at scale, so I gave it a try. I quickly appreciated how CUE provides both strong type checking and validation of constraints, unifies all configuration data, and provides clarity into where values originate from.

Take a look at Holos if you’re looking to implement the rendered manifests pattern or can’t shake that feeling it should be easier to integrate third party software into Kubernetes like we felt.

[1]: <https://github.com/prometheus-community/helm-charts>

[2]: <https://github.com/prometheus-community/helm-charts/tree/mai...>

[3]: <https://akuity.io/blog/the-rendered-manifests-pattern>

[4]: Why are we templating YAML? (2019) - <https://news.ycombinator.com/item?id=19108787>

[5]: Why are we templating YAML? (2024) - <https://news.ycombinator.com/item?id=39101828>

[6]: <https://cuelang.org/>