Show HN: Automated code instrumentation for structured events

21 points

2 years ago

Hey HN! Sam, Alex and Ben from Invaria (formerly Patchwork Technologies): http://invaria.io. Observability is built on telemetry (metrics, logs and traces). Telemetry data quality is often poor because instrumenting code is still a time consuming task which requires a lot of hand-finishing. We believe that structured event data is the right end goal, and we’re starting by making it easier to get there using structured logging.

For our Launch HN (Aug), we set up a demo instance with pre-baked results. We wanted to go further to allow HN readers to give the product a proper test drive. https://open.invaria.io includes some pre-analysed OSS (Go, TS and Python) repos (no sign-in required). It also allows users to enroll additional OSS repos (limited to Go for now) that they’d like to see analysed (magic link sign-in required for this to allow us to manage state). The demo focuses on batch analysis of existing repos, providing an overview of instrumentation health, line-level code improvements with rationale, and related GH pull request examples (in practice these are usually bundles of many changes). A quick video here to help you find your way: https://youtu.be/IM9EFl4gwVs. Any questions, please email us at [email protected].

Since our last post, we’ve been iterating with our first customer/design partner in order to drive up the quality and consistency of outputs. We’re hitting 85% change acceptance rate with our customer on their main Go repo (1m+ LoC, 1000s of logging statements). Logs are their main telemetry source for debugging, and they want structured logs as standard. Yielding consistent results from LLMs is hard (no news there), and that’s been a big focus for us since our Launch HN. Breaking down prompts, refining the context we fetch from the syntax tree, and providing examples is where we have focused efforts.

We’re building Invaria because getting high quality telemetry from applications is still time-consuming and hard. We discuss it here in more detail: https://www.invaria.io/post/your-telemetry-is-garbage. OpenTelemetry has helped bridge the gap, but the event data that comes from auto-instrumented tracing, e.g. at the API level, is not enough for the typical (imperfect) application. It’s usually necessary to manually add further span and attributes, and to clean up any logging that’s burned into spans.

We’re starting with logging statements as the simplest form of event-level telemetry. Invaria can upgrade existing logging statements, and instrument key events with new statements. The demo instance demonstrates ‘batch mode’ on legacy code, and we are beta testing a CI-time integration right now. The goal is to automate instrumentation at the CI step, so that developers can focus on the creative engineering work.

Once again, your honest feedback would mean a lot to us. We have a lot of conviction that observability needs a shake up and that better outcomes require higher data quality. We’d love to hear what you think works and doesn’t work in our current approaches, and whether Invaria solves a problem for you.

A note on the name change: we really like the name Patchwork, but there are several other software companies/products with similar names. It caused some confusion during launches. We like invariant, a word borrowed from the control theory domain. We thought making high quality telemetry a constant was a nice mission statement.