Pranay and Ankit here. We’re founders of SigNoz ( https://signoz.io ), an open source observability platform. We are building an open-core alternative to DataDog for companies that are security and privacy conscious, and are concerned about huge bills they need to pay to SaaS observability vendors.

Observability means being able to monitor your application components - from mobile and web front-ends to infrastructure, and being able to ask questions about their states. Things like latency, error rates, RPS, etc. Better observability helps developers find the cause of issues in their deployed software and solve them quickly.

Ankit was leading an engineering team, where we became aware of the importance of observability in a microservices system where each service depended on the health of multiple other services. And we saw that this problem was getting more and more important, esp. in today’s world of distributed systems.

The journey of SigNoz started with our own pain point. I was working in a startup in India. We didn’t use application monitoring (APM) tools like DataDog/NewRelic as it was very costly, though we badly needed it. We had many customers complaining about broken APIs or a payment not processing - and we had to get into war room mode to solve it. Having a good observability system would have allowed us to solve these issues much more quickly.

Not having any solution which met our needs, we set out to do something about this.

In our initial exploration, we tried setting up RED (Rate, Error and Duration) and infra metrics using Prometheus. But we soon realized that metrics can only give you an aggregate overview of systems. You need to debug why these metrics went haywire. This led us to explore Jaeger, an open source distributed tracing system.

Key issues with Jaeger were that there was no concept of metrics in Jaegers, and datastores supported by Jaeger lacked aggregation capabilities. For example, if you had tags of “customer_type: premium” for your premium customers, you couldn’t find p99 latency experienced by them through Jaeger.

We found that though there are many backend products - an open source product with UI custom-built for observability, which integrates metrics & traces, was missing.

Also, some folks we talked to expressed concern about sending data outside of boundaries - and we felt that with increasing privacy regulations, this would become more critical. We thought there was scope for an open source solution that addresses these points.

We think that currently there is a huge gap between the state of SaaS APM products and OSS products. There is a scope for open core products which is open source but also supports enterprise scale and comes with support and advanced features.

Some of our key features - (1) Seamless UI to track metrics and traces (2) Ability to get metrics for business-relevant queries, e.g. latency faced by premium customers (3) Aggregates on filtered traces, etc.

We plan to focus next on building native alert managers, support for custom metrics and then logs ( waiting for open telemetry logs to mature more in this). More details about our roadmap here ( https://signoz.io/docs/roadmap )

We are based on Golang & React. The design of SigNoz is inspired by streaming data architecture. Data is ingested to Kafka and relevant info & meta-data is extracted by stream processing. Any number of processors can be built as per business needs. Processed data is ingested to real-time analytics datastore, Apache Druid, which powers aggregates on slicing and dicing of high dimensional data. In the initial benchmarks we did for self-hosting SigNoz, we found that it would be 10x more cost-effective than SaaS vendors ( https://signoz.io/blog/signoz-benchmarks/ )

We’ve launched this repo under MIT license so any developer can use the tool. The goal is to not charge individual developers & small teams. We eventually plan on making a licensed version where we charge for features that large companies care about like advanced security, single sign-on, advanced integrations and support.

You can check out our repo at https://github.com/SigNoz/signoz We have a ton of features in mind and would love you to try it and let us know your feedback!