Show HN: Tangent – Security log pipeline powered by WASM

28 points

7 months ago

Hi HN! We’re Ethan and Danny, the authors of Tangent (https://github.com/telophasehq/tangent), a Rust-based log pipeline where all normalization, enrichment, and detection logic runs as WASM plugins.

We kept seeing the same problems in the OCSF (https://ocsf.io) community: 1) Schemas change constantly. Large companies have whole teams dedicated to keeping vendor→OCSF mappings up to date. 2) There’s no shared library of mappings, so everyone recreates the same work. 3) Writing mappers is tedious, repetitive work. 4) Most pipelines use proprietary DSLs that are hard to share and hard for tools/LLMs to generate.

Tangent takes a different approach: no DSLs – mappings and enrichments are just normal code compiled to WASM, shareable plugins – we maintain a community library (https://github.com/telophasehq/tangent-plugins), interoperability – we can run other engines’ DSLs (e.g., Bloblang) inside WASM for easy migration, full flexibility – plugins can validate schemas, call external APIs (https://github.com/telophasehq/tangent/blob/main/examples/en...), or perform complex transforms (https://github.com/telophasehq/tangent-plugins/blob/main/zee...).

Here's an example Python transformation plugin to drop all fields from a log except `message`:

  import json
  
  from typing import List
  
  from wit_world.imports import log
  
  # `log.Logview` is Tangent's zero-copy JSON accessor type.
  
  def process_logs(self, logs: List[log.Logview]) -> bytes:
  
      out = bytearray()
  
      for lv in logs:
  
          msg = lv.get("msg")
  
          value = msg.value if msg is not None else ""
  
          out.extend(json.dumps({"message": value}).encode() + b"\n")
  
      return bytes(out)

We have plenty more examples in the repo.

Because plugins are just Go/Python/Rust, LLMs can create new mappers with ease. For example, I asked:

  Generate a mapper from AWS Security Hub Finding to OCSF

and only had to make a few minor tweaks. (https://github.com/telophasehq/tangent-plugins/blob/main/aws...)

Performance-wise, a 16-core Amazon Linux box processes ~480 MB/s end-to-end (TCP → Rust-WASM transform → sink) on ~100-byte JSON logs. The CLI includes tooling to scaffold, test, and benchmark plugins locally. Here's a deep dive into how we are able to get this performance: https://docs.telophasehq.com/runtime.

We’d love to get your feedback! What do you think?

2 comments

import json from typing import List from wit_world.imports import log # `log.Logview` is Tangent's zero-copy JSON accessor type. def process_logs(self, logs: List[log.Logview]) -> bytes: out = bytearray() for lv in logs: msg = lv.get("msg") value = msg.value if msg is not None else "" out.extend(json.dumps({"message": value}).encode() + b"\n") return bytes(out)