Ask HN: Infinite Byte Streams AaS?

8 points

7 years ago

I’m looking for signal from fellow HN’ers on whether this is appealing as a managed service:

Append to a stream:

  $ curl -X PUT --data-binary @- https://service/my/stream/name << EOF
  {
    "some": “arbitrary”,
    “data”: [1,2,3]
  }
  {"payload”: 4}
  EOF

Appends are byte-oriented (here, they only happen to be JSON). The API promises that appends don’t interleave (streams are linearizable), and are durable upon a 200 OK response. PUT from cloud functions, lambdas, app engine, etc, or integrate streams as third-party webhook callbacks.

Read from a stream:

  $ curl "https://service/my/stream/name?offset=24&block=true"
    “data”: [1,2,3]
  }
  {"payload”: 4}
  {“next”: “append”}

Blocking reads receive new content as it commits to the stream. You can think of it as a restart-able firehose (offset=-1 reads from the current stream end). Permission a stream to partners, customers, or the world.

Content of the stream is batched into regular compressed files that persist to a s3://, etc bucket you own and permission to the service. Historical reads serve directly from your bucket, and the service doesn’t retain PUT content for any longer than is required to durably persist it. Files themselves hold only raw content — if you PUT JSONL, you’ll get JSONL files you can feed to your processing pipelines.

You may want to have multiple streams: annotate & organize them into topics and partitions using Kubernetes-syntax labels and selectors (--selector message-type=MyMessage,environment=qa).

Backstory: I’m describing a managed offering of Gazette’s broker (github.com/gazette/core). The project has a bunch of other neat stuff, but for now we’re exploring what might resonate as a distilled MVP. Thanks!