Show HN: NOS – A fast, and ergonomic PyTorch inference server

3 points

2 years ago

Hey HN, we built an inference server called NOS that can run a whole host of open-source AI models (LLMs, Stable Diffusion, CLIP, Whisper, Object Detection etc) all under one-roof.

You can pull and run NOS in a few lines of code - simply `pip install torch-nos`

and run `nos serve up --http`. You should now be able to talk to NOS over HTTP or gRPC (see README for examples).

NOS can run locally on your desktop (with a gaming GPU), in any cloud GPU (L4, A100s, etc) and even on CPUs (without any acceleration). We’ll soon support Apple Silicon, so you should be able to run your AI models locally on a Macbook.

Why are you building yet another inference server?

Most API server implementations today deeply couple the API framework (FastAPI, Flask) with the modeling backend (PyTorch, TF etc.) - in other words, it doesn’t let you separate the concerns for the backend (i.e. scale-out, memory-efficiency, async/batched execution etc.) from the API (auth, observability, telemetry etc.), especially if you’re looking to build a production-ready application.

Why use NOS?

We’ve tried to make it very easy for developers to add support for new models and take them to production.

Here are a few things we think developers care about:

- Simple API over gRPC or REST that supports batched requests, and streaming. - Support any OSS model with custom runtimes with pip, conda and cuda dependencies. - Serve multiple custom models simultaneously on a single or multi-GPU instance. - Local execution means that you control your data, and you’re free to build NOS for domains that are more restrictive with data privacy. - Fully containerized means that you can develop, test and deploy NOS locally, on-prem, on any cloud or AI CSP. - Written entirely in Python, Apache-2.0 License.

Try it out! Check out one of our demos in the NOS playground (https://github.com/nos-playground) (lots of video models) and let us know what you think!