Show HN: Nexa SDK – Build powerful and efficient AI apps on edge devices

27 points

2 years ago

Hey HN! Alex and Zack here from Nexa AI. We're excited to share something we've been working on.

Our journey began with the Octopus series --- action models for mobile AI agents (https://huggingface.co/NexaAIDev/Octopus-v2). We focused on making sub-billion parameter models excel at function calling, making high accurate and fast function-calling possible on mobile and edge devices. But as we delved into developing full-fledged on-device applications, we hit a roadblock.

We realized that optimizing for function calling (tool-use) alone wasn't enough. Building powerful on-device AI apps requires a diverse set of tools: language models with domain expertise, speech processing, image generation, embedding models and more. That's when we decided to create Nexa SDK --- a comprehensive toolkit that brings together everything developers need to build powerful and efficient AI applications that run entirely on-device.

Here's what Nexa SDK offers:

  - Support for both ONNX and GGML models.
  - An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements.
  - An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models. 
  - An OpenAI-compatible API server with optimization in function calling.
  - A Streamlit UI for rapid prototyping.
  - An intuitive CLI for easy model management.
  - Backend optimizations for latency and power consumption on edge devices.

We've designed Nexa SDK to be the go-to solution for developers pushing the boundaries of what's possible with on-device AI applications and AI on edge devices.

To showcase its capabilities, we've built several demo apps running entirely on your device (https://github.com/NexaAI/nexa-sdk/tree/main/examples):

  - AI soulmate with uncensored model and audio-in/audio-out interaction.
  - A quick interface for uploading and chatting with PDFs like your personal finance documents.
  - A meeting transcription app supporting multiple languages and real-time translation.

We're proud to share that the winner of yesterday's (Sep 7) House AGI "AI PC/ GenAI Goes Local" hackathon used Nexa SDK to build a local semantic image search (https://github.com/asl3/deja-view).

But we're just getting started! There are lots of exciting developments in our pipeline, and we can't wait to share them with you soon!

Check it out: (https://github.com/NexaAI/nexa-sdk)

Docs: (https://docs.nexaai.com/)

If you're excited about the future of on-device AI, we'd really appreciate your support. A star on our GitHub repo goes a long way in helping us reach more developers!

Cheers,

Alex & Zack

4 comments

- Support for both ONNX and GGML models. - An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements. - An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models. - An OpenAI-compatible API server with optimization in function calling. - A Streamlit UI for rapid prototyping. - An intuitive CLI for easy model management. - Backend optimizations for latency and power consumption on edge devices.

- AI soulmate with uncensored model and audio-in/audio-out interaction. - A quick interface for uploading and chatting with PDFs like your personal finance documents. - A meeting transcription app supporting multiple languages and real-time translation.