Ask HN: Affordable hardware for running local large language models?

31 points

2 years ago

A while back there was a post about running stable diffusion on a Raspberry Pi Zero 2 [1] which was slow but incredibly impressive! And that sparked my curiosity with this question, what is considered affordable hardware for running large language models locally today? I'm aware that there's a lot of work underway to make inference cheap to run at the edge but I'm curious to understand the landscape at present that anyone could purchase. I seen people running models on flagship smartphones but those are more expensive then a mac mini with worse performance.

By affordable I mean no greater then the cost of a current gen base model mac mini ($599) but ideally around the price of a raspberry pi 5 ($79) which when searching for budget PC gets mentioned[2]. Both devices have the same amount of ram in my case (8gb) but different performance observed given the importance of memory bandwidth. I mention these two because I've had experience running llama 3 via ollama on both with success although of slower speeds compared to a full workstation with a commodity GPU i.e. RTX 4090 which starts at ($1599). I'm interested in learning about what other devices are out there that people consider cheap and use for LLMs locally.

[1]: https://news.ycombinator.com/item?id=38646969

[2]: https://www.pcmag.com/picks/the-best-budget-desktop-computers

17 comments