We rented these servers from Hetzner, because we were using them for hosting anyway and their offering is inexpensive, on the order of 1/5th or 1/10th of what we would pay on AWS. When an engineer needed a server, we would manually order it on the provider's website, put Ubuntu on it, and give them SSH access. I wrote a short guide with recommendations on firewall and such, but the engineer was expected to set it up themselves.

Our team wasn't particularly big - it was 15 engineers - so manual management of these servers was not a big issue. This approach wouldn't scale well for bigger teams though.

You could automate provisioning and managing servers with Terraform, and I've heard that a friend did that at his company. There's also a product called Coder (coder.com) that you deploy on your infrastructure, define workspace templates with Terraform as a team lead, and your developers get a web UI where they can provision dev environments themselves.

If you want full automation you can also use a SaaS like Gitpod or Github Codespaces, where you define a workspace with a configuration file in your Git repository alongside your code. These services connect to your Git provider (like Github or GitLab), and you can provision a fully configured dev environment on demand on their cloud, which is a container or a virtual machine. The resulting workspace contains the latest version of your codebase. You can also set up CI for the dev environment, which they call "prebuilds", which you can use to install dependencies, compile your code, and more. This CI runs for every commit. The resulting artifacts are available in your dev environment when you open it. You can also specify init scripts that run when a workspace boots up, so you can start a database or your web app automatically.

When I tested these solutions I found Gitpod/Codespaces most pleasant to use from a developer's perspective. Any developer can make changes to the dev environment and they become instantly available to the rest of the team. Prebuilds are a great time-saver when you jump between projects or Git branches. Init scripts make your day-to-day coding more pleasant.

One downside is that you have to give these services access to all of your code. I feel particularly uncomfortable with Gitpod because they use containers and user namespaces for virtualization, and the latter had some security vulnerabilities in the past (see for example https://coder.com/blog/statement-on-the-recent-cve-2022-0185-vulnerability). Gitpod also somewhat restricts what you can do in the workspace: you can't run a Kubernetes cluster inside or even deploy the full Elastic Stack because you have to modify some restricted system limits. Codespaces uses VMs so I don't have the same reservations about it.

Another downside is that these services get pricey if you want a more powerful machine. A 16 core server that you use 6 hours a day will cost you about $170 a month on Codespaces.

Personally I wanted a solution that would work like Codespaces but would be self-hosted. It would use VMs under the hood so I wouldn't be restricted in the dev environment and it would integrate prebuilds and init scripts for boosted productivity. I couldn't find anything like that so I'm now building it with a friend. It's at https://hocus.dev if you want to check it out.

I'm curious how others usually manage dev servers, or do you even use them. I've heard that development on ARM macbooks is nice, but personally I've had problems with Docker on them, so I prefer a Linux box.