There are already various cool tools out there that allow you to build your own operator-like experience but they usually only automate web browser actions, or aren’t open sourced/cost a lot to get started. Spongecake allows you to automate desktop-based interactions, and is fully open sourced which will help:

- Developers who want to build their own computer use / operator experience - Developers who want to automate workflows in desktop applications with poor / no APIs (super common in industries like supply chain and healthcare) - Developers who want to automate workflows for enterprises with on-prem environments with constraints like VPNs, firewalls, etc (common in healthcare, finance)

Technical details: This is technically a web browser pointed at a backend server that 1) manages starting and running pre-configured docker containers, and 2) manages all communication with the computer use agent. [1] is handled by spinning up docker containers with appropriate ports to open up a VNC viewer (so you can view the desktop), an API server (to execute agent commands on the container), a marionette port (to help with scraping web pages), and socat (to help with port forwarding). [2] is handled by sending screenshots from the VM to the computer use agent, and then sending the appropriate actions (e.g., scroll, click) from the agent to the VM using the API server.

Some interesting technical challenges we ran into:

- Concurrency - We wanted it to be possible to spin up N agents at once to complete tasks in parallel (especially given how slow computer use agents are today). This introduced a ton of complexity with managing ports since the likelihood went up significantly that a port would be taken. - Scrolling issues - The model is really bad at knowing when to scroll, and will scroll a ton on very long pages. To address this, we spun up a Marionette server, and exposed a tool to the agent which will extract a website’s DOM. This way, instead of scrolling all the way to a bottom of a page - the agent can extract the website’s DOM and use that information to find the correct answer

What’s next? We're working on adding support in the UI to run this locally on your own machine, and to spin up other desktop environments like Windows and MacOS. We’ve also started working on integrating Anthropic’s computer use model as well. There’s a ton of other features we can build but wanted to put this out there first and see what others would want

Would really appreciate your thoughts, and feedback. It's been a blast working on this so far and hope others think it’s as neat as I do :)

Here’s the link to clone: https://github.com/aditya-nadkarni/spongecake