I created a browser extension that records your screen and then sends keyframes over to the GPT Vision API for annotation.

The use case for my software is creating interactive product demos.

Here is my app giving a demo of itself (a milestone I am extremely proud of!):

https://www.interactivedemo.ai/d/APRNDj9hoONXXDu

In the demo, you can see how I'm extracting key frames from the screen recording and sending them over to the LLM with various annotations. I use AWS Lambda with FFMpeg layers for that.

For the keyframes, I draw a red dot on the cursor using Python Imaging Library (PIL) and I also draw sequence and timestamp information under the frames. I have a prompt I've been tuning for a while and it's not perfect but, it still saves a ton of time and has great results.

The system prompt is:

"Generate brief, action-oriented web interaction descriptions. Use varied verbs, focus on direct results, and leverage provided context. Pay close attention to both original and scaled coordinates."

but the user prompt is my secret sauce ;-)

I am launching it on Product Hunt in a few days here:

https://www.producthunt.com/posts/interactivedemo-ai

I created InteractiveDemo.ai because I like how Screen Studio feels, but it is Desktop Mac only, and it doesn't have interactive features. Also, I like the ability to create my own A.I. features on top of interactive demos and just the space in general because I've been developing and showing off software for 24 years. It's a natural fit for me.

Check it out at https://www.InteractiveDemo.ai

I appreciate your support and am happy to answer any questions HN has.