I created a demo involving ChatGPT, OpenAI's Whisper running on-device for speech-to-text (because why not?), and ARKit. By defining a custom JavaScript environment with JavaScriptCore and describing it to ChatGPT, I was able to get it to produce functioning scripts that instantiate and manipulate 3D objects.

Sketchfab is used to import 3D assets on the fly. Google Poly had a much better catalog of assets for this sort of thing but sadly, it's no longer available.

The README.md describes how it all works but the gist of it is that user prompts are wrapped in a larger prompt that describes the environment (i.e., which functions are available to use and how they work) and constraints. The code is then executed directly. Assets for objects are fetched as needed by searching Sketchfab.