Outlines is a Python library that focuses on text generation with large language models. While the library mainly focuses on open source models we have an integration with the OpenAI API.
When trying to implement Tree of Thoughts we became frustrated with the iterative workflow in which other libraries forced us. What we wanted was to be able to pass an array of prompts or completions with arbitrary shape, ask for several samples, and get an array of completions back.So we implemented a vectorization routine inspired by NumPy's. This makes our OpenAI integration have the "broadcast" behavior described above. All call are performed concurrently.
We thought this could be of use to some folks here who are trying to implement complex workflows with the OpenAI API.