That's why I created the `transformrs` Rust library [3]. The name is from "transform", which is essentially what AI models do, and "Rust". It is also a nod to the transformers algorithm.
To allow for proper testing in CI, I considered to make a mock endpoint that I could run my tests against. However, AI APIs are so cheap nowadays that I figured I could just as well run the tests against the actual endpoints. So that is what currently happens. Locally and in CI I have set up 5 different API keys and I run multiple tests against all of these endpoints, so that's about 30 requests each time (tests run in 10 seconds). I change a file or each push to CI. It's incredible how stable these endpoints have become. 18 tests check whether the prompt "This is a test. Please respond with 'hello world'" actually responds with "hello world" and so far >99% of the times it does. CI is way less flaky than I would have expected.
In any case, I hope people will find the library useful. I'm curious for feedback and suggestions.
[1]: https://news.ycombinator.com/item?id=42952605