We are the team behind TKYO Drift, a new open source AI drift detection library. Our non-blocking, fully async tool takes whatever input/outputs your txt2txt workflow generates and embeds them using sentence similarity models to compare user inputs and model outputs over time to let you know when either user drift, model drift, or both is occurring.

We measure drift using Cosine Similarity, Euclidean Distance and Scalar Metric Population Stability indices for each user input or AI output you’d like to track. The library comes with a CLI command to spot check the average Cosine Similarity or Scalar Population Stability logs, as well as easily exportable logs as .csv files for your external data visualization tools.

Setup is simple; Run the npm install command npm install tkyodrift Install the python dependencies in the requirements.txt from our node_modules folder Pip install -r ./node_modules/tkyodrift/requirements.txt Add the import line to your code import tkyoDrift from ‘tkyodrift’ Add the function hook to your workflow as many times as you need for each input/output tkyoDrift(yourInput, ioType)

Once the function hook is in, the system will syphon your user inputs and begin tracking their similarity over time giving you access to fine grain details on how much your user’s inputs or model’s outputs are changing over time. The system works with multiple chains of thought as well, so you can add the function hook anywhere you need in order to track drift at different hand off points within your workflow.

For more in-depth instructions, please view our library’s ReadMe on Github or NPM and if you have any questions, please comment below or reach out to us at [email protected]

Here is our Website: https://www.tkyodrift.com NPM package:https://www.npmjs.com/package/tkyodrift Github: https://github.com/oslabs-beta/tkyo-drift

PS. This tool is primarily made in Javascript, and meant for use in Javascript environments. Python does very little in the main function (making one-off embeddings), but you will need to pip install the requirements.txt in the tkyodrift subdirectory of your node_modules folder for this to work. We rely on python transformers for embedding speed and python HNSW and KMeans libraries for quick data retrieval.