In the past we used migrations scripts such as db-migrate (https://www.npmjs.com/package/db-migrate) that would run in the CI newly created scripts. However it became clear that when the migration is long, the cost is huge.

We transition to using a dedicated server for migrations. However not having the migrations errors in the CI is troublesome as there is not a single place to check the deployment status, hence reducing the developer's ownership of the migration. We had more migration errors, and we almost hired a DevOps engineer to monitor data migrations.

We move to use a 4-step migration process to migrate the database from state OLD to NEW: 1. Create NEW schema (=create a new field, index, or partition key) that co-exists with the OLD schema. Also write to both NEW and OLD schema. 2. Run the data migration to populate the NEW schema (it is the step that can take from 1h to 20h) 3. Move the read from OLD to NEW (this can be done slowly from 1% of users to 100% if necessary, and revert is easy) 4. (optional) Remove OLD schema to clean out the database

The 4-step migration is great, developers can do 1, 3, 4 with PRs and 2 is ran on our migration server. However it is still a process that needs to be tough, learnt, and applied. There may be better practices and better third parties to do this.

How do you manage data migration in your organization?