Rollbacks
Before you consider a rollback, please read the README carefully.
Testing
Before performing a rollback at a customer site it is a good thing to practice the same rollback (between the same versions) on a test cluster and to write down the first and last migration numbers in the range.
Gotchas
Migrations have to be invertible
The migrations that fall in the necessary range for a rollback have to be invertible. This has to be checked both on the DB and on the source code side. For example if the up migration drops a column then the down migration cannot recover that data. This could be ok if the column has NULLABLE values, and the code in the previous version can tolerate a column with NULL entries.
If a migration in the range is not invertible then manual intervention is necessary. This involves for example populating a column with some default value. In some cases a down migration is not possible because it would break assumptions already made in code.
It is good to check all the migrations involved in a rollback for being invertible.
Large tables
A rollback at a customer could involve very large tables. In this case it makes sense to test the rollback with a test cluster that has similarly large amounts of data to get an idea how long the migrations will take and how large the transactions will be.
Squashed migrations
If the rollback overlaps with squashed migrations then careful sorting out of the migration ranges is required. This should definitely be tested on a test cluster.
Performing many migrations
The number of migrations between two versions can be large. Doing them one-by-one manually can become tedious, especially if we ask an admin at a customer site to do it. An option here is to use migrate similar to how we use it in dev.
The following are steps to accomplish this:
Before starting the rollback, check that the customer DB is at the expected migration number:
Without namespace:
kubectl exec $(kubectl get pod -l app=pgsql -o jsonpath='{.items[0].metadata.name}') -- psql -U sg -c 'SELECT * FROM schema_migrations'
With namespace ns-sourcegraph
(change as appropriate):
kubectl -n ns-sourcegraph exec $(kubectl -n ns-sourcegraph get pod -l app=pgsql -o jsonpath='{.items[0].metadata.name}') -- psql -U sg -c 'SELECT * FROM schema_migrations'
- Deploy the previous version of Sourcegraph onto the cluster. Make sure that the dirty DB flag is
false
. The frontends will complain that the DB is at a newer schema version. That is expected and ok. - Prepare a
migrations.tar.gz
package file with the contents of themigrations
directory at the current customer version. - Copy both
migrations.tar.gz
and themigrate
binary onto the pgsql pod. - We will run
migrate
on the pod to eliminate kubectl port-forwarding of the postgres DB port. This is important if migrations take a while and a dropped connection could interfere:
On the pgsql
pod and pgsql
container execute:
PGSSLMODE=disable PGUSER=sg ./migrate -source file://<path to the unpacked migration dir> -database postgres://localhost:5432/sg down <number of migrations downward to previous version>
Once done, check the migration number again to see if it is at the expected previous version. Also check that the dirty DB flag is still false
.
If everything looks good, restart the frontend pods.