Signals Data Factory - How to backup and restore SDF?

Backups for Signals Data Factory are handled via two seperate commands. The first will backup data for signals data factory and the second will backup metadata relating to Signals Inventa

Backup for Signals Data Factory

Within the primary-api and job-scheduler containers is a script called "backup.js". It currently takes a snapshot of all the data stored in the database and compresses it as a "zip" file. It then sends that file to stdout, so it can be piped to a file or into another process.

The zip file will contain one mongo archive file for the PKISSO database, and then a mongo archive file for each workspace that was present in the deployment.
If you include the --data flag, the zip will also include all the data currently stored in the system's object store.
Piping it to stdout also allows you to transfer the data easily across a docker utility. Execute the following command to perform the data backup:

kubectl exec -i $(kubectl get pod -l io.kompose.service=sdf-cs-signals-job-scheduler -o jsonpath="{.items[0].metadata.name}") -- node src/main/scripts/backup.js --data > backup.zip

The command could be scheduled via cron. The file backup.zip could be replaced with a variable that contains the date,
e.g for daily backups, $sdf_backup_$(date '+%d%m%y')

kubectl exec -i $(kubectl get pod -l io.kompose.service=sdf-cs-signals-job-scheduler -o jsonpath="{.items[0].metadata.name}") -- node src/main/scripts/backup.js --data > sdf_backup_$(date '+%d%m%y').zip

Restoring data

The restore script is essentially the inverse of the backup script. It is expecting a zip file from stdin in the same format that is produced by backup.js. It first restores the pkisso database to the location configured by the environment configuration settings, then restores all the workspace databases.

If you also include the --data flag, the restore script will try to restore all data in the system's object store.
Note: This will overwrite any files in the store.

Execute the following command to perform the data restore:

$ kubectl exec -i $(kubectl get pod -l io.kompose.service=sdf-cs-signals-job-scheduler -o jsonpath="{.items[0].metadata.name}") -- node src/main/scripts/restore.js --data < backup.zip

Backup for Signals Inventa

Signals Inventa also has its own MongoDB database for storing some metadata information, such as literals and layouts for Assay Details visualization. In order to backup the MongoDB database for Signals Inventa, on the client machine, execute the following command:

export SIA_MONGO_CONTAINER=$(kubectl get pods --selector=app=sld20 --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep mongo) && kubectl exec -it $SIA_MONGO_CONTAINER -- mongodump --gzip && kubectl exec -it $SIA_MONGO_CONTAINER -- tar czvf backup.tar.gz dump && kubectl cp $SIA_MONGO_CONTAINER:backup.tar.gz backup.tar.gz

sld20 is the name of the Signals Inventa helm release.

If you are not sure of the name, the command helm ls will list all releases. The name will be that for the Signals Inventa chart.

Once again the filename backup.tar.gz could be replaced with an alternate filename that includes the date.

Restoring data

To restore MongoDB database for Signals Inventa, on the client machine, switch to the directory where the backup.tar.gz file exists, then execute the following command.

export SIA_MONGO_CONTAINER=$(kubectl get pods --selector=app=sld20 --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep mongo) && kubectl cp backup.tar.gz $SIA_MONGO_CONTAINER:backup.tar.gz && kubectl exec -it $SIA_MONGO_CONTAINER -- tar zxvf backup.tar.gz && kubectl exec -it $SIA_MONGO_CONTAINER -- mongorestore --gzip

Where sld20 is the name of your Signals Inventa HELM release.

Note: After a restore is done, a 'Load All' needs to be completed to create a brand new index.

Project Revisions

If the backup data size is too large, the number of project revisions can be reduced (default 50)
To do this, alter the option under Administration -> System Configuration