Is it possible to enable parallel processing in Columbus?

Is it possible to enable parallel processing in Columbus?

Background Info:

There are 3 distinct job scheduling queues within Columbus: Import, Export and Batch Analysis.

By default, jobs within each queue are processed sequentially. However, Jobs within distinct queues may run in parallel. For example, only one Import job is permitted to run at any one time, although this may run in parallel with both an analysis and an export job.

This queuing system is shared by all users within Columbus on a first-come first-served basis, regardless of the user who is submitting the job.

Activating parallel processing:

It is possible to configure Columbus so that jobs within each job queue are processed in parallel.

Please note the parallel processing script is not extensively tested nor officially supported. Please bear this in mind when leveraging this script and use your own judgement on its use.

Parallel processing can be activated in two ways:

1) Transiently via a script file. This method comes with the caveat that if the Columbus service is restarted, the changes are lost and the script will need to be executed again to re-activate parallel processing.

The 'Update_NumberOfJobs.script' is available to download, here: Update_NumberOfJobs.zip

The following text in the script tells you how many jobs are executed in parallel.

// Batch Analysis Limit

set(newNumberOfBatchJobs=2)

// Import Limit

set(newNumberOfImportJobs=2)

// Export Limit

set(newNumberOfExportJobs=2)

…where “2” is the number of desired parallel jobs. Modify the above lines in the script as required.

Steps to run the script to activate parallel processing:

- Place the script file in any directory on the Columbus server and make executable

- Connect to the Columbus server, move to the directory where the script is located and execute the script with the following command

$ acapella Update_NumberOfJobs.script

If the script works successfully the output will be similar to the following:

START Updating Job Limit

Connect to Acapella Server on: localhost:8282

Connection round trip time: 0.0063 s

Get current Batch Job Limit

Current Batch Job Limit: 1

Set new Batch Job Limit: 2

Get current Import Job Limit

Current Import Job Limit: 1

Set new Import Job Limit: 2

Get current Export Job Limit

Current Export Job Limit: 1

Set new Export Job Limit: 2

DONE

2) Permanently via the Acapella server start-up script. This method ensures any modifications to the job queueing system will survive Columbus service restarts and server reboots.

 
Edit the /var/lib/acapella/acapellaserverd.script file on the Columbus server

Insert the following lines after the lines which start with "acc:startserver":

acc::setoption("BackgroundJobLimit", "Import", 2)

acc::setoption("BackgroundJobLimit", "Export", 2)

acc::setoption("BackgroundJobLimit", "Batch", 2)

…where “2” is the number of desired parallel jobs.

Next, restart the Columbus service:

$ sudo /etc/init.d/columbus restart

For information on how to query the existing job queue limits, please see the following FAQ post:

How do I query the current job queue limits?