Image Artist: How Image Artist Parallelises Batch Analysis

Overview

Image Artist (ImA) uses Slurm as its internal workload scheduler. During batch analysis, ImA does not process an entire plate as a single unit. Instead, each well is submitted as an independent analysis task.

For example:

• A 96-well plate generates 96 analysis tasks.
• A 384-well plate generates 384 analysis tasks.
• Two 384-well plates generate 768 analysis tasks.

These tasks are placed into the selected batch analysis queue and executed as compute resources become available.

Can Multiple Plates Be Processed in Parallel?

Yes.

Image Artist can process wells from multiple plates concurrently, provided sufficient CPU and memory resources are available.

There is no requirement for one plate to complete before another plate begins processing. When multiple batch analysis jobs are submitted, Slurm schedules individual well-analysis tasks from all queued jobs onto available compute resources.

As a result:

• Multiple plates can be analysed simultaneously.
• Wells from different plates may be processed at the same time.
• Larger jobs do not necessarily block smaller jobs from starting.
• The degree of parallelisation depends on available server resources and the queue used.

Understanding Task-Level Parallelisation

The key concept is that parallelisation occurs at the well level, not at the plate level.

What Determines Parallel Capacity?

The key concept is that parallelisation occurs at the well level, not at the plate level.

The number of wells that can be analysed simultaneously depends on:

1. Available CPU resources.
2. Available memory resources.
3. The queue selected for analysis. 
   • Refer to: Image Artist: What is the difference between the batch analysis job queues?
4. Other jobs currently running on the ImA server.

Image Artist will only start a task when the requested CPU and memory resources are available.

The maximum number of concurrent tasks is determined by whichever resource (CPU or RAM) becomes exhausted first.

Does Submitting More Plates Increase Parallelism?

Not necessarily.

Submitting multiple plates creates more work for the scheduler, but it does not increase the amount of compute available.

For example, if a system can run 10 well-analysis tasks simultaneously, it will continue to run a maximum of 10 tasks regardless of whether those tasks originate from:

• One plate
• Two plates
• Ten plates

Additional jobs are queued and processed as resources become available.

Throughput vs. Completion Time

Running multiple plates simultaneously can improve overall laboratory throughput because work begins on multiple datasets immediately.

However, because all jobs share the same pool of compute resources, individual plates may take longer to complete when many analyses are running concurrently.

In general:

• More available CPUs and RAM increase parallelism.
• Lower-resource queues (such as SImA quick) allow more tasks to run concurrently.
• Higher-resource queues (such as SImA default) provide more resources per task but reduce the total number of concurrent tasks.

Summary

Image Artist parallelises analysis at the well level. Multiple plates can be analysed concurrently, and wells from different plates may run at the same time. The maximum level of parallelism is determined by the CPU and memory capacity of the ImA server together with the resource requirements of the selected analysis queue. This means that submitting several plates is supported, but the total number of wells that can be processed simultaneously is ultimately constrained by available compute resources.