Signals Notebook: Duplicate batches across paginated Materials search results (Python Script via SNB API endpoint)

Description

When querying Materials → Compounds batches via the entities/search API, duplicates appeared across pages for large result sets. The issue was caused by non-deterministic ordering (score ties) when no explicit sort was provided. Adding a deterministic sort field (e.g., id/EID or name) to the search options stabilized pagination and eliminated duplicates.

Cause

- The search relied on default relevance sorting, which can vary between requests and pages when many documents share similar scores, leading to shifting items and apparent duplicates across pages.

Solution

- Specify an explicit, deterministic sort in the API request to ensure stable ordering across pages.

Steps

- Update the entities/search payload to include options.sort with a stable field (e.g., internal id/EID or name). For example: options: { sort: { "id": "asc" } }
- Re-run pagination with consistent page[offset] and page[limit] parameters.
- Verify that total count and unique IDs remain consistent across runs and pages.

Notes

- Deterministic sorting is recommended for any large, paginated result sets to avoid page drift and duplicates.