Inventa data publish fails with ECONNREFUSED due to Ceph RGW readiness issues

Keywords: Inventa ON-PREM, Data Publishing, Load All, Load New, Failed to start, ECONNREFUSED, Ceph S3, rook-ceph-rgw, Snapshot Queued, Signals Data Factory, sdf-cs-signals-job-scheduler

Description:
This KB article addresses an issue encountered in on-premises Inventa 3.6 deployments where data publication jobs ("Load All" or "Load New") fail immediately with a !Failed to start status, and subsequent snapshot jobs remain indefinitely in a Queued state.

During data ingestion, the job scheduler fails to upload dataset maps to the internal Ceph S3 storage endpoint (locker.data.ceph:80), generating connection refused (ECONNREFUSED) errors in the logs.

Information:

The underlying cause of this publication failure is related to the readiness and lock state of the internal Ceph Object Gateway (RGW):

• The sdf-cs-signals-job-scheduler service attempts to upload dataset map files to Ceph S3 on port 80, but receives connect ECONNREFUSED x.x.x.100:80.
• Inspection of the Kubernetes storage namespace shows the rook-ceph-rgw-signals-store pod stuck in a non-ready (0/1 READY) state.
• Ceph daemon logs reveal lock contention during lifecycle operations (e.g., failed to acquire lock on lc.X), preventing the RGW endpoint from accepting HTTP/S3 connections.

Resolution Steps:

1. Verifying System and Pod Health: Check the cluster and pod status to confirm if the storage gateway is unresponsive:

• Execute the system diagnosis script: ./scripts/collect-system-status-info.sh
• Inspect pod readiness in the storage namespace: kubectl get pods -n rook-ceph -l app=rook-ceph-rgw
• Verify whether the rook-ceph-rgw pod is in a 0/1 READY state or failing its readiness probe.

2. Restarting the Server / Host: To release stuck lifecycle locks and restore Ceph RGW services:

• Perform a graceful restart of the host server running the Signals Data Factory and Rook-Ceph services.
• After the reboot, verify that all application and storage pods return to a healthy 1/1 READY running state.
• Confirm that the Ceph S3 endpoint is reachable and listening on port 80.

3. Retrying Data Ingestion: Once pod readiness is restored, re-trigger the data publish ("Load All" or "Load New") operation in Inventa to confirm that dataset map packaging and publishing complete successfully.

Conclusion: When Inventa data publishing fails with S3 connection refusal errors due to Ceph RGW lifecycle lock contention, restarting the server resets the daemon state, restores gateway readiness, and enables data publishing to proceed normally.