Problem description:
When attempting to collect the SImA server log files as detailed here the output gives a warning "too many open files" which can sometimes cause the log collection script to fail.
Example message as seen in the CLI Windows when collecting the SImA server log files:
*** Collecting logs...
error sending request Get http://127.0.0.1:3100/loki/api/v1/query_range?direction=FORWARD&end=1708528652929831946&limit=100&query=%7Bswarm_stack%3D%22SImA%22%7D&start=1707311613692511039: dial tcp 127.0.0.1:3100: socket: too many open files
Query failed: Run out of attempts while querying the server
Docker project logs collected for project SImA
Docker service logs collected for service SImA_acapella
error sending request Get http://127.0.0.1:3100/loki/api/v1/query_range?direction=FORWARD&end=1708528696591006659&limit=100&query=%7Bswarm_service%3D%22SImA_batch-analysis%22%7D&start=1707897896969859178: dial tcp 127.0.0.1:3100: socket: too many open files
Query failed: Run out of attempts while querying the server
Solution:
"Too many open files" usually refers to the nofile limit in Linux being set too low. The nofile limit (ulimit -n) should be raised to at least 16384 for the user account which is executing the log collection (or all user accounts).
If the "too many open files" warning does cause the log collection to fail, you can also try to collect the logs by reducing the collection period which in turn would reduce the load e.g.
$ get_logs.sh collect --batch 100 --from $(date -Is -d "2 days ago")