Product: Columbus
Why do Batch Analysis jobs remain in a Submitting state when submitting to an AWS cluster?
Problem:
Batch analysis jobs remain in "Submitting" state when submitting to AWS.
The following error is seen in the /var/log/columbus/web/long-tasks.log...
[2020-01-21 17:28:50,722: ERROR/MainProcess celery.worker.job] Task cluster.tasks_submit_job.submit_jobs_to_cluster[89d12eb7-6b61-4629-863d-0db2c526ff18] raised unexpected: AttributeError("'NoneType' object has no attribute 'lower'",)
Traceback (most recent call last):
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/virtualenv/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/virtualenv/lib/python2.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/columbus/cluster/tasks_submit_job.py", line 96, in submit_jobs_to_cluster
count_successfully_submitted = ClusterJobSubmitter.submit(job_data, valid_datasets)
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/columbus/cluster/tasks_submit_job.py", line 71, in submit
submission_infos = cluster.submit(jobs, script_filename, parameter_filename)
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/virtualenv/lib/python2.6/site-packages/coca/cluster.py", line 388, in submit
cluster_id = self._start_cluster()
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/virtualenv/lib/python2.6/site-packages/coca/cluster.py", line 211, in _start_cluster
cluster_config['keep_alive_when_idle'] = 'true' if configuration.str2bool(self.emr_parameters.get('keep_alive_when_idle')) else 'false'
File "/usr/local/PerkinElmerCTG/Columbus2.9/webapp/virtualenv/lib/python2.6/site-packages/coca/configuration.py", line 18, in str2bool
return value.lower() in ("yes", "true", "t", "1")
AttributeError: 'NoneType' object has no attribute 'lower'
Cause:
The "keep_alive_when_idle" parameter has not been defined in the Columbus cluster configuration file for the AWS instance.
Solution:
Add the following line...
keep_alive_when_idle: False
...to the "[emr]" section of the cluster configuration file found in /etc/columbus