Columbus Plus Job stuck 'Submitting'

Columbus Plus Job stuck 'Submitting'

If you have started a job that does not advance beyond 'Submitting' it could be one of the following:

Celery is not running

Access the server's command line interface as the root user and issue the following command:

$ /etc/init.d/columbus status celery

Celery should return:

Celery: up [ OK ]

If not try a restart with:

$ /etc/init.d/columbus restart celery

The next step varies if you are using an on premises cluster install or an leveraging Amazon EMR 

On Prem

Keyless authentication is not working

The Columbus server needs to be able to connect to the submit host (AFA Master Node) via ssh in order to run create_cluster_job.py

Check you can ssh from server to submit host as the columbus user. On the Columbus server login or switch into the columbus user:

$ su - columbus

Check you can ssh to the submit host. Typically the command would take the form:

$ ssh columbus@SUBMIT_HOST

Where SUBMIT_HOST is the hostname of the submit host

If you are asked for a password thenin lies the problem.

AWS

Can be caused by incorrect setting in /etc/columbus/***.config file

If AWS availability zone is used in error, as oppose to region name. 

An example AWS region would be us-east-1, its availability zone being us-east-1e. 

If us-east-1e us used as the region_name in the config file, results in batch analysis stuck submitting. 

Typical error that is found in /var/log/columbus/web/long_tasks.log

EndpointConnectionError: Could not connect to the endpoint URL: "https://elasticmapreduce.us-east-2c.amazonaws.com/"

If you wanted to check the URL within the CLI use a command such as:

$ curl https://elasticmapreduce.us-east-2c.amazonaws.com && echo

Which returns Couldn't resolve host... in the above us-east-1e case but returning a resonable <MissingAuthenticationTokenException> in the case of us-east-1.

Though if none of the above helps gather up all Columbus logs and provide to Technical Support.