TCP connection persistence required in NTLM authentication process

TCP connection persistence required in NTLM authentication process

The NTLM authentication process consists of three HTTP requests (after an initial HTTP 401 response).

1. NTLMSSP_NEGOTIATE_MESSAGE (sent from the client to the server), Type 1 .
2. NTLMSSP_CHALLENGE (sent from the server to the client), Type 2 .
3. NTLMSSP_AUTHENTICATE_MESSAGE (the final request from the client to the server), Type 3 .

These requests, when coming to the server and originating from a client and potentially passing through a load balancer or proxy, must use the same TCP connection in order for authentication to work. Sever.log can be used to determine if the same IP + client Port is used for the HTTP requests. If the requests number 1 and 3 are using different remote (client) ports, request number 3 will be treated as a type 1 message in a new authentication flow and there will be an error (“Not a Type 1 message”) in server.log.

The following is an example from server.log showing the connection. Notice the client port being different when it fails.

Working NTLM auth

server.security.SecurityFilter: Filtering request: HTTP POST /ws/LoginService on port 80 from 10.25.125.15 at port number 53132
server.security.AuthenticationManager: Found HTTP header: Authorization NTLM TlRM...
server.security.SecurityFilter: Filtering request: HTTP POST /ws/LoginService on port 80 from 10.25.125.15 at port number 53132
server.security.AuthenticationManager: Found HTTP header: Authorization NTLM TlRM...
server.security.NtlmAuthenticator: NTLM authentication handshake completed, principal 'BOO\principaluser' successfully authenticated

Not a Type 1 message

server.security.SecurityFilter: Filtering request: HTTP POST /ws/LoginService on port 80 from 10.25.125.15 at port number 53132
server.security.AuthenticationManager: Found HTTP header: Authorization NTLM TlRM...
server.security.SecurityFilter: Filtering request: HTTP POST /ws/LoginService on port 80 from 10.25.125.15 at port number 53140
server.security.AuthenticationManager: Found HTTP header: Authorization NTLM TlRM...
server.security.NtlmAuthenticator: NTLM authentication error
  jespa.security.SecurityProviderException: Not a Type 1 message (NTLMSSP_NEGOTIATE_MESSAGE)

There can be other problems related to the connection. If the connection is reused for a different authentication session, there is a possibility that there will be an error "Invalid account name" since the final message contains the response for the wrong challenge (mixup among the authentication sessions).

The problem may be more prevalent when Web Player 6.5 or 7.0 is using NTLM+Impersonation for logging in to the Spotfire server due to the potentially large amount of logins originating from the same source IP within a limited period of time. The likelihood that the Web Player or a proxy will change its source IP or port during a NTLM handshake is then greater and furthermore the risk of running into this problem is lower when logging in from a Spotfire Analyst client directly on the Spotfire server.

When using a load balancer in front of the Spotfire Servers and using NTLM authentication you should  not only have session persistence (for the JSESSIONID cookie) but also connection persistence for NTLM to work. For example, F5 Big IP has some specific settings for this.