Upgrading Cisco Expressway forced mobile Jabber clients to logout randomly

264

Heyas folks,

after upgrading a customers Expressways to Version 14, Jabber clients on mobile devices showed „configuration change detected“ unconditionally.

No configuration change was made.
Sometimes it took hours or days until the message pops up.
Sometime you had it two or more times in half an hour…

We had a cluster of two CUCM with version 11.5.x
Version of Expressway (no Cluster) was 14.0.3.
Version of Jabber clients was latest 14.x on iOS devices.

The issue was only reported for Jabbers used in MRA mode. We downloaded the expressway diagnostics and the Jabber Problem Report and checked these for some occurences of the issue.

In the Jabber Problem Report, one was able to see the warning message that you got to force you signout. It is mentioned that the configuration changed on sipuri :

INFO  [0x0000000106c48580] [impl/system/ServiceEventManager.cpp(180)] [service-event-manager] [addEvent] - Suppress service event - Code:99; action:Signout; severity:Warning; description:Konfigurationsänderungen gefunden. Um die Änderungen zu übernehmen, melden Sie sich ab und wieder an.; extra information:SipUri

INFO  [0x000000017878f000] [efetch/ConfigRefetchHandlerImpl.cpp(863)] [config-refetch] [checkIfHomeClusterChanged] - Home cluster not changed or check is not done

So we started to dig deeper into the topic and the Expressway logs as well.
Here we see for example Request 1816 in the Expressway logs:

INFO  [0x0000000174077000] [etutils/src/http/CurlHttpUtils.cpp(1185)] [csf.httpclient] [configureEasyRequest] - *-----* Configuring request #1816 GET https://expe01.CUSTOMER.de:8443/d2drYXJsbWFyeC5kZS9odHR20xLmFkLndna2FybG1hcnguZGUvNjk3Mg/jabber-config.xml
INFO  [0x0000000174077000] [etutils/src/http/CurlHttpUtils.cpp(2002)] [csf.httpclient] [CurlHeaders] - Number of Request Headers : 1
DEBUG [0x0000000174077000] [etutils/src/http/CurlHttpUtils.cpp(1678)] [csf.httpclient] [addOauthToken] - Using authentication OAUTH with token

The response from Expressway towards the mobile Jabber client is here:

INFO  [0x0000000174077000] [ls/src/http/BasicHttpClientImpl.cpp(668)] [csf.httpclient] [performRequest] - *-----* HTTP response code 200 connect code 0 for request #1816 to https://expe01.CUSTOMER.de:8443/d2drYXJsbWFyeC5kZS9odHR20xLmFkLndna2FybG1hcnguZGUvNjk3Mg/jabber-config.xml
DEBUG [0x0000000174077000] [ls/src/http/BasicHttpClientImpl.cpp(578)] [csf.httpclient] [executeImpl] - Request #1816 -> local IP address: 192.168.5.43, destination IP address: 123.45.6.78

[...]

DEBUG [0x0000000174077000] [netutils/src/edge/EdgeUtilsImpl.cpp(306)] [csf.edge] [buildTokenizedBaseUrlEncoded] - Base Url before encoding: CUSTOMER.de/https/cucm1.CUSTOMER.de/6972

By checking deeply all previous logs we have found that that _cuplogin DNS record is being used in this deployment. That record caused similar issues in the past.

INFO  [0x0000000174077000] [ationdiscovery/DiscoveryLogUtils.cpp(53)] [service-discovery] [LogServiceInformationVect] - *-----* DNS services found: _cuplogin (cup1.CUSTOMER.de)
INFO  [0x0000000174077000] [vices/impl/DiscoveryHandlerImpl.cpp(669)] [service-discovery] [evaluateServiceDiscoveryResult] - ServiceDiscoveryHandlerResult return code SUCCESS
DEBUG [0x0000000174077000] [src/framework/ServicesDispatcher.cpp(38)] [services-dispatcher] [enqueue] - ServicesDispatcher.enqueue: BT-DiscoveryHandlerImpl::updateAuthenticationServices
DEBUG [0x0000000106c48580] [rc/framework/ServicesDispatcher.cpp(220)] [services-dispatcher] [executeTask] -  executed (ConfigServiceImpl::OnConfigChanged) in [0] milliseconds. Waiting time was [11] milliseconds
DEBUG [0x0000000106c48580] [rc/framework/ServicesDispatcher.cpp(207)] [services-dispatcher] [executeTask] -  executing (ConfigServiceImpl::OnConfigChanged)

So we changed the DNS records to be as recommended in the Expressway 14.x MRA guide. (https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/expressway/config_guide/X14-0/mra/exwy_b_mra-deployment-guide-14/exwy_m_requirements-for-mra.html#reference_CF6797423087D57C44244DE06C5CCF36)

We removed the internal DNS entry for_cuplogin._tcp.CUSTOMER.com and for some time, it seems that MRA is working fine again. Without any forced logouts on iOS Jabber clients.

UPDATE a few days later: Issue re-occured.
So we had to start from scratch and somehow a colleague just checked the jabber-config.xml files on the CUCM servers. And there it was:

The jabber-config.xml files had a different file size compared between CUCM Publisher andf Subscriber. We just copied the one from the Publisher over to the Subscriber… let’s see if this was the root cause finally.

Not sure why this wasn’t an issue in combination with the version of Expressway the customer used before…

Cheers,