Why Did My BizTalk Services Stop Working? Check Expired ACS Credentials…
January 25, 2016 Leave a comment
So… let’s say you were one of the early adopters of Microsoft Azure BizTalk Services (MABS) and actually put a solution in production. Everything goes swimmingly for about a year. Then one day your interfaces stop working. Moreover, you will see no error entries in tracking (actually, no entries at all for that interface), although you might see some errors from the client trying to send messages to MABS.
Although the error messages you see may not be very helpful, the length of time that the service has been deployed and running should lead you to suspect some expired credentials. As it turns out, there are multiple levels of credentials and places that they are managed. The SSL certificate might be an obvious one, but many folks forget about the ACS credentials behind the service.
Your first thought is that the SSL certificate you’ve uploaded may have expired. So you log into the BizTalk Services Management Portal (not the Azure portal), click Resources and inspect the certificates:
Sure enough, your certificate has expired! Well, that explains your client connectivity issues. So you simply obtain a new SSL certificate and then (using the classic Azure portal) upload it using the “Update SSL Certificate” link on the dashboard. If that works, then sweet – you’re all set (probably).
But what if you find that this action fails repeatedly with basically no helpful error message (something like “Could not add the SSL Certificate for BizTalk Service <service name>“):
Of course this could be caused by an invalid certificate or the wrong password provided. But assuming that has already been verified, what do you do now?
The key here is that the service has been live for at least a year. That makes the ACS credentials highly suspect, as the default expiration for a number of the keys, passwords and certificates (and there are quite a few) is one year from the service creation date.
Access Control Services (ACS) is how MABS is secured in a number of respects, including the administrative management functions (like uploading an SSL certificate for example). These settings are managed in their own ACS Management portal, which you can get to by clicking the “Connection Information” button at the bottom of the dashboard screen in the classic management portal:
This will open a dialog window, at the bottom of which is a link to the ACS Portal:
Now that you know how to view the settings, there is an excellent Channel 9 video by Jaganathan Thangavelu which shows you all of the relevant symmetric keys, passwords and certificates that should be checked and (if expired or expiring) should be replaced or modified. You can also read the Certificates and Keys Management Guidelines on MSDN. I won’t repeat all of the details here, but some MABS relevant items worth checking are:
- Service Settings –> Certificate and Keys: Service Namespace (Key & Certificate)
- Administration –> Management Service: Management Client (Key & Password)
Clicking the hyperlink for the relevant item (Symmetric Key, X.509 Certificate, Password, etc) takes you to a screen that allows you to edit the expiration date and/or replace the expired certificate.
So now that you’ve fixed expired ACS credentials up, will your service work again? Well, not quite… The next thing you have to do is synchronise those keys. Unfortunately, there isn’t a reliable way to do that through the portal or any other user interface. With a little help from Microsoft, I learned how to do this via PowerShell. But it needs a library to be installed first, and since that code relies on a legacy version 0.6.19 of the Windows Azure PowerShell (August 2013) you may find you have to rollback to an earlier install of this (I got this installer from here).
Ideally, you should backup your MABS instance before performing any operation like this. This can be accomplished by clicking the Backup button at the bottom of the dashboard screen in the classic portal, and selecting the storage account when prompted. However, if the ACS settings have not been synchronised, you may find this fails (worth trying anyway):
The instructions for performing the sync are:
Once you run this command, it can take several minutes and multiple attempts before the process finally succeeds:
Now you should be able to upload the new SSL certificate and get back up and running!
With so many moving parts and limited documentation, you can begin to understand a few of the challenges of using the BizTalk Services platform. It is no wonder the Microsoft is recommending the use of Logic Apps in favour of MABS for any new projects. After reviewing the Roadmap for Integration, it seems to be a given that MABS will eventually fade away into the sunset as Logic Apps is expanded to become a comprehensive integration platform. Already, most of the functionality in MABS is available through a premium App Service edition by way of the numerous API Apps and Connectors available in the marketplace. But that’s another topic…