Why Did My BizTalk Services Stop Working? Check Expired ACS Credentials…

So… let’s say you were one of the early adopters of Microsoft Azure BizTalk Services (MABS) and actually put a solution in production. Everything goes swimmingly for about a year. Then one day your interfaces stop working. Moreover, you will see no error entries in tracking (actually, no entries at all for that interface), although you might see some errors from the client trying to send messages to MABS.

Although the error messages you see may not be very helpful, the length of time that the service has been deployed and running should lead you to suspect some expired credentials. As it turns out, there are multiple levels of credentials and places that they are managed. The SSL certificate might be an obvious one, but many folks forget about the ACS credentials behind the service.

SSL Certificate

Your first thought is that the SSL certificate you’ve uploaded may have expired. So you log into the BizTalk Services Management Portal (not the Azure portal), click Resources and inspect the certificates:

Sure enough, your certificate has expired! Well, that explains your client connectivity issues. So you simply obtain a new SSL certificate and then (using the classic Azure portal) upload it using the “Update SSL Certificate” link on the dashboard. If that works, then sweet – you’re all set (probably).

But what if you find that this action fails repeatedly with basically no helpful error message (something like “Could not add the SSL Certificate for BizTalk Service <service name>“):

Of course this could be caused by an invalid certificate or the wrong password provided. But assuming that has already been verified, what do you do now?

The key here is that the service has been live for at least a year. That makes the ACS credentials highly suspect, as the default expiration for a number of the keys, passwords and certificates (and there are quite a few)  is one year from the service creation date.

ACS Credentials

Access Control Services (ACS) is how MABS is secured in a number of respects, including the administrative management functions (like uploading an SSL certificate for example).  These settings are managed in their own ACS Management  portal, which you can get to by clicking the “Connection Information” button at the bottom of the dashboard screen in the classic management portal:

This will open a dialog window, at the bottom of which is a link to the ACS Portal:

Now that you know how to view the settings, there is an excellent Channel 9 video by Jaganathan Thangavelu which shows you all of the relevant symmetric keys, passwords and certificates that should be checked and (if expired or expiring) should be replaced or modified. You can also read the Certificates and Keys Management Guidelines on MSDN. I won’t repeat all of the details here, but some MABS relevant items worth checking are:

• Service Settings –> Certificate and Keys: Service Namespace (Key & Certificate)
• Administration –> Management Service: Management Client (Key & Password)

Clicking the hyperlink for the relevant item (Symmetric Key, X.509 Certificate, Password, etc) takes you to a screen that allows you to edit the expiration date and/or replace the expired certificate.

Key Synchronisation

So now that you’ve fixed expired ACS credentials up, will your service work again? Well, not quite…  The next thing you have to do is synchronise those keys. Unfortunately, there isn’t a reliable way to do that through the portal or any other user interface.  With a little help from Microsoft, I learned how to do this via PowerShell. But it needs a library to be installed first, and since that code relies on a legacy version 0.6.19 of the Windows Azure PowerShell (August 2013) you may find you have to rollback to an earlier install of this (I got this installer from here).

Ideally, you should backup your MABS instance before performing any operation like this. This can be accomplished by clicking the Backup button at the bottom of the dashboard screen in the classic portal, and selecting the storage account when prompted. However, if the ACS settings have not been synchronised, you may find this fails (worth trying anyway):

The instructions for performing the sync are:

Download the Windows Azure BizTalk Services Management Sample from https://code.msdn.microsoft.com/Windows-Azure-BizTalk-91e1bdf3/sourcecode?fileId=83229&pathId=435215204 Unzip and open the WindowsAzureBizTalkServiceManage solution (in C# subfolder) in Visual Studio 2012 (or later). Open Utility.cs and, on Line 345, change “StoreLocation.LocalMachine” to “StoreLocation.CurrentUser” . Build the solution and the Microsoft.WindowsAzure.Management.BizTalkService.dll file will be created in the \bin\Debug folder. Follow the Prerequisites section at https://code.msdn.microsoft.com/Windows-Azure-BizTalk-91e1bdf3#content Open Windows Azure Powershell and run the following command to import the module you just built:Import-Module <dll path>\Microsoft.WindowsAzure.Management.BizTalkService.dll Now type “Get-AzurePublishSettingsFile” (this will open up your browser and allow you to download a .publishsettings file) Back in the same PowerShell window, run the following command to import the file for your subscription:Import-AzurePublishSettingsFile <path to downloaded file> If your .publishsettings file contains multiple subscriptions, you must ensure that the subscription containing your BizTalk Services instance is currently selected:Select-AzureSubscription “<subscription name>” Run the following command to kick off the sync operation:Sync-AzureBizTalkServiceACSKeys -ResourceName <name of your MABS instance>

Once you run this command, it can take several minutes and multiple attempts before the process finally succeeds:

Now you should be able to upload the new SSL certificate and get back up and running!

With so many moving parts and limited documentation, you can begin to understand a few of the challenges of using the BizTalk Services platform. It is no wonder the Microsoft is recommending the use of Logic Apps in favour of MABS  for any new projects. After reviewing the Roadmap for Integration, it seems to be a given that MABS will eventually fade away into the sunset as Logic Apps is expanded to become a comprehensive integration platform. Already, most of the functionality in MABS is available through a premium App Service edition by way of the numerous API Apps and Connectors available in the marketplace. But that’s another topic…