Why Did My BizTalk Services Stop Working? Check Expired ACS Credentials…

So… let’s say you were one of the early adopters of Microsoft Azure BizTalk Services (MABS) and actually put a solution in production. Everything goes swimmingly for about a year. Then one day your interfaces stop working. Moreover, you will see no error entries in tracking (actually, no entries at all for that interface), although you might see some errors from the client trying to send messages to MABS.

Although the error messages you see may not be very helpful, the length of time that the service has been deployed and running should lead you to suspect some expired credentials. As it turns out, there are multiple levels of credentials and places that they are managed. The SSL certificate might be an obvious one, but many folks forget about the ACS credentials behind the service.

SSL Certificate

Your first thought is that the SSL certificate you’ve uploaded may have expired. So you log into the BizTalk Services Management Portal (not the Azure portal), click Resources and inspect the certificates:

SNAGHTML74e2b8c

Sure enough, your certificate has expired! Well, that explains your client connectivity issues. So you simply obtain a new SSL certificate and then (using the classic Azure portal) upload it using the “Update SSL Certificate” link on the dashboard. If that works, then sweet – you’re all set (probably).

But what if you find that this action fails repeatedly with basically no helpful error message (something like “Could not add the SSL Certificate for BizTalk Service <service name>“):

image

Of course this could be caused by an invalid certificate or the wrong password provided. But assuming that has already been verified, what do you do now?

The key here is that the service has been live for at least a year. That makes the ACS credentials highly suspect, as the default expiration for a number of the keys, passwords and certificates (and there are quite a few)  is one year from the service creation date.

ACS Credentials

Access Control Services (ACS) is how MABS is secured in a number of respects, including the administrative management functions (like uploading an SSL certificate for example).  These settings are managed in their own ACS Management  portal, which you can get to by clicking the “Connection Information” button at the bottom of the dashboard screen in the classic management portal:

image

This will open a dialog window, at the bottom of which is a link to the ACS Portal:

SNAGHTML7a6ae2e

Now that you know how to view the settings, there is an excellent Channel 9 video by Jaganathan Thangavelu which shows you all of the relevant symmetric keys, passwords and certificates that should be checked and (if expired or expiring) should be replaced or modified. You can also read the Certificates and Keys Management Guidelines on MSDN. I won’t repeat all of the details here, but some MABS relevant items worth checking are:

  • Service Settings –> Certificate and Keys: Service Namespace (Key & Certificate)
  • Administration –> Management Service: Management Client (Key & Password)

SNAGHTML4e86302

SNAGHTML9ebcb03

Clicking the hyperlink for the relevant item (Symmetric Key, X.509 Certificate, Password, etc) takes you to a screen that allows you to edit the expiration date and/or replace the expired certificate.

Key Synchronisation

So now that you’ve fixed expired ACS credentials up, will your service work again? Well, not quite…  The next thing you have to do is synchronise those keys. Unfortunately, there isn’t a reliable way to do that through the portal or any other user interface.  With a little help from Microsoft, I learned how to do this via PowerShell. But it needs a library to be installed first, and since that code relies on a legacy version 0.6.19 of the Windows Azure PowerShell (August 2013) you may find you have to rollback to an earlier install of this (I got this installer from here).

Ideally, you should backup your MABS instance before performing any operation like this. This can be accomplished by clicking the Backup button at the bottom of the dashboard screen in the classic portal, and selecting the storage account when prompted. However, if the ACS settings have not been synchronised, you may find this fails (worth trying anyway):

SNAGHTML5301532

The instructions for performing the sync are:

  1. Download the Windows Azure BizTalk Services Management Sample from https://code.msdn.microsoft.com/Windows-Azure-BizTalk-91e1bdf3/sourcecode?fileId=83229&pathId=435215204
  2. Unzip and open the WindowsAzureBizTalkServiceManage solution (in C# subfolder) in Visual Studio 2012 (or later).
  3. Open Utility.cs and, on Line 345, change “StoreLocation.LocalMachine” to “StoreLocation.CurrentUser” .
  4. Build the solution and the Microsoft.WindowsAzure.Management.BizTalkService.dll file will be created in the \bin\Debug folder.
  5. Follow the Prerequisites section at https://code.msdn.microsoft.com/Windows-Azure-BizTalk-91e1bdf3#content
  6. Open Windows Azure Powershell and run the following command to import the module you just built:Import-Module <dll path>\Microsoft.WindowsAzure.Management.BizTalkService.dll
  7. Now type “Get-AzurePublishSettingsFile” (this will open up your browser and allow you to download a .publishsettings file)
  8. Back in the same PowerShell window, run the following command to import the file for your subscription:Import-AzurePublishSettingsFile <path to downloaded file>
  9. If your .publishsettings file contains multiple subscriptions, you must ensure that the subscription containing your BizTalk Services instance is currently selected:Select-AzureSubscription “<subscription name>
  10. Run the following command to kick off the sync operation:Sync-AzureBizTalkServiceACSKeys -ResourceName <name of your MABS instance>

Once you run this command, it can take several minutes and multiple attempts before the process finally succeeds:

SNAGHTMLa224472

Now you should be able to upload the new SSL certificate and get back up and running!

With so many moving parts and limited documentation, you can begin to understand a few of the challenges of using the BizTalk Services platform. It is no wonder the Microsoft is recommending the use of Logic Apps in favour of MABS  for any new projects. After reviewing the Roadmap for Integration, it seems to be a given that MABS will eventually fade away into the sunset as Logic Apps is expanded to become a comprehensive integration platform. Already, most of the functionality in MABS is available through a premium App Service edition by way of the numerous API Apps and Connectors available in the marketplace. But that’s another topic…

About Dan Toomey
Husband, father, Enterprise integration geek, Microsoft Azure MVP, Pluralsight author, Brisbane Azure User Group leader (@BrisbaneAzureUG), MCPD, MCT, MCTS & former professional musician.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

John Glisson - Geek of the Cloth

Thoughts on integration, technology and what-not...

prashantbiztalkblogs

My BizTalk Experiences

The CRUCIBLE

THINK: It's not illegal....yet.....

paulbouwer.com

life and technology

Abdul Rafay's BizTalk Blog

My experiences with BizTalk related to architecture, development and performance in my enterprise.

Mike Diiorio

Connected Systems and other thoughts

BizTalk musings

Issues, patterns and useful tips for BizTalk development

EAI Guy.net

Enterprise Applicaiton Integration and SOA 2.0

Connected Pawns

Mainly BizTalk & Little Chess

Man Vs. Machine

Why can't we all just get along?

Adventures inside the Message Box

BizTalk, Azure, and other tools in the Microsoft stack - Johann Cooper

Biz(Talk)2

Talk, talk and more talk about BizTalk

Richard Seroter's Architecture Musings

Blog Featuring Code, Thoughts, and Experiences with Software and Services

Sandro Pereira BizTalk Blog

My notes about BizTalk Server 2004, 2006, 2006 R2, 2009, 2010, 2013 and now also Windows Azure BizTalk Services.

BizTalk Events

Calendar of BizTalk events all over the world!

Mind Over Messaging

Musings on BizTalk, Azure, and Enterprise Integration

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.

%d bloggers like this: