"Flush failed to run" SQL error with BAM API

Today I encountered an unusual error when executing a pipeline component that utilises the EventStream API to write to BAM. The failure that showed up in the event log looked something like this:

A message received by adapter "WCF-SQL" on receive location "MyReceivePort" with URI "mssql://MyDatabaseInstance/MyDatabase?InboundId=Employee" is suspended.
Error details: There was a failure executing the receive pipeline: "MyReceivePipeline, MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1a2b345c67d89e0f" Source: "Log Message To BAM Receive" Receive Port: "MyReceivePort" URI: "mssql://MyDatabaseInstance/MyDatabase?InboundId=Employee" Reason: Flush failed to run. 

A quick Google search pulled up this helpful post by Yossi Dahan, which pointed me in the right direction. I knew that the connection string was all right, and I was using the BufferedEventStream rather than the DirectEventStream that Yossi referred to. (Incidentally, when using the BufferedEventStream your connection string actually points to the BizTalkMsgBoxDb database rather than the BAMPrimaryImport database.)

However, the clue was really in the second part of Yossi’s suggestion (and also in an anonymous comment), "…and related permissions…".  I could see that the BizTalk Application Users domain group had been assigned all of the appropriate roles in SQL Server, and I knew that all the host accounts had been dutifully added to this group when BizTalk was installed.

Er… hang on a moment. I double checked and found that the SQL Adapter was running under a new dedicated host account that had been created specifically for the data warehouse. A simple check on the account using the "net user /domain" command prompt unveiled the culprit. This account had not be granted membership in the BizTalk Application Users domain group.

Once that was accomplished, everything worked smoothly.

It would be nice to actually see an error that hinted towards permission issues. Perhaps the detail was buried somewhere in an inner exception, but the logging does not go past the first level.

BizTalk SB-Messaging Receive Adapter Suspends Brokered Messages Without a Body

When it comes to processing zero-byte messages, the built-in receive adapters in BizTalk Server are somewhat inconsistent (see this recent post by Mark Brimble for more information). However, it seems that most receive adapters do not successfully process messages without body content. For example, the File adapter will delete an empty file and kindly put a notification to that effect in the event log. The HTTP adapter will reject a POST request with no content and return a 500 “Internal Server” error. So it probably isn’t any real surprise that the Azure Service Bus Messaging adapter introduced in BizTalk Server 2013 also obstructs bodiless messages. The difference here though is that the message will be successfully received from the queue or topic (and therefore removed from Service Bus), but will immediately be suspended with an error like the following:

A message received by adapter “SB-Messaging” on receive location “SB-ReceivePort_Queue_SB” with URI “sb://<namespace>.servicebus.windows.net/TestQueue” is suspended.
Error details: There was a failure executing the receive pipeline: “Microsoft.BizTalk.DefaultPipelines.PassThruReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35” Source: “Pipeline ” Receive Port: “SB-ReceivePort” URI: “sb://<namespace>.servicebus.windows.net/TestQueue” Reason: The Messaging Engine encountered an error while reading the message stream.

Read more of this post

Why Did My BizTalk Services Stop Working? Check Expired ACS Credentials…

So… let’s say you were one of the early adopters of Microsoft Azure BizTalk Services (MABS) and actually put a solution in production. Everything goes swimmingly for about a year. Then one day your interfaces stop working. Moreover, you will see no error entries in tracking (actually, no entries at all for that interface), although you might see some errors from the client trying to send messages to MABS.

Although the error messages you see may not be very helpful, the length of time that the service has been deployed and running should lead you to suspect some expired credentials. As it turns out, there are multiple levels of credentials and places that they are managed. The SSL certificate might be an obvious one, but many folks forget about the ACS credentials behind the service. Read more of this post

User Group Presentation on Integration Roadmap

Dan PresentingLast night at our Brisbane Azure User Group meetup, I had the privilege of delivering a short presentation on the Microsoft Integration Roadmap that was revealed on Christmas Eve last year, and which I previously blogged about. I couldn’t find any “official” slide deck from Microsoft yet, so I put together a rough deck of my own incorporating some screenshots from the roadmap PDF, a few slides from previous Microsoft decks, and a couple of handmade ones of my own. Feel free to download this from SlideShare and use it if you like.

My presentation was preceded by an excellent session on Azure Application Insights given by Microsoft Solution Architect Todd Whitehead. Amazing to see how easy it is to get so much telemetry from Azure! Looking forward to using this feature more & more.

More photos and details of the Meetup can be found here.

Following the Roadmap to Microsoft Integration

Microsoft has just released a document detailing their roadmap to integration. With all of the recent activity in the cloud around integration – including the release of Microsoft Azure BizTalk Services two years ago, followed by a seemingly different change of direction with Azure App Service announcement earlier this year – there has been much confusion about where Microsoft was headed in the integration space. This has been challenging for partners and customers who want to ensure that they invest in the “right” technology when building out their enterprise integration capability.

I am pleased to say that this document finally delivers some much-needed clarification in this respect. Aside from reinforcing that “BizTalk is not dead” and confirming some key new features in the much-anticipated BizTalk Server 2016 release, it also shows how Microsoft is aiming to close the gap between traditional on-premises integration afforded by the server product and the modern API-based approach offered in Azure:

convergence

Read more of this post

Load Balancing with the Azure App Service File Connector

The File Connector is one of the built-in protocol apps that are available in the Marketplace when you go to provision and API App. Through configuration only, this app allows you to perform file-based operations from a Logic App in Azure, bridging the boundaries of your corporate network:

image

The documentation from Microsoft clearly explains how you can configure the app and then download a listener agent to install on your on-premises server associated with the file(s). In most cases, this would be a single server – but I got to wondering what would happen if you installed the agent on multiple servers? So I tried it out using both  the “Get File” and “Upload File” operation.

Turns out that the File Connector will talk to all of those servers, provided that you have set up the same base directory path on all of them. This path is configured at the time you provision the API App – not when you use it within a Logic App. The configuration of the “File Path” property within the instance of this app only defines the sub-directory within the base path, as well as the file name. Incidentally, if this sub-directory does not exist at runtime, it is automatically created for you in the case of the “Upload File” operation.  Unfortunately, this is not the case with the base directory – if that doesn’t exist you get a rather meaningless 500 error recorded in the tracking log:

The requested page cannot be accessed because the related configuration data for the page is invalid.

In any case, I set up a simple Logic App and decided to test using the “Get File” and ‘Upload File” methods respectively to fetch and store files from assorted VMs hosted on my laptop. Once I had the syntax down for specifying the file path (this post by Saravana Kumar proved very helpful), then I was amazed at how easy it was to get this working! Here is my Logic App:

Read more of this post

Latest Pluralsight Course is Now Live!

Pluralsight

Really pleased to announce that my latest Pluralsight course on Azure Hybrid Connections is now live!  This has consumed no insignificant amount of time and effort lately – which goes a long way to explaining the infrequency of my blogging lately.

This is only a short “first look” course, although it took me several weeks to complete it due to work commitments and a virus that stole my voice for awhile. There are two reasons for the short course:

  • Since this is still a “preview” feature in Azure, a short introductory course seemed prudent;
  • Turns out that in 1 hour 48 minutes I was able to cover pretty much everything you need to know about Hybrid Connections anyway!

Read more of this post

Collection of Microsoft Integration Stencils for Visio 2013

Great new set of Microsoft hybrid integration Visio stencils from Sandro Pereira – includes BizTalk, BizTalk Services, Azure App Service, PowerBI and more!

Sandro Pereira BizTalk Blog

In my two previous editions (here and here) I published some shapes to represent BizTalk Server physical architectures, Integration architectures (Cloud or Hybrid scenarios) or solutions.

In this resource I change the name of my Visio 2013 Stencils resource, now is called “Microsoft Integration Stencils for Visio 2013”, to include almost the Microsoft Integration Stack.

This package contains a set of symbols/icons (258 shapes) to that will help you visually represent Integration architectures (On-premise, Cloud or Hybrid scenarios) and solutions diagrams in Visio 2013. It will provide symbols/icons to visually represent features, systems, processes and architectures that use BizTalk Server, Microsoft Azure and related technologies.

  • BizTalk Server
  • Microsoft Azure
    • BizTalk Services
    • Azure App Service
    • Event Hubs
    • and so on
  • PowerBI and devices
  • Infrastructure
  • And many more…
BizTalk Server

BizTalk-Server-Stencils-Visio-2013-01

BizTalk Services

BizTalk-Services-Stencils-Visio-2013-02

Azure App Service

Azure-App-Service-Stencils-Visio-2013-02

Azure

Azure-Stencils-Visio-2013-04

Infrastructure

Infraestructure-Stencils-Visio-2013-05

PowerBI and Devices

PowerBI-Devices-Stencils-Visio-2013-06

That you can use and resize without losing…

View original post 124 more words

Issue with Hybrid Connections and the PowerShell API

While recently playing around with the PowerShell API Azure Hybrid Connections, I stumbled across an interesting “feature” when managing registrations within the on-premises Connection Manager.

As outlined in the MSDN documentation, there are five cmdlets that are included with the Hybrid Connection Manager that you download and install from the Azure portal:

  • Add-HybridConnection
  • Update-HybridConnection
  • Delete-HybridConnection
  • Get-HybridConnection
  • Set-HybridConnectionManagerConfiguration

The first three require a connection string as a parameter, which you can copy from the Hybrid Connection itself in the Azure Portal by clicking the “Manage Connection” button:

Read more of this post

Busy Days!

If you’re wondering why my blogging frequency has dropped off over the last couple of months, it’s because I’ve been busy with some other projects. Here are just a few of them:

Brisbane Azure Bootcamp

BAB2015_Dan_OrientationAlthough originally intended to be the Brisbane chapter of the Global Azure Bootcamp, we had to change the date to a week later since the global event coincided with ANZAC Day in Australia. This year marking the 100th anniversary of the event at Gallipoli, it seemed irreverent (and also impractical) to hold the event on that day. So we joined Perth in organising our own version of the all-day event. Alessandro Cardoso from Readify and myself were the main organisers, while Damien Berry set up the event website.

 

BAB2015_Dan_HCWhat a success! With over 50 registrations, we had a full house at Microsoft headquarters, with the event stretching from 8:30am to 5:30pm. Presenters included Readify’s Alessandro Cardoso and Andrew Harcourt, myself on Hybrid Connections, and Microsoft’s Steve van Bodegraven with a session on Azure ML and lean start-ups.

Lots of prizes were given out thanks to sponsors Mexia, Pluralsight, and the Global Azure Bootcamp. Also a big thanks to Readify for funding the catering.

Brisbane Azure User Group

For the next Brisbane Azure User Group event, I’ll be presenting with Mexia’s Lee Simpson on Logic Apps – so preparation for that is keeping me occupied as well.

Pluralsight

And if that was not enough, I’m also authoring my second Pluralsight course. While I’m not allowed to say too much about the content until it’s published, I can reveal that it is based on an exciting Azure technology directly related to hybrid integration. Only one more module to go! Expect an announcement in a few weeks. 🙂

Of course there are other things stealing my time (and sleep) away as well, including heavier than usual workloads at our current client and preparation for a looming deadline for lodging my income tax return. But I hope to be back to more regular blogging next month! There are plenty of exciting things to write about…

John Glisson - Geek of the Cloth

Thoughts on integration, technology and what-not...

prashantbiztalkblogs

My BizTalk Experiences

The CRUCIBLE

THINK: It's not illegal....yet.....

paulbouwer.com

life and technology

Abdul Rafay's BizTalk Blog

My experiences with BizTalk related to architecture, development and performance in my enterprise.

Mike Diiorio

Connected Systems and other thoughts

BizTalk musings

Issues, patterns and useful tips for BizTalk development

EAI Guy.net

Enterprise Applicaiton Integration and SOA 2.0

Connected Pawns

Mainly BizTalk & Little Chess

Man Vs. Machine

Why can't we all just get along?

Adventures inside the Message Box

BizTalk, Azure, and other tools in the Microsoft stack - Johann Cooper

Biz(Talk)2

Talk, talk and more talk about BizTalk

Richard Seroter's Architecture Musings

Blog Featuring Code, Thoughts, and Experiences with Software and Services

Sandro Pereira BizTalk Blog

My notes about BizTalk Server 2004, 2006, 2006 R2, 2009, 2010, 2013 and now also Windows Azure BizTalk Services.

BizTalk Events

Calendar of BizTalk events all over the world!

Mind Over Messaging

Musings on BizTalk, Azure, WCF, and Enterprise Integration

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.