Removing the BOM from Outgoing BizTalk Files

Today we had an issue with sending a file to an external SFTP client – they did not like the Byte Order Mark (BOM) that was automatically pre-pended to the file by the BizTalk file adapter.

A BOM is a short set of invisible characters that is added to the beginning of a UTF-16 or UTF-8 file to indicate the endianess (or byte order) of a text stream. Among other uses, it helps consuming systems determine which encoding (UTF-8 or UTF-16) has been used in creating the stream. The BOM is optional in most cases, and you typically will not see it unless you use a special text editor that can display ISO-8859-1 or CP1252 characters (WinDiff appears to work well for this).

Here is some text that starts with a UTF-8 byte order mark (BOM); notice the three initial odd-looking characters? Represents the sequence 0xEF 0xBB 0xB in hexadecimal, or 239 187 191 in CP-1252 decimal. You won’t see these non-printable characters in most text editors.

Most modern systems can cope with the presence or non-presence of the BOM (especially in UTF-8 where the BOM is really superfluous), but this client clearly had issues with it. So it was off to Dr. Google to find the quickest solution that hopefully involved no custom code (since we are nearly in the Acceptance Testing phase).

Microsoft’s documentation partially came to the rescue, although some claims in this article do not present all the facts (at least not for BizTalk 2010). For example, the statement “If you use a PassThruReceive pipeline or a PassThruTransmit pipeline, a byte order mark is not appended to a message” is questionable, since we were in fact using the PassThruTransmit pipeline and the BOM was definitely present in the output. In addition, the article exclusively suggests creating a custom pipeline as the solution – but this is only necessary if you are not outputting XML.

Our solution? Simply swap the PassThruTransmit pipeline for another OOTB option, the XmlTransmit pipeline. Doing this gives you access to a pipeline PreserveBOM instance property that will strip the BOM from the message for you if you change it from its default setting of “True” to “False”:

BOM_XmlTransmit

That was it! No code changes, no recompilation, not even a restart – just a simple binding update and the troublesome BOM is now history. Note the difference between the before & after output messages compared here in WinDiff:

BOM_WinDiff

BOM’s away!    Winking smile

John Glisson - Geek of the Cloth

Thoughts on integration, technology and what-not...

prashantbiztalkblogs

My BizTalk Experiences

The CRUCIBLE

THINK: It's not illegal....yet.....

paulbouwer.com

life and technology

Abdul Rafay's BizTalk Blog

My experiences with BizTalk related to architecture, development and performance in my enterprise.

Mike Diiorio

Connected Systems and other thoughts

BizTalk musings

Issues, patterns and useful tips for BizTalk development

EAI Guy.net

Enterprise Applicaiton Integration and SOA 2.0

Connected Pawns

Mainly BizTalk & Little Chess

Man Vs. Machine

Why can't we all just get along?

Adventures inside the Message Box

BizTalk, Azure, and other tools in the Microsoft stack - Johann Cooper

Biz(Talk)2

Talk, talk and more talk about BizTalk

Richard Seroter's Architecture Musings

Blog Featuring Code, Thoughts, and Experiences with Software and Services

Sandro Pereira BizTalk Blog

My notes about BizTalk Server 2004, 2006, 2006 R2, 2009, 2010, 2013 and now also Windows Azure BizTalk Services.

BizTalk Events

Calendar of BizTalk events all over the world!

Mind Over Messaging

Musings on BizTalk, Azure, and Enterprise Integration

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.