Problem.
If you notice a high rate of suspended messages due to communication failures between your BizTalk Server and an HTTP server, you should verify if the HTTP Web server you are accessing is Apache, Resin or another similar server of the UNIX / LINUX / Java family. Note that this could apply to other web servers as well, these are just examples I know of.
BizTalk Server always sends HTTP request with the HTTP header setting Connection: Keep-Alive. Moreover, BizTalk expects that the foreign HTTP server obeys to the header setting and does not check the Connection header setting received in the HTTP response.
To be more precise, the problem is double:
- Apache is not obeying to the Connection HTTP header setting and always closes the HTTP connection after every request (as in an HTTP 1.0 fashion). The HTTP/1.1 protocol allows this even if it is not the prefered behaviour.
- BizTalk Server does not react accordingly to the Connection header that is returned back in the HTTP response. This is wrong, following the HTTP/1.1 protocol definition, the client should always check the Connection header returned.
So, on one side there is a Apache closing connections for every HTTP request and on the other side BizTalk is expecting connections to be kept open between requests.
Actually, the source of the problem is not that the Web Server is Apache or not but related to how the Web Server is configured and how it behaves with the connection. Nevertheless, in my experience, this kind of problem only happened with Apache. But I also ever encountered 1 Apache server which kept the HTTP connection open between requests and so no fix was required for that case as no communication problem arose.
For this reason, if you have communication problems between BizTalk and a foreign web server, you should first check if the web server is explicitly closing the HTTP connection after every request.
There is a Microsoft KB that explains the issue and also how to fix it.
Identification of the problem.
As mentioned in the KB, it is important to make sure that your problem is exactly what the KB refers to, so, I will explain how I diagnosed it on my BizTalk Server.
Looking in the BizTalk Administration Console, I saw an abnormal amount of suspended service instances, most of them being of the type “Message”. I also noticed that all the suspended messages belonged to only a subset of our BizTalk applications.
There was clearly something wrong and by checking the Windows Event Viewer, I saw that some BizTalk send ports had an unusual failure rate at the communication level. This explained why so many messages were suspended as messages failing to be sent through a port will be suspended.
The Windows Event Viewer showed messages such as:
A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: Cannot access a disposed object.
Object name: ‘System.Net.Sockets.NetworkStream’.
And
A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
After scratching my head on this issue for a while, I found out that all failing messages where those sent to a port using an HTTP adapter configured with an URI pointing to an Apache server. Other ports using the HTTP adapter but configured to access an IIS server were working fine!
I then used a network sniffer to see what was going on exactly and I saw something inconsistent in the HTTP headers:
The header of the HTTP request contained:
Connection: Keep-Alive
But the header of the HTTP response contained:
Connection: close
server: IBM_HTTP_Server/6.0.2.3 Apache/2.0.47
So, even if the HTTP request specifies “Connection: Keep-Alive”, the Apache Server still close the connection and replies with a “Connection: close” in the header of the HTTP response. For reference regarding the Connection HTTP header, see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10
Solution.
As it is not possible to change the Connection: Keep-Alive header setting in the HTTP adapter shipped with BizTalk Server 2006, I googled a bit and found out the KB 924638 I mentioned earlier in this article which provides 1 solution and 1 workaround for the problem.
The hotfix mentioned in the KB contains a library that makes it possible to modify the Keep-Alive configuration for the HTTP connection. However, as there is no option to set the header setting from the BizTalk Administration Console, it is required to create a custom pipeline which writes a KeepAlive property in the context of the message going through the pipeline. Later on, the HTTP adapter will read the KeepAlive message context property value and create the HTTP connection accordingly.
You can find here a zip file containing 2 Visual Studio projects, one for the Custom Pipeline Component and another one for the Custom Pipeline. In this custom pipeline, the KeepAlive property is not hard-coded and so its value can be set either from the BizTalk Administration Console at run-time or from Visual Studio at design-time.
To summarize, there are 3 steps to apply the solution:
- Call Microsoft Support so that they email you the hotfix containing an executable which updates the Microsoft.BizTalk.HttpTransport.dll (as the hotfix is not directly available to download from the KB).
- Create a custom pipeline which has a property that permits the user to specify the Keep-Alive setting.
- Use the custom pipeline in the send ports that are pointing to web servers explicitly closing connections, such as some Apache web server.