Posted on

AdWords API and .Net client

I have been recently given the task to be in charge of a project using the AdWords API.

The AdWords API actually consists of a serie of Web Services which let you access to your AdWord account through an API instead of a normal user interface. The documentation regarding the Web Services can be found at: https://developers.google.com/adwords/api/docs/
To access the AdWords API using a .Net client, there is a .Net solution with pre-generated Web Service proxies that you can find at http://code.google.com/p/google-api-adwords-dotnet/.

The documentation regarding the .Net client is pretty scarce but there is an AdWords API forum where you can find solutions to your problems and also post your own questions.

Please note that the problem I describe hereunder applies to the latest .Net client available at time of writing (mid-January). This blog post applies only if you encounter the same problem with your version of the .Net client code. Would a future release not show the problem, no fix will be needed.

The only problem I have had so far with the proxy classes is that they constantly threw me an error message saying that “The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.”
This problem is similar to what I have previously encountered in BizTalk projects when .Net HTTP requests access a Web Server which ignores the keepAlive instruction. This is often the case with Apache servers (or should I say their administrators? – I am not sure if this is a setting they can change). Anyway, for people interested, I have posted more details about the problem in a previous post: BizTalk HTTP Adapter and Apache Keep-Alive issue.

So, the first thing I had to do to fix the issue is to modify every Web Service proxy classes so that the HTTP request used to make the Web Service call has its KeepAlive setting set to false.
All Web Service proxy classes that are using SOAP extends from the SoapHttpClientProtocol class. Web methods belonging to the WebService are called using the Invoke() method. Nevertheless, we, as client programmers, do not call the Invoke method directly, it is instead the proxy class which wraps the Invoke() call into a method matching the WebService method signature. For example, the ReportService proxy class delivered in the AdWords API .Net client has a method validateReportJob() which matches the WebService method defined on the server and which encapsulates the call to the Invoke() method:
public void validateReportJob(ReportJob job)
{
this.Invoke(“validateReportJob”, new object[] {job});
}

The SoapHttpClientProtocol.Invoke() method is implemented using the protected method GetWebRequest () which creates an instance of a WebRequest object that the Invoke() method uses to establish an HTTP connection with the web server.

With this in mind, all we have to do is to override the GetWebRequest() method in our ReportService proxy class so that we can intercept the WebRequest object created and modify the properties we want to customize.

Note that the WebRequest class is in fact an abstract classs and that the actual object returned at runtime is a class implementing the protocol used. In cases of Web Services and everything that runs over HTTP, it is the HttpWebRequest class.
So in our case, I have added the following code in the ReportService proxy class:

protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
System.Net.HttpWebRequest webRequest = (System.Net.HttpWebRequest)base.GetWebRequest(uri);
webRequest.KeepAlive = false; //set the HTTP Header to not keep connections alive between requests
webRequest.Timeout = 1200000; // The length of time, in milliseconds, until the request times out (20 minutes here)
return webRequest;
}

With this piece of code in each of the Web Service classe proxies, “the connection closed” exception is not thrown anymore as both the .Net client and the Apache server that runs on http://adwords.google.com have the same KeepAlive setting.

Posted on

Custom Pipeline Component deployment gotcha.

When you create custom pipeline, you might want to compile and install the custom pipeline component in the GAC first before adding the pipeline component in the toolbox. That way, the custom pipeline solution will reference the custom pipeline component from the GAC instead of its location on the disk.
The reason behind this is that if you install the pipeline component library on your production servers only in the GAC and if you did not GACed the pipeline component before using it in the custom pipeline, an exception is raised saying that the custom pipeline cannot be found (as the custom pipeline refers to a custom pipeline component on the local drive instead of the GAC). See a full explanation of the problem on Stephen W. Thomas’ blog.

Note that if you follow the BizTalk documentation on Deploying Pipeline Components, it is advised that the assembly should be deployed in both the GAC and the <installation directory>\Pipeline Components folder. I would advise to follow this way so it won’t matter how you or your developers create and compile the pipeline and their components; it will always work when the code is moved to the production environment and thus avoid unnecessary stress in the case it would have gone wrong because one developer referenced his component in a different way.

EDIT for BizTalk Server 2006 R2:
I have just notice after having published the post that the documentation has been changed for BizTalk Server 2006 R2 and that now the documentation says:

All the .NET pipeline component assemblies (native and custom) must be located in the \Pipeline Components folder to be executed by the server.

and:

You do not need to add a custom pipeline component to be used by the BizTalk Runtime to the Global Assembly Cache (GAC).

So now, Microsoft officially advises to put the custom pipeline component libraries only in the \Pipeline Components and not in the GAC anymore. If you still have BizTalk Server 2006 installed on your machine like I do, you will see that the local BizTalk documentation still says as mentioned in my original post.

Posted on

BizTalk Server 2006 Custom Functoid Documentation Mistake

When you develop a custom functoid, there are 5 important tasks to do:

  1. Create a resource file with the various resources such as functoid’s name, description, tooltip and icon.
  2. Create the class that will implement the functoid. This class must derive from the Microsoft.BizTalk.BaseFunctoids.BaseFunctoid class.The constructor must override the bas class constructor and call a certain number of methods and properties so that the custom functoid can run properly at run-time as well as integrate well into the Visual Studio’s mapper.
  3. Create a member method in the functoid class that will actually implement the functionality of the custom functoid (its business logic).
  4. Compile and sign the functoid project into an assembly with a strong name key file (so that the assembly containing the custom functoid can be deployed in the GAC).
  5. Copy the assembly to C:\Program Files\Microsoft BizTalk Server 2006\Developer Tools\Mapper Extensions and add the functoid in Visual Studio’s ToolBox. You must also install the assembly into the GAC so that the functoid is available to BizTalk at runtime.

Today I found out that the BizTalk Server 2006 documentation has a mistake in the custom functoid development section: Using BaseFunctoid, which relates to the point number 2 in my previous list of tasks.
That section says that the constructor of the functoid must make a call to the SetupResourceAssembly method to point to the resource assembly. It details:

Include a resource file with your project. If building with Visual Studio, the resource assembly must be ProjectName.ResourceName.

In my experience this is not true, when I build my custom functoid assembly in Visual Studio, I have to call the SetupResourceAssembly method with FunctoidNameSpace.ResourceName as parameter instead of ProjectName.ResourceName. If I follow the documentation’s advice, the resource information such as functoid’s name and icon do not appear in Visual Studio’s toolbox, proving that the resources could not be found by the IDE.

The only case that it would work as mentioned in the documentation would be if the project name and the namespace of the functoid were identical, which happens by default in Visual Studio. Indeed, when you create a new project with Visual Studio 2005, a default namespace will be created for the project, its value being the project name. You can see the default namespace value by right-clicking on your project files and click “properties”.

Posted on

BizTalk Expression Shape IntelliSense

Whenever you want to call IntelliSense from the Expression Shape editor, you can do so by just hitting the keys CTRL + SPACE. You will see the different namespaces, classes and objects (such as local variables and messages) available from the scope of your Expression Shape.

Note that this shortcut key combination actually comes from the Visual Studio .Net IDE. Thus, it is available for any “traditional” .Net project types (C#, VB.NET, Windows form, Web App…)

BizTalk Expression Shape IntelliSense

Posted on

HTTP Adapter and Apache Keep-Alive issue

Problem.

If you notice a high rate of suspended messages due to communication failures between your BizTalk Server and an HTTP server, you should verify if the HTTP Web server you are accessing is Apache, Resin or another similar server of the UNIX / LINUX / Java family. Note that this could apply to other web servers as well, these are just examples I know of.

BizTalk Server always sends HTTP request with the HTTP header setting Connection: Keep-Alive. Moreover, BizTalk expects that the foreign HTTP server obeys to the header setting and does not check the Connection header setting received in the HTTP response.
To be more precise, the problem is double:

  • Apache is not obeying to the Connection HTTP header setting and always closes the HTTP connection after every request (as in an HTTP 1.0 fashion). The HTTP/1.1 protocol allows this even if it is not the prefered behaviour.
  • BizTalk Server does not react accordingly to the Connection header that is returned back in the HTTP response. This is wrong, following the HTTP/1.1 protocol definition, the client should always check the Connection header returned.

So, on one side there is a Apache closing connections for every HTTP request and on the other side BizTalk is expecting connections to be kept open between requests.

Actually, the source of the problem is not that the Web Server is Apache or not but related to how the Web Server is configured and how it behaves with the connection. Nevertheless, in my experience, this kind of problem only happened with Apache. But I also ever encountered 1 Apache server which kept the HTTP connection open between requests and so no fix was required for that case as no communication problem arose.

For this reason, if you have communication problems between BizTalk and a foreign web server, you should first check if the web server is explicitly closing the HTTP connection after every request.

There is a Microsoft KB that explains the issue and also how to fix it.

Identification of the problem.

As mentioned in the KB, it is important to make sure that your problem is exactly what the KB refers to, so, I will explain how I diagnosed it on my BizTalk Server.

Looking in the BizTalk Administration Console, I saw an abnormal amount of suspended service instances, most of them being of the type “Message”. I also noticed that all the suspended messages belonged to only a subset of our BizTalk applications.
There was clearly something wrong and by checking the Windows Event Viewer, I saw that some BizTalk send ports had an unusual failure rate at the communication level. This explained why so many messages were suspended as messages failing to be sent through a port will be suspended.
The Windows Event Viewer showed messages such as:

A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: Cannot access a disposed object.
Object name: ‘System.Net.Sockets.NetworkStream’.

And

A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

After scratching my head on this issue for a while, I found out that all failing messages where those sent to a port using an HTTP adapter configured with an URI pointing to an Apache server. Other ports using the HTTP adapter but configured to access an IIS server were working fine!

I then used a network sniffer to see what was going on exactly and I saw something inconsistent in the HTTP headers:
The header of the HTTP request contained:
Connection: Keep-Alive
But the header of the HTTP response contained:
Connection: close
server: IBM_HTTP_Server/6.0.2.3 Apache/2.0.47

So, even if the HTTP request specifies “Connection: Keep-Alive”, the Apache Server still close the connection and replies with a “Connection: close” in the header of the HTTP response. For reference regarding the Connection HTTP header, see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10

Solution.

As it is not possible to change the Connection: Keep-Alive header setting in the HTTP adapter shipped with BizTalk Server 2006, I googled a bit and found out the KB 924638 I mentioned earlier in this article which provides 1 solution and 1 workaround for the problem.

The hotfix mentioned in the KB contains a library that makes it possible to modify the Keep-Alive configuration for the HTTP connection. However, as there is no option to set the header setting from the BizTalk Administration Console, it is required to create a custom pipeline which writes a KeepAlive property in the context of the message going through the pipeline. Later on, the HTTP adapter will read the KeepAlive message context property value and create the HTTP connection accordingly.

You can find here a zip file containing 2 Visual Studio projects, one for the Custom Pipeline Component and another one for the Custom Pipeline. In this custom pipeline, the KeepAlive property is not hard-coded and so its value can be set either from the BizTalk Administration Console at run-time or from Visual Studio at design-time.

To summarize, there are 3 steps to apply the solution:

  • Call Microsoft Support so that they email you the hotfix containing an executable which updates the Microsoft.BizTalk.HttpTransport.dll (as the hotfix is not directly available to download from the KB).
  • Create a custom pipeline which has a property that permits the user to specify the Keep-Alive setting.
  • Use the custom pipeline in the send ports that are pointing to web servers explicitly closing connections, such as some Apache web server.