IIS 503 Service Unavailable Problem (One of many)

-------------------------------------------------------
Goto blog home
Visit my new blog dedicated to Internet of Things, Embedded Programming & Automation
-------------------------------------------------------
The 503 problem can occur due to a variety of reasons. Literature on diferent scenarios is available at the link below. The focus of this article is an issue that occurred with incorrect installation of McAfee Host Intrusion Prevention (disclaimer: this was not a problem with the McAfee software, but is attributed to the way it was installed on the server).
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/55f71614-ef1b-4015-b9c8-a42c1e700c25.mspx?mfr=true

Once you face the 503 issue, the first step is to find out the root cause. First find the actual error logged in the HTTP Error log available on the server path (default location and filename is configurable):
%windir%\System32\LogFiles\HTTPERR

A sample entry is provided below (the IP addresses have been masked for the purpose of my client's privacy):
2009-10-28 16:57:23 10.x.x.xxx 1590 10.xx.xx.xxx 80 HTTP/1.1 GET /testsite/default.aspx?Op=All 503 1 AppOffline TestSiteAppPool

The second last column value 'AppOffline' is of interest as it will tell you exactly why this error occurred. The exact meaning of this word can be found at
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/55f71614-ef1b-4015-b9c8-a42c1e700c25.mspx?mfr=true

For example, 'AppOffline' means the application pool has been put into Rapid Fail Protection and has been disabled automatically. At this point we have pined down the actual issue. In my case this was not sufficient so I looked at the error logs from event viewer and found the following entry:
The HTTP Filter DLL C:\Program Files\McAfee\Host Intrusion Prevention\eng\isapi\IsapiStub.dll failed to load. The data is the error. Data Bytes 7e 00 00 00

Next I checked and found that the above ISAPI filter did not exist. The IIS metabase got modified to route incoming requests to a filter in the Host Intrusion Prevention directory. But the filter was not present on the server due to improper installation performed by our server admin group. We got the Host Intrusion Prevention removed from the server by our server admins and got them to correct their automated installation scripts.