IIS 6 gives 403 forbidden when using W3C link checker

[Katt]

I recently moved several websites to a new server running IIS 6 and then ran W3's link checker on the sites and it gave me the following on every page:

The link is forbidden! This needs fixing. Usual suspects: a missing index.html or Overview.html, or a missing ACL.

Checking the IIS logs I see this:

2009-10-16 15:08:13 W3SVC156278702 192.168.100.17 HEAD /west/west.shtml - 80 - 128.30.52.70 W3C-checklink/4.5+[4.160]+libwww-perl/5.823 403 1 0

The pages giving these errors are all working properly with GET, and display properly within a browser, but I'm worried that these HEAD requests being denied may cause problems with spiders. What can I do?

[wige]

I see two possibilities. The server may be denying HEAD requests for some reason, but this is unlikely, as HEAD requests are often used to check if a cached version is up to date.

Personally, I think it is more likely that the test utility is being blacklisted. If you look at the user agent in the log entry you show, it contains "libwww" which is a library used by a wide variety of spam bots, and a lot of servers will block any user agent that contains that string.

You can check with your hosting provider to get a specific reason why the request is being denied and to find out if there is a workaround available, but it is possible your hosting provider will not allow the scans.

[Katt]

Thanks for the response wige! This is a dedicated server at rackspace... I can do what I want with it but I am not well versed in IIS. I could ask them to take a look but I really want to learn more about IIS myself. I searched every way I could think of but couldn't find the same scenario. I did read somewhere that IIS 6 blocks HEAD requests by default, but couldn't find out where or how to change this. I had an IIS 5 server at rackspace before and it did not behave this way.

Thanks again... any further suggestions?

[wige]

Check this article from Microsoft, it covers the Lockdown Wizard which offers some security override and troubleshooting tools: How to install and use the IIS Lockdown Wizard Unfortunately I don't work with IIS too often so there is not much else I can suggest beyond this.