Bad Scooter! No Biscuit!

I was perusing my weblogs yesterday when I saw the familiar sight of Scooter, the spidering engine of AltaVista (no, Scott, not you). Scooter pulled down my robots.txt file, as all good spiders should. But then it did something quite unexpected: it followed links that I had specifically prohibited it from following in my robots.txt file!

This file has been used on my site for years. Its purpose is really to cut down on spammers from grabbing my email addresses through search engines. Whenever I see a search engine not obeying the robots.txt, it has a pretty good chance of getting banned from my site.

I’ve never known Altavista to be rogue player, so I emailed their support staff demanding an explaination. They then asked me for logfiles, which I provided them. As of yet, the ball is still in their court. I demanded an answer from them as to why they would break the rules. They certainly got some ‘spaining to do.

I’ll update this story when I get a response back from Altavista. In the meantime, you might want to check your logfiles, too, to see if you’re also a victim.