I received an email this evening from Philippe Martin at RankCrawler, apologizing for the bad bot behavior:
Dear Mark Turner,
I apologize for not properly identifying our crawler (RankCrawler) by using the user agent. Our reverse-dns go to rankcrawler.com but we don’t use our own user agent. We will fix this problem soon. We have stopped to crawl your website as soon as I read your message.
We DO NOT crawl with the IP 94.23.51.159 as you claim in your second blog post about Rancrawler. It should be another company that we don’t know and that uses the same ISP (OVH is a very large ISP). We uses at this time only 5 IP that goes to rankcrawler.com.
I apologize again for this problem and I hope you will let our crawler access your website once we properly identify our crawler with our own user agent.
Thank you for your message,
Philippe Martin
http://rancrawler.com
I’m pleased that Mr. Martin chose to respond to my complaint and as such, I will allow RankCrawler to access MT.Net once again.