Looks like a bot has been scouring my website without properly identifying itself. I noticed that my older posts were getting a lot of unexplained hits. I checked the logs, looked up the IPs, and discovered the visitors were bots from the rankcrawler.com domain. The bots don’t properly identify themselves in their user agent field, as good bots should do:
Some of the bots came from these IPs (though there may be others):
87.98.249.75
87.98.133.249
91.121.26.45
94.23.152.34
94.23.153.8
As you can see, Rankcrawler prefers to disguise itself as a regular browser. This is a no-no.
87.98.249.75 – – [29/May/2009:23:56:09 -0400] “GET /page/2/ HTTP/1.0” 200 34160 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6”
87.98.249.75 – – [30/May/2009:00:11:16 -0400] “GET /2006/07/ HTTP/1.0” 200 41171 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6”
91.121.26.45 – – [29/May/2009:20:47:22 -0400] “GET / HTTP/1.0” 200 34467 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”
91.121.26.45 – – [30/May/2009:00:01:23 -0400] “GET /2008/05/ HTTP/1.0” 200 27858 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”
91.121.26.45 – – [30/May/2009:12:36:00 -0400] “GET /2005/11/24/happy-thanksgiving/comment-page-1/ HTTP/1.0” 200 4699 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”
91.121.26.45 – – [30/May/2009:12:36:26 -0400] “GET /2005/11/21/google-rtp/comment-page-1/ HTTP/1.0” 200 11139 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”
94.23.152.34 – – [29/May/2009:23:57:59 -0400] “GET /2009/05/28/metered-bandwidth-what-about-metered-tv/ HTTP/1.0” 200 13604 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.18) Gecko/20081107 Firefox/2.0.0.18”
94.23.152.34 – – [30/May/2009:00:03:11 -0400] “GET /2009/05/24/zydecopious/ HTTP/1.0” 200 5652 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.18) Gecko/20081107 Firefox/2.0.0.18”
… and again:
94.23.153.8 – – [30/May/2009:00:55:09 -0400] “GET /2004/11/ HTTP/1.0” 200 27697 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
94.23.153.8 – – [30/May/2009:00:56:49 -0400] “GET /2006/05/ HTTP/1.0” 200 31919 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
94.23.153.8 – – [30/May/2009:01:03:11 -0400] “GET /2008/12/ HTTP/1.0” 200 29715 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
The rankcrawler.com domain name is a new one, registered on 8 May. Here’s the whois record for it:
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.domain: rankcrawler.com
owner: – –
organization: RankCrawler
email: florent75@gmail.com
address: 3 Northbrook Road
city: London
state: —
postal-code: SE13 5QT
country: GB
phone: +44.7913656191
admin-c: CCOM-1426824 florent75@gmail.com
tech-c: CCOM-1426824 florent75@gmail.com
billing-c: CCOM-1426824 florent75@gmail.com
nserver: a.ns.joker.com 207.44.185.100
nserver: b.ns.joker.com 66.197.237.21
nserver: c.ns.joker.com 207.44.185.10
status: lock
created: 2009-05-08 10:51:21 UTC
modified: 2009-05-08 10:51:21 UTC
expires: 2010-05-08 10:51:21 UTCcontact-hdl: CCOM-1426824
person: – –
organization: RankCrawler
email: florent75@gmail.com
address: 3 Northbrook Road
city: London
state: —
postal-code: SE13 5QT
country: GB
phone: +44.7913656191source: joker.com live whois service
query-time: 0.012858
db-updated: 2009-05-30 16:54:41
They’d better get a clue soon if they hope to stay in business.
[Update: 1 June 2009] RankCrawler has seen the error of its ways and will soon properly identify itself.