From last few months, I have been seeing Microsoft`s MSN Bots all over my website and that I was being hit with massive amounts of allotted bandwidth (Last Check – 800MB in 3 Days).

Soon i realized that my logs were actually being spammed by Microsoft.

Apparently all of the traffic was originating from 65.55.* IP Block. Doing an IP whois on the addresses returns this data:

OrgName:    Microsoft Corp

OrgID:      MSFT

Address:    One Microsoft Way

City:       Redmond

StateProv:  WA

PostalCode: 98052

Country:    US

NetRange:   65.52.0.0 – 65.55.255.255

CIDR:       65.52.0.0/14

NetName:    MICROSOFT-1BLK

NetHandle:  NET-65-52-0-0-1

Parent:     NET-65-0-0-0-0

In each case, the IP of the visitor was owned by Microsoft.

As I looked into it, I found a thread on many forums discussing the same thing had been started way long back. When asked about it, MSN Dude (who apparently at one time anyways was verified as working for MSN) gave this as his reasoning for the behaviour:

The traffic you are seeing is part of a quality check we run on selected pages. While we work on addressing your concerns, we would request that you do not actively block the IP addresses used by this quality check; blocking these IP addresses could prevent your site from being included in the Live Search index.

Makes sense… filling our traffic logs with bullshit queries, putting extra loads on our servers, and generally acting irrationally sounds like just the kind of “quality check” most websites need, and as we all know threatening to ban our sites from your search engine if we don’t let you do whatever the hell you want is always the way to go.

Annoying as all that is, however, it is still nothing compared to the real clincher here. This bot sent by Microsoft to attack our innocent websites isn’t just downloading the HTML pages… it’s also downloading CSS files and JavaScript, including downloading AdSense code from Google.

For now i have disallowed Msnbot using robots.txt file, Lets see if it obeys robots.txt standards and leave my site alone.

I would be very interested in hearing other peoples thoughts, opinions, and experience with this, if they have any.