4.11.2008

Man. Yahoo's bot... slurp... is killing me. Their bots have been pounding my site for weeks, sucking down lots of bandwidth.

I decided to make a robots.txt and try to disallow crawling on some of the more byte intense portions of the site. But, after just watching my log for a few minutes, I could see yahoo slurp get the new robots.txt, and completely ignore it! Dangit!

That is annoying. Apparently, I'm not alone. Cmon yahoo... get your stuff together.

No comments: