To catch a thief

jemima and I have been on similar quests lately -- she's trying to stymie the thieves who willy-nilly link to your webspace and eat up the bandwidth, while I'm trying to figure out how to get false referrers out of my weblogs. As I said to jemima, it's no fair that I'm getting over 60 hits from webdevboard.com* and yet NO feedback. Turns out, after going undercover on the message board, I figured out what the deal was -- webdevboard sends out its spider for nefarious purposes; they say it's free service to track down broken links and then to email the site owner. I say they're spamming my weblogs and I want them to stop eating my bandwidth already.

So I did some research on the topic. I already looked into the image stealing problem some time back and made some adjustments to my .htacess file to prevent any image stealing. (Before the modification, I did replace the stolen graphic with another graphic that read 'I stole bandwidth from seema.org and all I got was this lousy image' but I was too chicken to leave it up for more than a few minutes). But this spamming spider is a new beastie to contend with. I am grateful for one thing -- I'd often wondered how certain sites were showing in my logs as referrers and now I have a better idea of how that might be the case.

So, if you have a spammity spider problem, here are some links. Most of the 'good' spiders like google, yahoo, etc., will respect your robots.txt. It's the bad spiders like webdevboard's that don't. Here are some suggestions on how to keep the bad spiders away and to modify your .htaccess file accordingly. If you need to know who the spiders are, check here. If you don't have a robots.txt at all, get started here. And finally, more information on the uses for .htaccess files can be located here.

Two Olympics links today: Geek Trivia: Olympic muddle winners and US women take the silver in gymnastics.

* Yes, I didn't link to webdevboard. com on purpose because they are Evil (tm) and do not deserve a direct link. Unfortunately, I had to give them the extra publicity, but those spammity spammers deserve no referrer from me!