Attack of the Bots

For those of you that know much About Search Engine Optimization you will have heard of "Bots" - the automated programs that crawl the web and fill the search engines with the data they use when people search for things... over the last few days the server on which bosunsmate.org resides has been pummeled by thousands of these - to the point where it actually crashed the server several times!

Now, having lots of bots visit your web site is generally a good thing. Something, in fact, that many web designers dream About because it means that your site is "popular" and being placed well within the search engines. However when they start to descend on the server at a rate of 50-60 per second things aren´t quite as happy... You don´t then want to Block them because the bots actually help your rank - so... what to do???

What we did was to add a random delay to all our high traffic web sites IF the visitor was one of these bots (I have some fancy detection code for this) thus slowing down the site ONLY for the bots - from between 5 to 30 seconds. We also stop the display of advertisements to the bots on a few of the sites the have lots of ads. The result: well - the server is alive anyway ;) We also created a report that shows what server traffic is the result of bots - right now hovering around 21% of the total - and holding - even after we added the delay and the restriction to ads.

So all is well in the digital world... until the next catastrophe ;)

---