6 replies to this topic

[lee.n3o]

I have a very large forum which always seems to be rammed with spiders/bots.... at times they out number the amount of real people on the forum by 3 to 1!! They are causing alot of extra strain on my database...

Now I still want the forum to be crawled alot, but can I restrict the amount of bots allowed on the forum at one time without annoying the SE's?? Or any negative effects?? If so would I do this with my robots.txt file??

Thanks

[lee.n3o]

Its actually nearly all Yahoo bots!! ... Theres bloody loads of em... I might disallow just them I think for the moment, as they are just going nuts...

[eKstreme]

Yeah, Yahoo!'s bots can really hit a website. I've seen that a few times.

Also, given the return in traffic, I'm contemplating blocking the Ask bots - why should they load the site if I'm not getting anything in return?

One thing for forums: if you have multiple display options (say a mobile version or low-image version), use robots.txt to block access to these alternates.

Pierre

[EGOL]

I would ask the question... "Is this costing me anything?" If the answer is no then I would let them crawl. If the BW was costing significant extra money or if my site responsiveness was slogging then I would think about doing something... don't know what I would do because the bots are usually beneficial.

[lee.n3o]

Actually its ok... I hadn't set a memory limit on SQL server... The bugger was just taking everything it could... Everything seems to be ok now.

[Respree]

This may be of interest.
http://help.yahoo.co...p/slurp-03.html

[lee.n3o]

hehehe thanks Respree... I did just that earlier.. Never even heard of the crawl delay before!