Jump to content

Cre8asiteforums

Discussing Web Design & Marketing Since 1998

  • Announcements

    • cre8pc

      Thank you! Cre8asiteforums 1998 - 2018   01/18/2018

      Internet Marketing Ninjas released many of the online forums they had acquired, such as WebmasterWorld, SEOChat, several DevShed properties and these forums back to their founders. You will notice a new user interface for Cre8asiteforums, the software was upgraded, and it was moved to a new server. Thank you for your support as we turn 20 years old.  
Sign in to follow this  
lee.n3o

Restricting The Amount Of Bots?

Recommended Posts

I have a very large forum which always seems to be rammed with spiders/bots.... at times they out number the amount of real people on the forum by 3 to 1!! They are causing alot of extra strain on my database...

 

Now I still want the forum to be crawled alot, but can I restrict the amount of bots allowed on the forum at one time without annoying the SE's?? Or any negative effects?? If so would I do this with my robots.txt file??

 

Thanks

Share this post


Link to post
Share on other sites

Its actually nearly all Yahoo bots!! ... Theres bloody loads of em... I might disallow just them I think for the moment, as they are just going nuts...

Share this post


Link to post
Share on other sites

Yeah, Yahoo!'s bots can really hit a website. I've seen that a few times.

 

Also, given the return in traffic, I'm contemplating blocking the Ask bots - why should they load the site if I'm not getting anything in return?

 

One thing for forums: if you have multiple display options (say a mobile version or low-image version), use robots.txt to block access to these alternates.

 

Pierre

Share this post


Link to post
Share on other sites

I would ask the question... "Is this costing me anything?" If the answer is no then I would let them crawl. If the BW was costing significant extra money or if my site responsiveness was slogging then I would think about doing something... don't know what I would do because the bots are usually beneficial.

Share this post


Link to post
Share on other sites

Actually its ok... I hadn't set a memory limit on SQL server... The bugger was just taking everything it could... Everything seems to be ok now.

Share this post


Link to post
Share on other sites

hehehe thanks Respree... I did just that earlier.. Never even heard of the crawl delay before!

Share this post


Link to post
Share on other sites
Sign in to follow this  

×