Jump to content

Cre8asiteforums Internet Marketing
and Conversion Web Design


Photo

Restricting The Amount Of Bots?


  • Please log in to reply
6 replies to this topic

#1 lee.n3o

lee.n3o

    Cre8asite Tech News Reporter

  • 1000 Post Club
  • 1556 posts

Posted 22 June 2007 - 05:19 AM

I have a very large forum which always seems to be rammed with spiders/bots.... at times they out number the amount of real people on the forum by 3 to 1!! They are causing alot of extra strain on my database...

Now I still want the forum to be crawled alot, but can I restrict the amount of bots allowed on the forum at one time without annoying the SE's?? Or any negative effects?? If so would I do this with my robots.txt file??

Thanks

#2 lee.n3o

lee.n3o

    Cre8asite Tech News Reporter

  • 1000 Post Club
  • 1556 posts

Posted 22 June 2007 - 06:43 AM

Its actually nearly all Yahoo bots!! ... Theres bloody loads of em... I might disallow just them I think for the moment, as they are just going nuts...

#3 eKstreme

eKstreme

    Hall of Fame

  • 1000 Post Club
  • 3399 posts

Posted 22 June 2007 - 06:58 AM

Yeah, Yahoo!'s bots can really hit a website. I've seen that a few times.

Also, given the return in traffic, I'm contemplating blocking the Ask bots - why should they load the site if I'm not getting anything in return?

One thing for forums: if you have multiple display options (say a mobile version or low-image version), use robots.txt to block access to these alternates.

Pierre

#4 EGOL

EGOL

    Professor

  • Hall Of Fame
  • 5361 posts

Posted 22 June 2007 - 07:14 AM

I would ask the question... "Is this costing me anything?" If the answer is no then I would let them crawl. If the BW was costing significant extra money or if my site responsiveness was slogging then I would think about doing something... don't know what I would do because the bots are usually beneficial.

#5 lee.n3o

lee.n3o

    Cre8asite Tech News Reporter

  • 1000 Post Club
  • 1556 posts

Posted 22 June 2007 - 07:30 AM

Actually its ok... I hadn't set a memory limit on SQL server... The bugger was just taking everything it could... Everything seems to be ok now.

#6 Respree

Respree

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 5901 posts

Posted 22 June 2007 - 07:41 AM

This may be of interest.
http://help.yahoo.co...p/slurp-03.html

#7 lee.n3o

lee.n3o

    Cre8asite Tech News Reporter

  • 1000 Post Club
  • 1556 posts

Posted 22 June 2007 - 10:41 AM

hehehe thanks Respree... I did just that earlier.. Never even heard of the crawl delay before!



RSS Feed

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users