Restricting The Amount Of Bots?
Posted 22 June 2007 - 05:19 AM
Now I still want the forum to be crawled alot, but can I restrict the amount of bots allowed on the forum at one time without annoying the SE's?? Or any negative effects?? If so would I do this with my robots.txt file??
Posted 22 June 2007 - 06:43 AM
Posted 22 June 2007 - 06:58 AM
Also, given the return in traffic, I'm contemplating blocking the Ask bots - why should they load the site if I'm not getting anything in return?
One thing for forums: if you have multiple display options (say a mobile version or low-image version), use robots.txt to block access to these alternates.
Posted 22 June 2007 - 07:14 AM
Posted 22 June 2007 - 07:30 AM
Posted 22 June 2007 - 10:41 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users