9 replies to this topic

[JohnMu]

I'd love to hear your thoughts on this one

I pushed my small server into publishing the AOL search database (several GB of data) and set up an interface to query it. All queries are done through parameters in the URL. All database query results are cached since the queries can take a bit of time (usually around 10 seconds, sometimes up to a minute). The result pages are filled with links to other queries so that users can "click and play" (and wait ).

If a search engine crawler were to stumble upon those pages, it could go crazy following links and pushing my server into a "meltdown" .

I currently have the following set up to avoid that, but I'm not sure if it's the best way:

- "bad bots" (recognized by user-agent, IP or referrer) are treated to a 404 (with a small captcha-like field to authenticate real users and allow them access)

- "good bots" (+the Mediabot) are allowed access and shown the cached data.

- When "good bots" try to access pages which are not yet cached (not visited by a human visitor), they are treated to a 500 (server error; again with a captcha-like element for real users)

My main question is: will the "good bots" treat the site differently if they are shown error code "500" for all the non-cached URLs they find? So far, I've noticed that the Mediabot will keep re-trying the URL (with a longer wait in between) -- the Mediabot isn't that much of a problem, since it only accesses pages which were opened anyway (except for a small bug in their URL parsing ). Could the search engines assume that the server went offline? Or will they reduce the crawl-frequency in general for the whole site (eg should I instead move the DB to a separate domain)? Or should I remove all non-cached URLs as links from the pages when a search engine crawler comes by (that would take a bit of processing power, but if it's worth it....)?

John

[eKstreme]

Why don't you have a special search script that is blocked by robots.txt? This search script will handle all quries for you.

The other thing: add a captcha to the searches themselves. This will trap spammers too

Pierre

[JohnMu]

Why don't you have a special search script that is blocked by robots.txt? This search script will handle all quries for you.

Because I don't mind if the (good) bots read my cached pages. It's just that I don't want them to crawl the whole database and block access for the "real" users.

Adding a captcha to all queries would be a possiblity, but it would be hard to get it to work "right" (I think). I suppose it could work for the whole session, letting the user enter the captcha once (and only if real database queries are being done), hmm...

John

[eKstreme]

Yes, but that doesn't stop the scrapers! The only effective block against scrapers who want to use your database is a strong captcha. Imagine having every bad bot out there trying to find out about viagra + friends from your database... ooh fun.

Case in point: one site of mine has a guestbook. Spammers loved it for two reasons: easy to post to to get a link from a good domain, easy to scrape for emails and more links to scrape. End result? It became the most visited page on the site at one point!

So I added a captcha to even read the guestbook. That cut requests by more than 80% (I'm not kidding!) I still get spammers visiting, but I know they won't come back.

Funny thing is: I was once reading through the log files, and noticed a typical scraper pattern for one of the IP addresses. Then, the captcha was submitted a few minutes later, and the same scraping pattern started. It seems to me that the spammer personally visited the guestbook to manually kick-start any session-based captchas, but got thwarted. It made me happy

So, again, given how straining your database is on your server, I would put a captcha for every search. I know it's inconvenient, but a good explanation would go a long way to alleviating any pain your visitors might feel. Everyone understands spam and everyone hates it. They'll understand.

Pierre

[yannis]

John

The cached pages are an 'adsense' golmine waiting to be mined. So start thinking that the cache is good it will grow and as it grows it will take less and less time for real queries. (You will need plenty storage space!).

The problem is as follows:

1. Let real users through - This is not difficult with a little turing test.

2. Let good bots in - no problem just exlude the bad bots!

3. How to exclude bad bots

"Bad bots" (recognized by user-agent, IP or referrer) are treated to a 404 (with a small captcha-like field to authenticate real users and allow them access).

Agreed this is a good strategy.

When "good bots" try to access pages which are not yet cached (not visited by a human visitor), they are treated to a 500 (server error; again with a captcha-like element for real users).

I do not think the search engines will like this. Rather send them to a page where the query is automatically phrased using php, the page created on the fly and cached. This way also the bots will create all the cache you need! Or cross check all the links and create the full cache yourself via your own 'bot'.

Goldmine John do not let it go!


Yannis

Edited by yannis, 17 August 2006 - 07:58 AM.

[JohnMu]

Just a short feedback:

The correct code would be 503 (network / server unavailable)

The error 500 seems to have scared the Googlebot off for a few days! Even the Mediabot is not visiting my site anymore and certainly NOT targeting those database entries. I have even changed paths in the URL and reduced it from 4 parameters to 2, moving the real query to the end. Adsense is partially targeting based on the URL, but the Mediabot is not visiting the pages to see what is to be found. Perhaps this is just for a short time? Perhaps there is some sort of "OMG - the server is dead" timeout (several days?) on a whole bunch of 500 errors.

The Googlebot has currently returned, crawling 1-5 pages/hour from the database, the 503 errors don't seem to bother it much, it keeps on anyway. It's nowhere near the usual crawl-frequency the site had, however. I bet it's adjusting the crawl speed based on the errors.

The Mediabot is, like I said, not coming at all anymore. It's visiting the other URLs in my site as visitors come, just not the AOL database ones. The funny thing is that I can provoke the Mediabot to come for a visit with the "right" parameters attached to the end of the URL (I've been playing with that and have sent some mails to Google about it). It just doesn't visit the natural URLs in the AOL database section of the site... Hmm... (or maybe the server is cacheing the accesses and not passing them through to the application? that would be neat -- off to check the server logs tonight, otherwise off to install a packet sniffer later on).

John

[JohnMu]

Feedback #2: Google has been visiting those pages - my mistake: I set it up to do server side caching (with asp.net - it's not perfect, but - surprise - it works). With the server side caching it would not trigger my logging functions in the code, instead just logging to the server logs for the domain. It sure surprised me, I never got around to testing it . Adsense targets after a little bit.

John :embarrassed: :embarrassed:

[Black_Knight]

Personally, I'd pick a few select (i.e. popular and thus frequently or perma-cached) searches and give them a rewrittten (no parameters) URL. I'd use the robots.txt to block the spiders from all actual search URLs with parameters, probably by using a directory that I omit from the rewritten URLs.

[JohnMu]

Regarding the bot control, I think it's working and it was fun to figure it out

But regarding the "value" of it all for the search engines, I think you're probably right, Ammon. In the end, a compiled table of lots of keywords is not really worth that much -- sure visitors might find it "on accident" when searching, but it's not like they'll find what they're looking for in a database like that (I doubt people searching for "myspace" will want to see what other people who also searched for "myspace" went to look at ). I have currently optimized the spidered-page quite a bit, using javascript to display lots of things that spiders shouldn't spot (like my links out to the clicked URLs). I'm still wondering how it would be taken up by the spiders (and how the engines themselves throttle their crawlers when they get stopped on every non-cached query - or not).

Once I get the time, I'll move it to a separate domain (or subdomain) and try to figure out a way to provide value on the database pages itself: perhaps allow people to comment on them, rate them, link out to other articles, etc.

What I also want to work out is how to provide a way for people to submit "statistics" that will be run (and cached) in the off-hours. I'm sure that given the chance, people will come up with some things I'd never think about and it would be great to offer them a way to do that without having to import all that data (and of course great to get links from their postings about their findings ).

Why can't the day have a bunch of extra hours?

John

[eKstreme]

But regarding the "value" of it all for the search engines, I think you're probably right, Ammon. In the end, a compiled table of lots of keywords is not really worth that much -- sure visitors might find it "on accident" when searching, but it's not like they'll find what they're looking for in a database like that (I doubt people searching for "myspace" will want to see what other people who also searched for "myspace" went to look at ).

Depends on what you're trying to do. Let's take your myspace example: suppose you have a page on your site about myspace, or even , a series of pages each one discussing some aspect of myspace. You can (and the gray/black hat in me says you should ) add a secondary page, say on a blog or a tool's output (your case) that lists all the keywords related to myspace. One idea would be to use the Overture keywords data or the AOL data in your case.

On a good domain name, the secondary page will rank for pretty much all the keywords in the list, for some being higher than the others. The end result is a very wide funnel that you can use to capture traffic to your primary pages.

So, if you think about it this way, you might have created the ultimate traffic capturing code!

As a start, I would cache the top keyword searches you have done, link them up to some articles on the subject, throw in some AdSense, and you get some money off it. Even better, get some people to write some unique articles for you in exchange for a link.

My 2-darker-than-white cents.

Pierre

PS - I'm talking out of experience on this one