Del.icio.us Bans Search Engine Spiders Options

[AbleReach]

del.icio.us Bans Search Engine Spiders - from Colin Cochrane

It appears that within the past 2-3 days the popular social book-marking site del.icio.us has started blocking the major search engine spiders from crawling their site. This isn't a simple robots.txt exclusion, but rather a 404 response that is now being served based on the requesting User-Agent.

While I was doing some Photoshop work for a site of mine tonight I needed to grab some custom shapes to use to make some icons. I recalled having bookmarked a good resource for custom shapes in del.icio.us, but after searching my bookmarks using my del.icio.us add-in for Firefix, I couldn't find it, so I pulled up my browser and went to my profile page on del.icio.us to do a search...

So, what do we think? Has anyone else noticed this kind of SE behavior?

[eKstreme]

A 404 is not the best way to block SE crawlers - 403 is better.

I'm wondering, is delicious trying to implement a bot authenticating scheme and it's going wrong?

Whatever it is, I'm not surprised. On Yahoo! search last week, there were JS errors that pop-up alerts (message boxes) that say something or the other is broken. Also, the rds.yahoo.com redirector that all SERP results go through would time out once in a while. Sounds to me that Y! is not doing lots of good quality control. Very sloppy of them, especially given their situation.

Pierre

[EGOL]

Rant !

I question the competence of Yahoo's IT department. They have lots of bugs in their other properties and when paying customers - who are very competent - report an error, documenting it for them in great detail, they are blown off with a form letter - about one week after the problem is reported. Multiple reports of the problem results in more run-arounds. IT responds only after strong complaints are made to sales. This is a "head where the sun don't shine" blunder done by someone on their staff who does not understand search engines.

Now for a slightly more rational reply... They are probably under very heavy attack by link grubbers who are bookmarking pages by the hundreds manually or by the thousands automatically. So, their approach is to block the indexing of their site to eliminate any possible search ranking value for links within their site. Instead of announcing it, they do it quietly - which is a mistake because it will make them look sleazy rather than forthright. It also will not solve their problem because the people who spend a lot of time or money to grub delicious links will not realize that those links have been devalued and continue their practice of bookmarking every page that they (or their clients) own.

Now what I really think... This makes about as much sense as changing the title tag of every site that pays to be included in the Yahoo Directory to the name of the company that owns the site.

Don't bet any money that you can't afford to lose on Yahoo!

[bwelford]

Yes, it seems to be true. Changing my user agent to Googlebot and going to my del.icio.us web page gives me a 404 error. As all have said, this may be a sign of a debilitating decline for del.icio.us and Yahoo! is in no position to invest massively in a property that has uncertain monetization.

[bobbb]

Wonder if they banned Slurp??

[bwelford]

Sebastian on Sphinn suggests this is nothing new.

[bragadocchio]

If I had to guess, I would go with the "yahoo IT department messed something up" scenerio rather than anything else.

It seems like they are in a constant state of flux and turmoil these days, but I've always wondered about many of the decisions that they have made in the past.

I'm going to try to remember to export all of my bookmarks there later tonight.

[EGOL]

They might be trying to sabotage the ship before MSN boards.

[bragadocchio]

I'm not sure of the synergy between Yahoo and Microsoft.

I suspect that it would end up being a disaster.

Yahoo needs a delicate and intelligent touch; not a maniacal one.

[iamlost]

Yahoo needs a delicate and intelligent touch; not a maniacal one.

There is nothing wrong with Yahoo that a few thrown chairs and purple faced tantrums can't fix.

[bragadocchio]

You may be right, iamlost

The new folks from Microsoft might end up looking more like this:



Of course, that can't do the real thing (video) justice.

[eKstreme]

They might be trying to sabotage the ship before MSN boards.

If that mentality holds true, it'll be the reason why Yahoo! is in such a mess: the employees are not dedicated to the company or its products. If they believe they are the best, and continue to fight to be the best, they wouldn't be a buy-out target, they wouldn't be losing traffic, and they certainly wouldn't be killing the babies they're supposed to be nurturing.

Whether that mentality is there or not is another question. I doubt it but wouldn't be surprised if it were true.

Pierre

[storyspinner]

I'm thinking with Bill on this - a mistake.

But, again - you have to stop and think about Social Bookmarking. If you are utilizing any social media for links - that's just crazy. 95% of social media sites have nofollow - Delicious is no different here - and the other 5% have no page rank to pass anyway (usually too new or just spammy sites).

Whether or not Delicious blocks the crawlers (personally,its stoopid on their part since it's a YAHOO company) isn't the most important thing. Declicious is about sharing with others - people who use delicious still will use delicious whether its in the index or not, they will still click on links and come to bookmarked sites.

If you look at it from and SEO perspective, sure you think "are they insane"? It is nutz to take this extreme action - especially if you want your content found.

However, when I stop and I look at this from a purely social media standpoint - I come up with the question "so what" when it comes to blocking crawlers. As a social media site, it's about "sharing" with others, not with sharing with the search engines.

[yannis]

There can be two logical explanations:

(01) This was done on purpose either with or without management approval

or

(02) This was an error on somebody's part.

My assertion this belongs to (01) and I have a different explanation. Any successful bookmarking site is a resource a 'corpus of user vetted links' if you like. They can be used to 'train' algorithms to detect such things as 'popularity', 'categorization', 'freshness' and many more markers that can help with both ranking as well as indexing. This can be achieved either there is a rel=nofollow or not. Search engines if they need to compete in the long run they need to understand content better.

PageRank does not do that. It only allowed webmasters to vet other sites by linking to them. Something similar to what social websites do now! Except the vetting is open to anyone. There is more. If a link has been bookmarked on a social website, there is a good chance that spiders can find summaries of the page in question in posts etc and use it to understand content better.

(And assuming a world in the no distant future where most links will be no-follow, we can kiss PageRank good-bye).

There is definitely something into this. Thanks for highlighting it Elizabeth.

Yannis