12 replies to this topic

[eKstreme]

Hot off the press: Introducing Robots-Nocontent for Page Sections.

As they explain it:

We heard what people were asking for so we did a little homework and are now happy to introduce the 'robots-nocontent' tag.

This tag is really about our crawler focusing on the main content of your page and targeting the right pages on your site for specific search queries.
...
To do this, webmasters can now mark parts of a page with a 'robots-nocontent' tag which will indicate to our crawler what parts of a page are unrelated to the main content and are only useful for visitors. We won't use the terms contained in these special tagged sections as information for finding the page or for the abstract in the search results.

There is also a help page.

So, let's have it. What do you think? I'm still not convinced - I need to think a bit more about this...

Pierre

[bragadocchio]

Sounds like the opposite of the Y!Q section tagging that you can do for yahoo!

Wonder why they felt this was necessary?

Example of Y!Q tagging:

<div class="yqcontext">

    <p>A fuel-efficient compact spacecraft has made it into lunar orbit,

        signaling Europe's first successful mission to the moon and putting 

        the inexpensive probe on course to study the lunar surface, 

        European Space Agency officials said Tuesday.</p>

    </div>

I do wonder about this:

<div class="robots-nocontent"> This is the navigational menu of the site and is common on all pages. It contains many terms and keywords not related to this site</div>

What are you doing building navigation for your site that contain keywords that aren't related to your site?

[JohnMu]

What happens when you accidentally break the closing tag or nest tags incorrectly?

I can understand section targeting with Adsense, where the Adsense bot only has a limited time to read a page, but for search the crawlers should be able to extract unique content of a page and ignore the site-wide navigation, etc. There must be more behind it :-).

John

[eKstreme]

I'm not happy that it's a pseudo-CSS class. This goes the against the spirit of not having structural markup (in a way). I really liked that the AdSense targeting was comments based - nothing to do with the markup.

And like John, I think there is more to this we haven't seen yet... integration with Panama, anyone?

Pierre

[JohnMu]

The Adsense comment-based marker is neat because it does not require that you have clean code. You can start it anywhere, end it anywhere and even exclude certain parts in between - you do not have to verify that you are closing all opened tags in between.

Some of the people blogging about it see MSN and Google following Yahoo's lead - what do you think?

I have a site that has a giant menu on top that is duplicated across all major pages. Just checking a site:-query on Google, Yahoo and MSN (ask.com won't index me, lol) I see very relevant snippets on Google and MSN - they ignore the duplicated menu and headers/footers. Yahoo however shows parts of the duplicated pages for almost every URL (however, to be complete, the site-query on Yahoo seems to also search for the URL in the page, which is included in the footer). Of course a site:-query does not really show how content from a page itself is extracted, but if the site:-query doesn't bring unique content from those pages, it is probable that the duplicated content is also used for other queries.

How well does Yahoo handle duplicated content within your pages in comparison to Google or MSN?

John

[yannis]

I need to think about this one, as well! Some good uses in the mean-time, here are some suggestions:

(01) You can post explicit content including pictures of your ex-mother in law as excluded content in a page full of tutorials about her lace-work, hoping she will one day find it!

(02) You can now stop scratching your head about three column layouts and various other methods where you pull the content first and the rest follows. As this is done in many instances to present content first to the robots, you can really and I mean really, simplify your CSS. This will also solve 67.25% of all IE compatibility problems.

(03) You do not have worry about how to write a robots.txt file all you have to do is use a class in the body of no-follow!


Yannis

[eKstreme]

What would happen if you add the class to the body tag? Is Y! doing any form of sanity checking?

Pierre

[yannis]

What would happen if you add the class to the body tag? Is Y! doing any form of sanity checking?

I am not sure if a sanity check is required as part of their employment contracts or not, but I am pretty certain they will skip reading the whole page!


Yannis

[JohnMu]

How far could this be a security issue with user-generated content? Imagine you can smuggle a <div class="that-class"> into a page ...

John

[eKstreme]

How far could this be a security issue with user-generated content? Imagine you can smuggle a <div class="that-class"> into a page ...

Surely you can argue the same for the AdSense comments? Mind you, I don't know which one is worse: a drop in revenue or a drop in SERPs. Neither is good!

Pierre

[JohnMu]

You're right - it doesn't make much sense . If a user can insert html tags like that, things are pretty messed up already, it won't matter much if they drop out of the Yahoo index as well...

John

[Ruud]

I'm not happy that it's a pseudo-CSS class. This goes the against the spirit of not having structural markup (in a way).

Yes and no. I think it is or would be nice if structural markup could be extended with attributes like header, footer, menu, main-content. A "not-main-content" is not too bad in that view.

Whether this thing will be useful... might begin to be if others pick it up. But I don't really see the value of it.

[yannis]

Yes and no. I think it is or would be nice if structural markup could be extended with attributes like header, footer, menu, main-content. A "not-main-content" is not too bad in that view.

A class or pseudo-class is not really structural mark-up!. It is not a tag. Only (X)HTML tags can be considered that they define structure. CSS classes are just names for structural elements.

The idea of extending CSS for other uses is not new. With them you can effectively define a 'pseudo-language' if you wish. But so can you with mark-up commented out (see how templates are defined by Dreamweaver for example).

I would welcome robots trained to recognize more than microformats, using CSS. If the Wikipedia robots can roam the web one could use category mark-ups as used in the Wikipedia itself, or Technorati tags. Robots need human assistance and with the threat of having a website banned, might not bring the problems of the early days of the web and META tags, I mentioned previously.

Regards

Yannis