Reasons For ‘supplemental Result’? Options

[billshorters]

What are all the possible reasons for a page to have ‘Supplemental Result’ next to the url in the Google search results?

[Wit]

• pages that haven't been updated for a long time (i.e. many many months)
• pages that have lost most of their backlinks or that don't pass the "minimum number of backlinks to get indexed" test any longer (this one is graded on a curve LOL)
• pages that have lost ALL of their backlinks (read: orphaned pages)

PS: I'm not gonna speculate about "manual" editing of the G index...

[Black_Knight]

Or if anyone needs a more 'basic' answer, that helps them understand "what does supplemental result mean?" then what causes a result to have "supplemental" next to it is that page not being in the main index, but instead only being known of through links to it from pages that are in the index.

Supplemental usually means "Google knows of this URL, but has not spidered the document recently for it to be in the main index".

[Ruud]

An old explanation was that it had to do with rare queries? And that supplementals were shown as a "filler" if the query didn't return (enough) results from the main index?

But to be honest, the supplemental index baffles me a bit.

One thing I can't figure out for example is traffic. If a document receives traffic with some staying power equal to the time on page of the rest of the site, why would it remain supplemental?

After Big Daddy was said to be complete I tried making a document supplemental - and can't. It has no incoming links save two to get it spidered.

[Black_Knight]

I tried making a document supplemental - and can't. It has no incoming links save two to get it spidered.

To go supplemental, Ruud, it must not get spidered.

A supplemental page is a page that was NOT spidered, but is known about from links on pages that were. The growth of supplemental pages is due to the changes in crawling priority. Lower the crawl priority of a page enough and it will go supplemental, but its a tricky balance.

[rynert]

Ammon, is that a page that was NOT spidered EVER, or just within a recent timeframe?

[Wit]

Not spidered lately.

Or else it would just show up as URL-only (i.e. without title and description "snippet")

For some reason, I've never seen the words "supplemental result" on one of those, but that doesn't necessarily mean that these results are pulled from the "main" index rather than from the "supplemental" one....

Guess that doesn't really matter, because url-only results are most often the pages that we blocked using robots.txt, but that somehow got picked up by Gbot anyway. We tried to keep them off the serps, so we can't be bothered with them being supplemental or not, right?

[JohnMu]

Could the supplemental index be seen as a parallel document index (with hit-lists, forward and inverted indicies) (using "backrub" as a structure)? Assuming documents with slightly insufficient value (for the current crawl/index-threshold) are moved into the supplemental index first, Google could save a lot of bandwidth by just restoring them once the URLs have more value (versus throwing them out and having to re-crawl everything to find those URLs).

This is particularly important when links are re-evaluated or thresholds are adjusted: it's not just the links to that site (the one "going supplemental") but also to a very large extent the links to the links (to the links, etc) that matter. By de-valuating a few major value-passing links, many full sites downstream from there could be effected. Those sites (not just single pages!) will be in a position where they used to be fully indexed (perhaps with 1000's of URLs), but now do not have enough value to merit being indexed anymore. Google could assume that those sites will "fight" for their value and work on getting more good links in the future. By keeping the old (insufficiently valued) URLs warm in a supplemental index, Google can restore the sites complete indexing as soon as enough valuable links are found. It saves a lot of bandwidth and time.

The same could be applied if a site loses value through penalties (or of course if one of the major upstream sites loses value through penalties).

I wonder how many sites (and links of links of those sites) indexing is dependant on the wikipedia or DMOZ (+clones). Imagine if Google were to devaluate those two sources (and their clones)... didn't they just do exactly that a while back?

Just a guess

John

[Sorvoja]

I am afraid that I don't know all the causes for a page being moved, but I know how to
get them back in the main index. First I check if the site is crawlable, then I will check for duplicate content (or lack of content), then I will check the backlinks. A crawlable site with ok content with a rasonable number of backlinks are never in the supplemental index.

[Halfdeck]

A supplemental page is a page that was NOT spidered, but is known about from links on pages that were.

Ammon, I completely disagree.

First, you're describing what's known as "url only" pages blocked by robots.txt. Google doesn't spider the content of those pages but it lists the urls in the index because, according to Matt Cutts, people link to them.

Second, a page listed in the main index can also show up as supplemental depending on the query.

It's quite likely that every site has an older copy of all its indexed pages in the supplemental index (I'll elaborate if you want).

Some site owners over at WebmasterWorld have been discussing an issue where on Bigdaddy data centers, the site wouldn’t be crawled as much in the main index. That would result in Google showing more pages from the supplemental results for that site.

http://www.mattcutts.com/blog/gone-supplemental/

(Though he writes "the site wouldn't be crawled as much in the main index" - which doesn't make sense - I believe he means "the site wouldn't be listed as much in the main index").

Third, there are plenty of supplemental listings where the site's navigation link text, meta description, etc are displayed in the SERPs.

Fourth, when two urls points to the same content, often one will "go supplemental." For example, www.domain.com and domain.com pointing to the same /index.html or www.domain.com/index.htm and www.domain.com/, etc. There is no way in hell Google can judge the two urls point to the same content unless it takes a look at the contents of both those pages.

[Black_Knight]

First, you're describing what's known as "url only" pages blocked by robots.txt. Google doesn't spider the content of those pages but it lists the urls in the index because, according to Matt Cutts, people link to them.

No, I'm not, but there is a connection.

URL only pages was where the term 'supplemental' began. This was Google's own term for pages it had not spidered yet, but could still return in results just from link text and relevancy data. If you try looking for some of the oldest articles about Google at searchenginewatch.com you'll see just how long there has been a supplemental index.

The more recent version of supplemental pages - which are marked supplemental in listings - are obviously somewhat different, and are a development. But why did Google choose to call these supplemental when that was already their term for URL only pages? Is there a connection between the two things?

In practice wherever I have seen supplemental pages I have seen pages that have low crawl priority. In many cases, pages 'go supplemental' just before dropping out of listings entirely. In other cases, the supplemental listing gets picked up again at a later time, but sometimes other URLs from the site go supplemental instead. In such cases, the percentage of supplementals versus indexed pages seems to stay around the same, even though which pages are in which category varies month by month. Exactly as if the spider simply wasn't getting to crawl all of the pages of a site, leaving some to have only older indexing data, (which could be far out of date), and thus is deprecated in some manner (like slapping a clear Supplemental label on it).

Second, a page listed in the main index can also show up as supplemental depending on the query.

Now that I haven't seen yet. Was the same datacentre involved in both queries?

[JohnMu]

Could the supplemental index not be used as a warning-sign for webmasters? There is usually a good reason for reduced crawler priority and quite often the underlying issues are things which the webmaster can remedy on the site itself (eg bad in-site linking practices, missing redirects from obsolete URLs, etc).

I don't understand why Google would choose to hide it in that case - it would make more sense to mail the webmaster (if they can) and tell them that if they don't "clean up" those URLs will drop out completely sooner or later. A simple warning like "10% of your pages has been moved to the supplemental index - please check your site if you do not want them to be removed completely" could go a long way... not only would it help the webmasters to fix their site, it would also help all the search engines by giving them clean, well-linked URLs.

I think there is a lot to be said for better communication between webmasters and the search engines - both sides profit when a search engine can inform the webmaster of trouble and the webmaster actually fixes it.

John

[Black_Knight]

Regarding the quote from Matt Cutts you were having trouble with, Halfdeck, look at the wider context:

on Bigdaddy data centers, the site wouldn’t be crawled as much in the main index. That would result in Google showing more pages from the supplemental results for that site.

This was during the testing of BigDaddy, when that index was on test servers, and was not the main index. A lot of sites were not being crawled as much in the test of BigDaddy as they would (and were) on the main servers.

As noted, a decrease in crawling results in supplementals showing for uncrawled pages.

Matt refers to adjusting the variable for crawl priority to increase this somewhat across the majority of sites (it seems the prioritisation algorithm was just a little too harsh):

several people responded with enough details that we identified and changed a threshold in Bigdaddy to crawl more pages from those sites.

This was also the first occasion I recall that Matt alluded to some pages which are crawled, still not actually getting indexed:

we can still fetch those pages, it’s just that they don’t always make it live. I believe someone checked in another threshold yesterday based on the meeting that I had some with crawl/index folks.

So there, a different value from crawl priority determines whether a crawled page will actually go into the live index.

Matt was a lot clearer on this in some later blog posts:
Matt's post on Google's indexing timeline is far more specific. One particular gem of insight was in the comments where Matt said:

it’s by design in Bigdaddy that we crawl somewhat more than we index in Bigdaddy. If you index everything that you crawl, you never know what you might be missing by crawling a little more, for example. I see at least one indexed post from your forum, so the fact that we’ve been visiting those pages is a good indicator that we’re aware of those pages, and they may be incorporated in the index in the future

John, I would certainly say that having pages suddenly go supplemental is a warning sign to SEOs. It tells you that your site has a low crawling or indexing priority that has fallen below a working threshhold level.

It could be that inbound links you'd built have been discounted due to discovery of its connection to a bad neighborhood. It could be that some of those inbound links have simply dropped in visibility (as with links in blog posts once the post drops from the front page, and exists only in archives). It could be that some of your own linking patterns have caused your internal site links to lose value.

It is a red flag - it may not give you any real specific information, but it certainly is there to warn you of potential danger.

[Halfdeck]

BK, I agree with your last two posts.

Now that I haven't seen yet. Was the same datacentre involved in both queries?

Most likely. I think I ran my queries on gfe-eh.google.com. I'm dying to post an example but I can't recall exactly what query I ran.

I first heard about it from gs1md on WMW, and frankly, I found it hard to believe, but seeing my root domain url displayed as supplemental for odd queries which contain text that does NOT appear in the cache is good enough proof for me to believe that 1) supplemental pages are spidered and 2) even a page listed in the main index has a copy of it in the supplemental database:

http://www.webmasterworld.com/google/3060898-3-10.htm

Speculation:

I believe there are several types of supplementals, but let me talk about one scenario.

Google keeps an older copy of indexed pages in the supplemental pages. These pages are usually masked by pages in the main index so as far as we know, they don't exist. When pages from the main index are dropped, supplemental pages become visible - but they were ALWAYS there. Your page didn't suddenly "go supplemental."

Webmasters at WMW racked their brains trying to figure out why their sites suddenly "went supplemental" during BD release. I interpret Matt's answer as: "they didn't go supplemental. Pages in the main index were just dropped (due to new trustbox or whatever you want to call it, affecting crawl priority, pickier indexing, devaluation of paid/traded links etc)" .. "That would result in Google showing more pages from the supplemental results for that site."

Example (ignore the unrealistic timeline):

1. Aug 1. 2005 Google crawls www.domain.com/index.html and stores it in the main index and a copy of it in the supplemental database.

2. Aug 2, 2005. Google crawls domain.com/ and does the same thing. So now we have 4 DB records, 2 in the main index DB and 2 in the supplemental.

At this point, a site: search shows:

www.domain.com/index.html Cached (aug 1, 2005) Similar pages
domain.com/ Cached (Aug 2, 2005) Similar Pages

3. Oct 1, 2005. Google recrawls both pages and updates the main index cache (cache dates updated to Oct 1, 2005 for both urls). Records/timestamps in the supplemental database are left untouched.

4. Nov 1, 2005. Google runs a duplicate content check on the domain and finds the content on the two urls are identical. It decides to drop the url www.domain.com/index.html from the main index, which reveals a supplemental url cached on Aug 1, 2005. So now, a site: search returns:

www.domain.com/index.html Supplemental Result Cached (Aug 1, 2005) Similar pages
domain.com/ Cached (Oct 1, 2005) Similar Pages

[Halfdeck]

PageRank is the primary factor determining whether a url is in the main web index vs. the supplemental results...

- Matt Cutts

http://www.mattcutts.com/blog/fall-weather.../#comment-87795

I'm actually a little surprised by what he wrote.

[JohnMu]

Why surprised?

Pagerank is nothing else but a measure for the links pointing to a page. Of course you never know which pagerank he *really* means. Do you think there is only one internal pagerank and that it's calculated in the same way that the exported pagerank is? Small changes in the calculations can make a large change across the web, small changes in the algorithm (how they "guess" at pagerank, since they will hardly re-calculate it for the full web) can make a gigantic difference, not to mention the possible ways they could be discounting links from being valued for pagerank (penalties based on content perhaps effecting the dampening factor for passing pagerank to links from that penalized page)...

All he's saying is that a page who's value has been cut (links devaluated somewhere upstream; lower "pagerank") is passed into the supplemental index. Nothing new to me

John

[hvacdirect]

John,

I agree with your assessment that pages that have devalued links pointing to them will go supplemental. But I'd also add a couple more possible reasons.

1) Duplicate Content: Duplicate pages, one will stay in the normal index, the others go supplemental.

2) 404s. Seems like a deleted page stays around for a year.

3) 301s. Redirected pages stick around for a year as well. With the original content and URL being cached at a date earlier than the 301. Recrawls will happen but at a very slow rate.

4) Pages with only the home site links. I've seen deeper pages, perhaps 3 levels down, go supplemental. These pages only have links to them through the site structure, and no other external links. Though the home page may be increasing in page rank, thus increasing the link power to said pages, they still go supplemental since no one else links to them. Through experimentation, one link no matter how lowly it is from an external site will push it back into the index. To further back this up, I've launched new sites, started the crawling off with 2 or three good links. The site gets crawled and indexed in the normal manner. About 6 weeks later the deeper pages go supplemental. Since I was in control of the original links, I know they are still there. On the contrary, seed the site with a good solid PR6+ link and the rest of the pages have a better chance of staying non-supplemental.

I have also seen a change in the way supplemental pages are treated in the index. On some SERPS supplemental results will rank before non-supplemental. Traffic has remained pretty consistent on sites that have gone supplemental, except the traffic to the home page has increased. Google appears to be giving credit to the home page for content on the supplemental pages.

The big drawback of course with a page in the supplemental index is that it doesn't get crawled that often and the cache will be ancient. If the page hasn't changed, no problem, but if it has then you are not getting visitors for the new content. So I've started a new practice. If I am going to update a page significantly, and its supplemental, I'll rename it, and 301 redirect the old one, maybe even seed it with a homepage link. Now you will have two pages in the index with similar content, but the older one will eventually drop out, but still return some visitors in the mean time.

my 2 cents.

John

This post has been edited by hvacdirect: Oct 12 2006, 05:54 PM

[bragadocchio]

What are all the possible reasons for a page to have ‘Supplemental Result’ next to the url in the Google search results?

So that they can index more pages by capturing less information for lower ranking pages.

Nice description of a multiple staged index in this document:

Multiple index based information retrieval system

[Halfdeck]

So that they can index more pages by capturing less information for lower ranking pages.

Nicely put Bill.

Another problem with conventional information retrieval systems is that they can only index a relatively small portion of the documents available on the Internet. It is currently estimated that there are over 200 billion pages on the Internet today. However, even the best search engines index only 6 to 8 billion pages, thereby missing the majority of available pages. There are several reasons for the limited indexing capability of existing systems. Most significantly, typical systems rely on a variation of an inverted index that maintains for every term (as discussed above) a list of every page on which the term occurs, along with position information identifying the exact position of each occurrence of the term on the page. The combination of indexing individual terms and indexing positional information requires a very large storage system.

And Google keep insisting they're nowhere near out of storage space. What is clear is Google will not index everything under the sun.

The indexing system 110 is responsible for identifying phrases in documents, and indexing documents according to their phrases, by accessing various websites 190 and other document collections. The front end server 140 receives queries from a user of a client 170, and provides those queries to the search system 120. The search system 120 is responsible for searching for documents relevant to the search query (search results), including identifying any phrases in the search query, and then ranking the documents in the search results using the presence of phrases to influence the ranking order. The search system 120 provides the search results to the presentation system 130. The presentation system 130 is responsible for modifying the search results including removing near duplicate documents, and generating topical descriptions of documents, and providing the modified search results back to the front end server 140, which provides the results to the client 170.

Interesting. But that patent describes what might happen when someone runs a query on Google. I'm thinking more of the process involved during crawling.

Why surprised?

Well, I dunno. Maybe I was baiting

Seriously though, Matt Cutts mentioning PageRank isn't surprisng to me. I believed that to be the case for a while. In this WMW thread, I wrote:

I also think once a site goes heavily supplemental, you need to regain some trust with Google to get pages back into the main index (i.e. organic inbounds/PageRank). It would be one way Google guards itself against 100,000,000 page spam sites sitting in the supplemental index, and preventing any periodical on-page tweaks from reinjecting the site into the main index.

I also wrote here:

To be or not to be supplemental.. is a question that hinges on more than one factor alone. I believe PageRank is a factor, but a PR of 8 may not save you if the page is an identical copy of a page with a PR of 10.

Google's algo is not a single IF/ELSE statement.

It's hard to put my reaction into words, so instead I'll quote gs1md from this thread:

Heh, Marcia, you’re gonna love this Matt Cutts comment:

>> PageRank is the primary factor determining….

Note: The “primary” factor.

Jeez. No mention of Duplicate Content, and Redirects and 404 URLs at all.

Do you think there is only one internal pagerank?

No. Imo, they need to maintain *at least* two internal PageRank in case they need to backtrack. Assuming they're constantly improving the way they calculate PageRank (e.g. improving how they identify certain type of links: bought links, reciprocals, footer non-internal links, etc), each test DC with a different PageRank calculating mod I'd assume would have a different set of internal PageRank.

and that it's calculated in the same way that the exported pagerank is?

If I go strictly by what Matt Cutts has said so far (I know, that's assuming a lot), TBPR is exported internal PageRank translated on a 0-10 scale. The only inaccuracy involved with TBPR is PageRank updates continuously while TBPR updates every few months, and internal PageRank is more granular than TBPR. I'd compare it to reading the news every three months instead of every day, and instead of being able to read full articles, you only get to read the headlines.

[bragadocchio]

But that patent describes what might happen when someone runs a query on Google. I'm thinking more of the process involved during crawling.

It describes two stages of ranking. The first could look primarily at pagerank, to decide which documents would end up in the primary index, and which would go into a secondary, or supplemental, index.

The scoring algorithm for pre-ranking the documents may be the same underlying relevance scoring algorithm used in the search system 120 to generate a relevance score. In one embodiment, the IR score is based on the page rank algorithm, as described in U.S. Pat. No. 6,285,999. Alternatively or additionally, statistics for a number of IR-relevant attributes of the document, such as the number of inlinks, outlinks, document length, may also be stored, and used alone or in combination in order to rank the documents.

A year ago, Anna Patterson (inventor named on that patent application) told us that they expanded the size of their index by three times for their seventh birthday, at the official Google Blog.

Google might not be using the process described in this patent application, or the other four that she filed over the past year on a phrase indexing system. But they are worth looking at, and considering carefully

Multiple index based information retrieval system (20060106792)
Phrase-based searching in an information retrieval system (20060031195)
Phrase-based indexing in an information retrieval system (20060020607)
Phrase-based generation of document descriptions (20060020571)
Phrase identification in an information retrieval system (20060018551)

It does seem to provide a framework that addresses some of the behavior that we are seeing.