Yahoo! Sitemaps??? Options

[rustybrick]

I found this over at http://www.seo-scoop.com/direct_link.cfm?thepost=482

Go to http://submit.search.yahoo.com/free/request

Is the line I quote below new?

You can also provide the location of a text file containing a list of URLs, one URL per line, say urllist.txt. We also recognize compressed versions of the file, say urllist.gz.

I heard rumors of Yahoo! offering this at the past SES conference, but they did not say when...

[rustybrick]

Confirmed, its brand new. It was launched the thursday of SES.

[bragadocchio]

Nice.

Thanks for bringing this to our attention, Barry. I think that it could be useful.

I like the simplicity of it, as a list of URLs in a text file. It looks like it is definitely worth testing.

[nuts]

I just tried submitting a urllist.txt.gz file. It produced an error message saying : timeout. Then I submitted the plain urllist.txt file. It took it.

[Respree]

I just tried it too. Don't think the timeout is a function of the file type. I tried it with a .txt file and it timed out too. Tried it a few minutes later and it went just fine.

That was easy. Don't know how I missed this thread.

(Belated) Thanks for bring this up, Barry.

[organicseo]

I submitted a text file to yahoo containing urls of my site and yahoo indexed that text file itself.

Try http://search.yahoo.com/search?p=site:yell...kha.com+sitemap

Its idiotic.

I've removed that txt file to another location, but the cached version still remains.

[nuts]

Hi Organic, welcome.

How long did it take before the sitemap was indexed?

I notice it is not named "sitemap.txt", maybe that is the reason the file was indexed.

I also notice that there are 356 results for yellowpages.sulekha.com. Were these on yahoo before you submitted the sitemap?

[organicseo]

actually i had more pages in yahoo before that. but the sitemap.txt is indexed and is in yahoo serp's, which is not good and submitting a text based sitemap to yahoo is useless and your competitors can see the entire urls in your sitemap in yahoo serp's like http://search.yahoo.com/search?p=site:yell...kha.com+sitemap

[Respree]

I'm wondering if anyone knows if the Y! sitemap feed simply adds fed URLs it's index directly, or if it's designed simply to help the regular crawler?

[JohnMu]

The Yahoo-function can't be compared to Google Sitemaps :-)

What the submission of the URL list does is just save you the manual work of adding them all to the submit URL listing. Yahoo is generally a heck of a lot faster at just finding your site (as a registrar it checks all new registrations after about 10-30 days, according to my tests). If you have links from other sites, it'll find the site much faster and crawl it fully if you have enough "good" links. Remember: how long does it take manually entered URLs to get into Yahoo's index? Why should a URL-listing function make it easier than the paid inclusion?

One thing to keep in mind - the file you submit to Yahoo is read once, and one time only, when you submit. You can remove the file afterwards. No need to gzip it, it's a one time bandwidth hog. For the same reason, there's no need to keep the file updated, Yahoo doesn't look anymore, ever (at least since I started keeping track in August).

Google Sitemaps checks the sitemaps file(s) twice daily (at the moment, perhaps that'll change?) and on a site that is "valued" enough Google will crawl new URLs based on the contents of the sitemaps file. Try it on a good site, add a new URL that isn't linked within your site, Google will pick it up within hours (depending on the value of the site, of course). Sitemaps is amazing in that regard!

(shameless plug - my GSiteCrawler can generate Google Sitemaps and Yahoo URL lists, and lots more, and is still free :-))

Another thing about Google Sitemaps - the statistics are getting better and better all the time; I would not be surprised if Google had a full-scale webmaster control panel through the sitemaps interface by spring 2006. At the moment it's still a bit limited, but I think they're just practicing -- it's good to see where a crawler like Google runs into trouble when looking at your site, it makes "fixing" it much easier.

Another thing to keep in mind, Google Sitemaps will most likely remain Google-proprietary, even if they say otherwise. The main issue is (at the moment), that you can call your sitemap files whatever you want and you have to register them at Google's site in order for Google to pick them up. How would Yahoo find out about them? They can't guess at it (at least they haven't so far)... MSN is supposedly working on something similar based on RSS (anyone have more on that?), I don't know what Yahoo is planing. Or do they put all the money into making a better crawler that doesn't need help? Hmm.