Rand Fishkin Interviews Vanessa Fox, Google Insights Options

[AbleReach]

http://videos.webpronews.com/2007/04/12/se...nessa-and-rand/
Fun interview with a lot about what may be up and coming in Google and Google's Webmaster Tools.

Take-aways:


Use robots.txt to tell spiders about your site map. After validating your Google site map, reference it in your robots file with a line like this:

CODE

Sitemap: <http://yoursite.com/sitemapfile.xml>

Ask.com is also supporting this.
reference: http://www.sitemaps.org/protocol.html#submit_robots


To avoid as much of a long dip in rankings when changing to a new domain, first move old content to the new site as is, with 1-to-1 301's from the old site to the new. Start any re-design after the new site is indexed and present in SERPs.

This post has been edited by AbleReach: Apr 13 2007, 01:52 AM

[JohnMu]

just a tiny note: there are no brackets around the URL

also: you need to make sure that the sitemap:-line is after a user-agent-block and that it does not have empty lines within the block, eg:

Good:

CODE

user-agent: *
Sitemap: http://mydomain.com/sitemap.xml

user-agent: Googlebot
Disallow: /somestuff
Sitemap: http://mydomain.com/sitemap.xml

Corrected:

CODE

Sitemap: http://mydomain.com/sitemap.xml

user-agent: *
Disallow: /stuff

user-agent: Googlebot
Disallow: /somestuff

Details I'm sure everyone here would have right anyway

John

Edited as corrected

This post has been edited by softplus: Apr 16 2007, 01:29 PM

[AbleReach]

My bad re the brackets.

However, if I read this correctly, it's now ok to put it on its own, and Google will get it. Ask spiders will soon be understanding it, too.

This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file.

From http://www.sitemaps.org/protocol.html#submit_robots

[bwelford]

It seems to me, Softplus, that the point you're making is an important one. You're pointing out that the sitemap line cannot go anywhere if you have a robots.txt file that has other exclusions. It must then come suitably early otherwise it would apply only to the bot that is covered by the part of the txt file where you've located the code. That's different from what has been said in the official language.

[Vanessa Fox]

Actually, the Sitemap line can go anywhere in the file -- it's independent of the user-agent blocks. All search engines will pick it up regardless of where the line is located.

If you don't want all the engines to pick it up, you can block the Sitemap file itself from particular user agents, and in that case, only the allowed bots will actually retrieve the file.

[bobbb]

only the allowed bots will actually retrieve the file

only the allowed bots (that follow the rules) will actually retrieve the file.

[bwelford]

Thanks for the clarification, Vanessa. It's much easier to be able to keep it simple.

[JohnMu]

Nice to see you here, Vanessa

Thanks for the clarification!

Is there a reason why the sitemap-file is not on the same level as the allow/disallow statements (and user-agent bound)?

John