What Sould I Use, Robots Txt Or Robots Meta Tag?
Posted 06 May 2012 - 09:15 AM
I have a certain page on my website that I don't want to be indexed by google, yahoo and bing. In order to block the search engines from indexing the page I'm torn between a robots txt file or a robots meta tag. What should I do?
If I decide to go the route of the robots txt file. The file itself would look something like this?
Posted 06 May 2012 - 11:09 AM
The easiest 'fix' is to allow the SE bots to crawl the page(s) but require that they not be indexed.
Using meta tags to block access to your site, Webmaster Tools Help, Google:
As the article goes on to say the usual method is:
To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.
<meta name="robots" content="noindex">Note: 'follow' is the default so you do have to include it. However, I always specify defaults so that I'm not relying on third parties behaving as they should.
<meta name="robots" content="noindex, follow">With 'follow' even though the page is not indexed nor (supposedly) returned in query results page values will flow as normal. If you replace 'follow' with 'nofollow' that may not happen, accrued page value may be bottled/evaporated.
If you really don't want the SEs to know the page content then you need to actively block them from the page, which may or may not actually work well depending on your skill level.
Posted 06 May 2012 - 06:11 PM
I'm going for the go the meta tag route. Below is a copy of my meta tags. From what I've read this should work. What do you guys think? Thanks for your help.
<title>My title will go here</title>
<meta name="description" content="my description will go here" />
<meta name="keywords" content="my keywords will go here" />
<meta name="googlebot" content="noindex">
<meta name="robots" content="noindex">
Posted 07 May 2012 - 01:47 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users