Jump to content

Cre8asiteforums Internet Marketing
and Conversion Web Design


What Sould I Use, Robots Txt Or Robots Meta Tag?

  • Please log in to reply
4 replies to this topic

#1 MainStreet228


    Mach 1 Member

  • Members
  • 468 posts

Posted 06 May 2012 - 08:45 AM

Hi Everyone,

I have a certain page on my website that I don't want to be indexed by google, yahoo and bing. In order to block the search engines from indexing the page I'm torn between a robots txt file or a robots meta tag. What should I do?


If I decide to go the route of the robots txt file. The file itself would look something like this?

User-agent: *
Disallow: /the-page-url-goes-here.php

Thanks everyone!

#2 iamlost


    The Wind Master

  • Site Administrators
  • 5474 posts

Posted 06 May 2012 - 10:39 AM

The problem with robots.txt is that it does NOT work for what you want. It only tells a bot not to crawl the specified page, dir, etc. Given that all the major SEs 'find' your URLs by methods other than crawling your site and that they have bots that ignore robots.txt it is common for such 'disallowed' pages to be indexed and returned in search results.

The easiest 'fix' is to allow the SE bots to crawl the page(s) but require that they not be indexed.
Using meta tags to block access to your site, Webmaster Tools Help, Google:

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

As the article goes on to say the usual method is:
<meta name="robots" content="noindex">
Note: 'follow' is the default so you do have to include it. However, I always specify defaults so that I'm not relying on third parties behaving as they should. :)
<meta name="robots" content="noindex, follow">
With 'follow' even though the page is not indexed nor (supposedly) returned in query results page values will flow as normal. If you replace 'follow' with 'nofollow' that may not happen, accrued page value may be bottled/evaporated.

If you really don't want the SEs to know the page content then you need to actively block them from the page, which may or may not actually work well depending on your skill level.

#3 MainStreet228


    Mach 1 Member

  • Members
  • 468 posts

Posted 06 May 2012 - 05:41 PM

Than you Iamlost for your help.

I'm going for the go the meta tag route. Below is a copy of my meta tags. From what I've read this should work. What do you guys think? Thanks for your help.

<title>My title will go here</title>
<meta name="description" content="my description will go here" />
<meta name="keywords" content="my keywords will go here" />
<meta name="googlebot" content="noindex">
<meta name="robots" content="noindex">

#4 Wit


    Sonic Boom Member

  • 1000 Post Club
  • 1599 posts

Posted 07 May 2012 - 01:17 AM

You don't really have to single out googlebot. It obeys the "regular" robots' exclusion just fine. But this should work, yes

#5 MainStreet228


    Mach 1 Member

  • Members
  • 468 posts

Posted 07 May 2012 - 07:56 AM

Thanks Wit!

RSS Feed

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users