Jump to content

Cre8asiteforums Internet Marketing
and Conversion Web Design


Robots.txt Miscue?

  • Please log in to reply
4 replies to this topic

#1 iamlost


    The Wind Master

  • Site Administrators
  • 5488 posts

Posted 26 January 2013 - 02:58 PM

Donna Fontenot ‏@DonnaFontenot

There's nothing more disheartening to an SEO than to see another "pro" highly-paid SEO ignore SEO 101...robots.txt blocking search engines.

Yup, I've been eavesdropping on her Twitter stream again :)

Now, I don't have any idea of the context of this tweet. Given that it is Donna I presume someone made an accidental unintended disastrous booboo. Regardless, it seems a topic not much mentioned: the control of SE bots. If you are one of the multitude who are wide open to every visitor, bot and human, that comes a-calling the following is likely an unfathomable mystery as much why bother as how... but that is too much effort... feel free to bale at this point, I don't mind at all. I've always preferred quality over quantity. :)

I block SEs each and every day from all my sites. Actually, only a very few are allowed in, and of those only their main indexing bots; no media bots, no fetcher bots, no no no. And then there are large sections of my sites, varying from ~30% - ~40% of pages are simply off limits to those allowed indexing bots. Yes, they still try and at least occasionally get in by various means but so long as they then honour the meta noindex the leak is moot; if not then they hear from my law-type-person.

I'm among the most extreme of the selective SE bot access types that I know. That is not what is important, what is is that there may well be good business reasons for blocking in whole or in part some or all SE bots. If that is the case then good on ya; if, however, you inadvertently mistakenly made the following robots.txt:

User-agent: *
Disallow: /

Please accept my heartfelt sympathy in your bereavement, where should I send the flowers?
Yes, it is bad of me to be rolling on the floor laughing hysterically but the relief in rather you than me was simply too much...
Is it too too much to hope that you are a competitor? Oh, please, with whipped cream and a cherry on top, oh please...

#2 DonnaFontenot


    Peacekeeper Administrator

  • Hall Of Fame
  • 3840 posts

Posted 26 January 2013 - 04:28 PM

The story behind the tweet:

A relative is a local real estate broker for a major real estate franchise. The regional headquarters came out with new whizbang websites that they convinced the entire southeast region of local (and locally-owned) offices to pay for. One of the many "benefits" of this was the awesome seo behind the technology, as well as their regional "professional" seo who would make sure the sites were rarin' to go.

After a few months of going live, my relative asked me why they weren't ranking well. Before I even looked, my first thought was, "well..it's real estate..tough competition". But oh no. A site: search quickly underscored the problem

A description for this result is not available because of this site's robots.txtlearn more.

and robots.txt revealed:

User-agent: *
Disallow: /

Oh my. The same was revealed to be the case for others in the region.
SEO 101 folks. If you want search engines to index you, don't put locks at the gate.

#3 jonbey


    Eyes Like Hawk Moderator

  • Moderators
  • 4757 posts

Posted 26 January 2013 - 09:11 PM

I will never forget the time I blocked Google.


#4 cre8pc


    Dream Catcher Forums Founder

  • Admin - Top Level
  • 14789 posts

Posted 26 January 2013 - 09:47 PM


Looky what I found in Google!

Attached Thumbnails

  • indexno.png

#5 clandestino


    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 985 posts

Posted 27 January 2013 - 02:36 AM

I had a prospect's SEO do this --

User-agent: *
Disallow: /

Ooops, the bot never gets to line 3.

I got there 3 days after googlebot. His rankings didn't come back like before.

Edited by chuckfinley, 27 January 2013 - 02:43 AM.

RSS Feed

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users