Jump to content

Cre8asiteforums Internet Marketing
and Conversion Web Design


Photo

Do search engines "prefer" a type of directory str


  • Please log in to reply
24 replies to this topic

#1 JimZim

JimZim

    Gravity Master Member

  • Members
  • 140 posts

Posted 21 December 2002 - 02:23 AM

[size=10]I've not seen it stated anywhere whether or not the SE's have a preference as to how a site might set up its directory tree. My question is does it matter at all to them how I organize my site? I am hoping to find some suggestions as to how to best organize the SEO work I do for the site pages.

If I am going to have pages optimized for a keyword for a search engine then I am going to end up with a lot of pages and want to be able to effectively use the robots.txt. Thus the impetus for asking!

Thanks, and good weekending to all!


#2 enigma

enigma

    Mach 1 Member

  • Members
  • 330 posts

Posted 21 December 2002 - 06:19 AM

Blimey, James, you're an insomniac :D

This is probably common-sense, however, I tend to bung all pages directly in the root; support files like mail scripts and asp/perl files go in a separate directory as do image and css files. But if you've got a huge site then I've seen many guys logically split files into sub-folder groups. I've noted discussion elsewhere suggesting the core pages should be as close to the root as possible since (it is suggested) spiders may not like going deep into a directory structure. The only reason for this I can think of is the string storage space necessary to keep track of the URI as they recurse the structure. Having said that, I imagine if you keep it short like root/this/that/the_other.html rather than root/product_files_section/thingamybob_bin/big_thingies/this_is_product_a.html you should be ok. Er...I think :)

#3 cre8pc

cre8pc

    Dream Catcher Forums Founder

  • Admin - Top Level
  • 13365 posts

Posted 21 December 2002 - 10:04 AM

Bottom line for every site I build or have built since 1995 is the index and sitemap are definite root level pages. My entire site and most sites I build have every page at root level except .js, CSS, .cgi, etc. With the various limits from each engine on if or how many directories it will even enter, I put everything at the front door. So far, in all the cases, the entire site is eventually crawled and nearly every page added, esp. the smaller sites. I can count the pages on the server, and count the pages returned by the engine, and the numbers match.

Having a text only sitemap helps with this too. To help categorize pages for easy maintenance, I assign file names that tell me what hub they belong to.

Kim

#4 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 21 December 2002 - 11:11 AM

I used to think that pages closer to the root directory had a better chance of being spidered and getting a higher PageRank. However, after a bit of thought, I realized it really doesn't matter where in the physical directory of the server the pages lie -- it's the linking structure structure that makes a difference.

In other words, if you have a page buried deep in the directory:

yourdomain.com/directory1/directory2/directory3/directory4/page.htm

It can be easily found if you have a link to it from the main page (and also get a good pagerank.

Just as if you have a page:

yourdomain.com/page.htm

it can be buried (according to search engines) if you don't have a link to it from the main page.

The first example will be more apt to be spidered than the second. It's all in the linking, not in the physical structure.

Jill

#5 Guest_PhilC_*

Guest_PhilC_*
  • Guests

Posted 21 December 2002 - 12:33 PM

I still prefer to have shallow directories - only 1 or 2 deep. But I agree with Jill, although I don't have evidence for it.

There has often been talk about how "deep" the different spiders go, but "deep" can mean 2 things:- (1) deep, as in the number of physical directory levels and (2) deep, as in the number of steps/links a spider has to take to reach the page. In the second case, as Jill said, a spider (and user) may need a lot of steps/links before it arrives at a page that is physically located in the root directory.

I've always assumed the second definition, although I still prefer to have shallow directories as well.

Phil.

#6 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 21 December 2002 - 01:03 PM

It's really a number of clicks thing...that's a good way to think of it.

J

#7 JimZim

JimZim

    Gravity Master Member

  • Members
  • 140 posts

Posted 21 December 2002 - 05:35 PM

Damn, these folks are good (even you enigma :roll: )

So, one follow up (I say one, but you all know there is likely to be three or four ;)

How do you all name and organize your pages for your own benefit (given the info above)? Realizing that this becomes your own personal choice, I am wondering how you are keeping track of the pages where they are submitted to, etc. I have read on a number of sites (links from these pages, mostly) that it is not good to use the SE's names or initials in the directory or file names. My inclination would be to create an optimized page for the site, say "best_forums_on_the_planet.html" and then submit it to the engines. Of course I would want to make it optimized for that engine, and in doing so, the page would be different than if it were optimized for that other SE.

Given the talks herein about robots.txt, I would of course not want those two pages to be spidered by the same engine, so I would want to put them in different directories and disallow each from the other (presuming that I am going to have many more engine specific optimized pages). So I wonder if one might use a code system to indicate the search engine name for the directory that you are putting your SE optimized pages into? Or is all of this just not really an issue "in reality"?

Thanks further for your thoughts and ideas. Most appreciated.


#8 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 21 December 2002 - 05:50 PM

Oh man, you're actually still creating different pages for different engines?

Just use your actual site, and optimize it with great keyword rich text and tags, and you can rank high in ALL of the search engines. You're making things so much tougher on yourself when trying to optimize different pages for different engines.

Not to mention that you're making a mess in the engines by submitting that kind of thing.

Jill

#9 JimZim

JimZim

    Gravity Master Member

  • Members
  • 140 posts

Posted 21 December 2002 - 06:14 PM

Oh man, you're actually still creating different pages for different engines?


[size=10]I feel an "Uh oh" welling up in side!

Just use your actual site, and optimize it with great keyword rich text and tags, and you can rank high in ALL of the search engines.  You're making things so much tougher on yourself when trying to optimize different pages for different engines.


[size=10]Everything that I have been led to believe is that you can't expect to rank high across the boards with a single KWP optimized page. Is this simply not true any more? Believe me when I say I don't want to be ignorant or beligerent, however, I do want to be able to maximize my opportunities to bring the best page for that search engine as high on the SERP as possible. Is there something inherently wrong with want to do that? Wanting to put in the work/extra time/extra efforts? Or is this just considered "Wrong"? I ask because of your next comment:

Not to mention that you're making a mess in the engines by submitting that kind of thing.


[size=10]Please, please explain just how you consider this to be "making a mess". I truly don't understand this statement or thought process.

Again, my goals/designs are not to "make messes" or to fall into the category of the spamming SEO; no, just to be able to get some relevant pages to rank well.

Do you have an example of a KWP that you have optimized for on a single page, that has ranked high across a number of SE's? If so, would you be willing to share that page here as I truly think it would be beneficial (at leas for me, but I am hoping for others as well ;))

Thanks again for the input. I really am finding the mind expansion good!


#10 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 21 December 2002 - 06:45 PM

Everything that I have been led to believe is that you can't expect to rank high across the boards with a single KWP optimized page. Is this simply not true any more?

It's never been true.

And where are you learning this? On sites from 1995? You're talking about old-time optimization strategies that rarely work any more. Certainly not for very long.

Creating doorway/gateway pages whether one for all engines or lots for different engines, is simply a short-sighted strategy. By doing that, you're constantly chasing your tail and the latest SE algorithm.

By working with the actual pages of your site, making them the best they can be for the search engines and your users, you're on your way to a long-term strategy that will give you high rankings that will last for years to come.

Please, please explain just how you consider this to be "making a mess". I truly don't understand this statement or thought process.

Think about what the search engines are there for. They are there to find websites that exist in the engines and fit a particular searcher's query. By creating "extra" pages for no other reason but to more yourself above your competitor's, you're showing no respect for the engines, and are basically cluttering their databases. These kinds of actions are the bane of their existence. If it wasn't for people adding pages like this to their indexes, their results would be much easier, and the better sites would be found just cuz they were better. Instead, the engines spend much of their time trying to combat the very stuff you want to do.

You may find my article (http://www.highranki...gatewaymyth.htm): "The Myth of Gateway Pages" interesting. It was written a few years ago, but makes sense for this discussion.

As far a page optimized for all engines, I'm sure I have plenty of them (although they may rank high for some keywords in some engines and other keywords in other engines). However, I believe my friend Doug's site may do the trick for you. Check the term "search engine optimization" without quotes in all the engines, and you should see freemoneyservices.com in the top 10. (Doug is the administrator at the IhelpYou forums where I'm also a moderator.)

I don't mean to pound on your personally Jim, I'm just really surprised that you were still using the methods you mentioned, as I thought they went out with the Macarena!

Jill

#11 cre8pc

cre8pc

    Dream Catcher Forums Founder

  • Admin - Top Level
  • 13365 posts

Posted 21 December 2002 - 08:30 PM

I used to get surprised reactions from SEO people for never optimizing mine or client sites for different engines, but that's what I did. For a long time, Alta Vista was the one to optimize for. I focused on that engine and could tweak when necessary. For all of them, I took them as a whole, and focused on the basics. Good title tag, good meta description, excellent content to back it up and content-based sites. I know I'm in the minority on this, or at least was. The sites I worked on privately were small businesses, and therefore small sites that didn't need to jump through hoops to be indexed. For rank, they worked on links, fresh content, sticky, branding, marketing, etc.

When you optimize hubs, these are doorways to the site. No need to create additional pages, so I never did doorways either. Again, in the minority there.

I was and am a lazy SEO :shock:

My feeling was that people were impatient and wanted instant results as soon as their site was submitted. I coached my clients to be patient, trust me, and their sites would be found and rank well over time because they were solidly built, structured to attract spiders and focused on specific keywords.

Nowadays, most "engines" are portals that pull results from Ink or Google and Directories and Overture as additional feeds. I think many people find studying individual algorithms and aiming for each engine's specific ones like the challenge and see value in it. I focused on the users, not the crawlers because in the end, my clients wanted sales and leads, and if their sites ranked well, but weren't built for customer satisfaction, they'd fail regardless.

Separate pages in directories for each engine was a big thing in the late 1990's and 2000 but it seemed to die out sometime in 2001. I haven't heard many people referring to it. Everything is PageRank now, and the emphasis on Google. This doesn't mean optimizing per engine is wrong. Everyone has to decide for themselves what they feel is worth doing and what they want to accomplish.

Kim

#12 Black_Knight

Black_Knight

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 9339 posts

Posted 21 December 2002 - 08:39 PM

Everything that I have been led to believe is that you can't expect to rank high across the boards with a single KWP optimized page. Is this simply not true any more?

It's never been true.


Or rather, it has never been true of Jill's methods, but has been debated openly in the Does SEO=Spam thread, where Both Phil and myself made our cases clearly and in detail about where this is still true, why it is still true, and what your chances are of ranking #1 for 'Casino' on all engines without optimising for each and every engine's algo.

Using specifically optimised pages for each and every engine is not generally necessary. However, sometimes it still is. It's also faster than building up hundreds of inbound links usually, but not as stable, because the engines deliberately tweak the algos quite often just to upset the really specifically optimised (rather than generically optimised) pages in the index.

This topic has split into a separate discussion covering the differences between 'hard' and 'soft' levels of SEO, the stances of engines to SEO techniques, and the ever popular discussion about Google vs. SearchKing.

#13 Black_Knight

Black_Knight

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 9339 posts

Posted 21 December 2002 - 08:54 PM

[quote=JimZim]How do you all name and organize your pages for your own benefit (given the info above)? Realizing that this becomes your own personal choice, I am wondering how you are keeping track of the pages where they are submitted to, etc. I have read on a number of sites (links from these pages, mostly) that it is not good to use the SE's names or initials in the directory or file names. My inclination would be to create an optimized page for the site, say "best_forums_on_the_planet.html" and then submit it to the engines. Of course I would want to make it optimized for that engine, and in doing so, the page would be different than if it were optimized for that other SE.

I go for an organized canonical structure where I can, rather as web directories do, and for similar reasons. It doesn't hurt at all that most search scientists are looking more and more at 'the semantic web' and at off-page criteria (which includes information architecture) for relevancy ranking.

I never separate pages for differing SEs by directory, but rather play to their tastes, offering a wide range of differently optimised pages from which each engine can choose it's own 'most relevant' without there being 'cookie cutter' doorways. The best doorway pages are those that even an SEO can land on without seeing that they are doorways built for ranking reasons.

The trick is to build tons of content. Some on long pages of text, some on short, some of each with high key word density, others with lower density, etc, etc.

#14 Jbishop

Jbishop

    Gravity Master Member

  • Members
  • 138 posts

Posted 31 December 2002 - 11:00 AM

Although I'm kinda afraid to jump into this ring...

Is there an SEO benefit to integrating keywords into the directory structure?

[ie. all-florida.net/st-augustine/st-augustine-hotels.html]

KW: st augustine hotels

#15 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 31 December 2002 - 11:02 AM

Won't hurt, but other things are much, much, much more important.

J

#16 JimZim

JimZim

    Gravity Master Member

  • Members
  • 140 posts

Posted 31 December 2002 - 01:14 PM

Although I'm kinda afraid to jump into this ring...


[b][size=11]Be afraid, be very afraid, JB!

:shock: j/k

So, we wouldn't risk penalization for going overboard by using the KW's in the directory structure? If it doesn't hurt and only helps, I'm all for it.


#17 Advisor

Advisor

    Honored One Who Served Moderator Alumni

  • Hall Of Fame
  • 1142 posts

Posted 31 December 2002 - 01:19 PM

As with everything in SEO, use common sense. If it makes sense to your users to have it laid out that way, then it will make sense to the search engines.

Jill

#18 Jbishop

Jbishop

    Gravity Master Member

  • Members
  • 138 posts

Posted 31 December 2002 - 02:54 PM

:shock:

Jumping out of ring now!

#19 Guest_PhilC_*

Guest_PhilC_*
  • Guests

Posted 31 December 2002 - 10:12 PM

Jumping out of ring now!


hehe....a wise move :D

#20 backword

backword

    Ready To Fly Member

  • Members
  • 12 posts

Posted 02 January 2003 - 10:00 AM

To come back to the original question...
I've got a greymatter http://noahgrey.com/greysoft/ blog and I've had hits from hackers (the greymatter php form http://foshdawg.net/...opic.php?t=2167 explains why and how) looking for defenseless sites... Anyway, the point is that they enter part of the local file name and diectory structure something like archives/00000001.php (Greymatter archives are stored in an archives directory by default), and Google obligingly lists all pages with this in the url.
Why am I bringing this up? Because it means that, if all else fails, Google will list sites by file and/or directory name. So if you sell different things from your site then having directory names with relevant keywords almost certainly will help, because Google pays attention to such things. But don't go calling all your directories 'Britney' now...

#21 Tmeister

Tmeister

    Ready To Fly Member

  • Members
  • 15 posts

Posted 02 January 2003 - 05:38 PM

Interesting thread here! I would agree that engines use link structure in determining depth as it is a better indication of true depth. Although it isn't well known it is very hard to determine depth using directory structure because most links are relative. So an indexer must really map the whole site before this can be a truly effective means of determining depth. As this ../../adirectory is essentially the same as /adirectory when it is already below the first level. When building our spider we actually tried to do it based on directory structure but the added overhead in indexing and mapping quickly made it evident this wasn't a good method of determining depth. It also wasn't a true indication of the contents importance to the site. Pages are sometimes logically put three or four levels deep on a large site for maintenance purposes.

To those who put everything in the root. On top of this being a dogs breakfast to maintain it also takes some real legitimate tools out of your SEO arsenal! In my testing I've determined that most engines using domain names in the algo are also using both file and directory names. Simply put the only way to parse a domain name is to also parse the directory and file or in plain english break the URI down into its various schema. That being subdomain.Site/directory/file. It makes sense that if they feel the domain name is important enough to include in the algo that an actual file name or directory would be as important or more important than a domain name which is often hard to parse becuse there are no delimiters between the words. Hence the increased use of - in domains. The domain misinformation is propogated by those who stand to gain from this myth. Namely domain registrars it is IMHO one of the biggest myths about SEO.

As to the old argument about specializing content for individual engines it is IMHO a stupid premise. Simply because there are a finite number of page and offsite properties that can be legitimately optimized. If you have done the optimization of the site properly then you have optimized all of these elements and attributes to their fullest potential. Whether an engine uses them isn't a consideration. If one uses them then you are covered for that 1 engine. The fact is titles and only 3 or 4 attributes or elements are really effective on a consistent basis. Anything else is subject to algo changes and indexing methods. The reasons for this becomes evident if you take the time to study the HTML specifications. The specification actually says a title and headings are to outline the content of the file. Headings were never a text formatting element they are supposed to act as descriptions of the content contained near or within the Heading and subheading elements. Lazy unknowledgeable authors turned them into Text formatting elements.

In regards to the Google and Bob King debate, IMHO, Bob made a big mistake when he launched the program. That mistake was telling people they were buying PR or even intimating this. A good SEO would have known this and definitely wouldn't need to be told. IMHO, Google took exception to the light in which this program cast on their index and especially PR which is what sets them apart. Google dislikes reciprocal linking as this is a clear indication of webmasters conspiring to possibly inflate relevancy. This conspiracy factor is what has hurt blogs in the recent changes. People just don't think about how bragging or discussing in public "manipulation" of engines effects them. SE read these forums and they do make judgements about companies and individuals based on what they see in these forums. Some benefit some get hurt by talking about spammy methods they use. Or how they conspired with a group of even like sites to manipulate relevancy. No engine can afford to have it known they can be manipulated in this manner. It only results in more of the same.

The best links are usually not reciprocal links but a case of a user linking to a site based on merit (or info) not whether they have reciprocated the link. Personally I never link to a site solely because they linked to mine.
In fact it is a negative for the reasons mentioned above. Number 1 reason should be will it be useful to a user on my site. If it is I could care less whether they link to me. I receive literally hundreds of requests monthly and haven't linked to one. 9 times out of ten they aren't even remotely relevent to the topic of the site. IMHO, to link to them would only dilute not increase my PR. Another SEO myth propagated by those who stand to gain from it. Content is and will always be king! If you build a site based on content that gives people a reason to come both SE and users will reward you. If you are developing websites on the premise that you will fool users and engines you are really only fooling two people, **yourself** and the client foolish enough to hire you!

Just my .02 Canadian

#22 cre8pc

cre8pc

    Dream Catcher Forums Founder

  • Admin - Top Level
  • 13365 posts

Posted 02 January 2003 - 06:25 PM

A hearty welcome to this forum, Webmaster T :wave: It's nice to see you here!

Your comments were good reading. I especially like hearing from those who have programmed search apps because when you get your hands into the pie, it's easier to see how the thing works.

I'm still a diehard root level page-placer, but for larger sites, this is completely nuts to do. I don't hear much these days about how many levels (directories) engines venture into, or limits on pages indexed...would like to get current data on that somehow.

People just don't think about how bragging or discussing in public "manipulation" of engines effects them. SE read these forums and they do make judgements about companies and individuals based on what they see in these forums.


I've wondered if this is truly so, especially when so many popular SEO professionals also write publically about techniques. I don't see Jill Whalen or Ralph (Fantomaster) or Brett (WMW) mincing words or sounding afraid to air their views. I rant too, but try to be fair as well. I believe everyone has a right to expression, and if they want to share techniques - so be it. Sometimes talking about it, or explaining something to someone doesn't mean the person actually does the technique. The day a search engine penalizes a website because its owner explained how to cloak would be sad day on the Internet.

But the fear may be real, and there is a place for anonymous SEO discussion here:

http://webworkshop.net/seoforum/

If you build a site based on content that gives people a reason to come both SE and users will reward you.


You're singing my tune :)

Thanks for stopping by. Hope to hear more from you!

Kim

#23 cre8pc

cre8pc

    Dream Catcher Forums Founder

  • Admin - Top Level
  • 13365 posts

Posted 03 January 2003 - 10:38 AM

Jill,

The scenerio you just described, about trying to locate local sites, is an example of ways I'm working on developing user personas to benefit SEO. So far personas are used in website development. I'm interested in taking this further by getting into the minds of searchers and incorporating their needs into the design AND the search. Usability isn't just navigation labels and meeting site objectives (to name a few). In my mind, it's also about creating a website that will be found. A background in SEO helps, but I think Human Factors has been overlooked as an added bonus. For example, how do you know what keywords to pick unless you deeply understand the behavior of the user?

What you described is a scenerio I'd work up as a persona and apply as a test case to a website.

Just wanted to point out how useful it is to understand the true needs of Internet users.

Kim

#24 JimZim

JimZim

    Gravity Master Member

  • Members
  • 140 posts

Posted 03 January 2003 - 12:59 PM

Hi, all. Just moved this long post that was clearly off topic, NOW!, to the following location. Thanks for following and commenting if you have the time.

http://www.cre8asite...p?showtopic=668

#25 Guest_PhilC_*

Guest_PhilC_*
  • Guests

Posted 05 January 2003 - 04:16 PM

Dunno what URL you posted that was broken, Lots0, but this one works fine:- http://gooogle.searchking.com

Phil.



RSS Feed

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users