Best Way To Move A Large Site To New Domain Options

[Kal]

Hi gang

A large client of mine is about to move their site (10,000+ pages) from their existing domain to a different one. The new domain is a generic keyword in their industry and so is better for both branding and recall. They want to take advantage of it by changing their site name and domain at the same time and start getting link juice from other sites linking to them with their new keyword name in the anchor text. They've owned the new domain for a couple of years and have currently got it parked to the same IP address as the existing site (parking has been in place for at least a year). What they want to do is swap the set-up so that the new domain is the main one and the old domain is parked to the same IP as the new one.

Now, I'd like your advice as to how they can best achieve this with the least amount of pain. I've recommended they use 301s (as per Google's advice) and asked them to get their programmers to implement it via mod re-write if possible. I've heard that it's better to move larger sites in 1 or 2 stages to aid Googlebot indexing. Anyone like to comment on that? They've also got some link popularity built up on their current domain and a toolbar PageRank of 4. Should they ask existing sites linking to them to change the anchor text to the new brand as well as the domain linked to? Anything to be aware of before flipping the switch? Thanks in advance.

[Kal]

No takers huh? I'm fairly confident of the advice I am giving to the client, but you should never assume anything in this industry and I still learn something every day. I was hoping to get some feedback from mods who've done this. I've helped clients move domains before, but never on this scale and never to a domain that has already aged, especially one that has been parked. If I've broken the forum protocols for asking advice or used the wrong topic or something, please let me know.

[A.N.Onym]

I don't think 10k pages counts as a large website. Just ensuring that 301 redirects work on a separate domain should be enough before embarking on the move, I think.

If you can, it'd help to have individual 301 redirects on main (traffic) pages, just in case.

Essentially, correctly redirecting all your visitors from the old pages to the new ones is the only thing you should be concerned about.

While you are moving, a couple of more things to remember:
- have a .htaccess file to redirect from either non-www version or www version of the site to another
- make sure you use only one version of the page, with or without a trailing slash (/) and redirect to that kind of pages. Linking to such pages using the same pattern (with or without /) is, of course, required.

Here's a couple of threads on the topic:
http://www.cre8asiteforums.com/forums/inde...showtopic=47812
http://www.cre8asiteforums.com/forums/inde...showtopic=46139

This post has been edited by A.N.Onym: May 15 2007, 07:23 PM

[phaithful]

Moving a site, large or small... you should employ the same tactics through out. This is a pretty high level pass, but here are a few tips.

1. Move all the content as is to the new domain. Although this seems like a perfect opportunity to redesign and change the site information architecture... if they are serious about protecting their rankings than you don't want to make too many changes at once.

2a. As you already anticipated... you'll want to 301 redirect your site into chunks. 301 redirect a subset of your pages that have a decent search ranking so you can monitor the transition. You might want to note how long it takes for the engines to re-index those pages to the new domain so you can plan future phases.

2b. It's not a good user experience to have parts of the site on one domain and part on another domain. So the suggestion here is a bit "gray hat" but in my experience I've never been penalized for this tactic.

What you'll do is copy the entire site onto the new domain. So you'll be running 2 duplicate sites in parallel for a short duration. Javascript redirect all your users to the new site (this is the gray part... you're in essence "cloaking"). The intent here is for the user experience and not deception.

During this time... the users will now see the new site, their bookmarks will work, and you won't have lost the search engine traffic.

Now in your phased 301 approach, 301 redirect parts of the site over to the new domain.

As a result you should see the URLs in the SERP slowly change to the new domain. You'll want to monitor all your important pages very closely. If pages that you haven't 301 redirected start slipping... that might be an indication that you're running a duplicate content risk. In this case it might be necessary to step up the transition plan to limit loss.
-------------

In most cases, regardless of the implementation you'll see a dip in traffic and then a recovery to the old levels. Sometimes this dip is as much as 60% (that's why you want to phase this approach). It's always a good idea to supplement the traffic with an SEM buy which will also help promote the new brand.

Note: This is very high level... there is a lot of planning that is involved when moving a large site from one domain to another while still trying to sustain the link equity.

[Kal]

Thank you very much. Ah, the trailing slash issue is a good point, I'll take that on board. Thanks also for those thread referrals. Yes, we want to redirect from each individual URL to the new one with the same content. Page URLs will be the same apart from the top domain. Can this be achieved on a global scale with mod_rewrite or some other way or does it have to be done page by page in .htaccess?

[AbleReach]

2a. As you already anticipated... you'll want to 301 redirect your site into chunks. 301 redirect a subset of your pages that have a decent search ranking so you can monitor the transition. You might want to note how long it takes for the engines to re-index those pages to the new domain so you can plan future phases.

Would the the xml site map of both versions of the site be the same? Meaning, would whatever had already been 301-ed to the new version appear in the new sitemap, and whatever had not yet been 301-ed would not be in the new site map?

Also, would you use nofollow, noindex in the metas of pages that are replaced by 301? Or in robots.txt? As the old pages were replaced in SERPs and by 301's would you remove pages from the old site's sitemap?

This post has been edited by AbleReach: May 15 2007, 08:27 PM

[A.N.Onym]

Kal, simple domain redirection without changing page URLs is simple and can be done with a couple of lines in .htaccess.

Wouldnt noindex and nofollow remove the pages from the index without moving the incoming links value through a 301, which is slow? No idea here myself,tho.

[phaithful]

Would the the xml site map of both versions of the site be the same? Meaning, would whatever had already been 301-ed to the new version appear in the new sitemap, and whatever had not yet been 301-ed would not be in the new site map?

Although I'm in that camp where I don't believe an XML site map should be submitted to engines... YES... the XML site map of the the new version would be identical to the old. I would even suggest moving over pages that you aren't going to keep around very long. Or at the very least 301 the link equity from those pages your getting rid of to similar topical pages that could benefit from the additional traffic (however marginal)...

Better yet... employ the same javascript redirect tactic to the new page you don't want on your site to a new location. And once the engines have properly redirect your entire site, then simply replace the javascript redirect to a 301 redirect.

Also, would you use nofollow, noindex in the metas of pages that are replaced by 301? Or in robots.txt? As the old pages were replaced in SERPs and by 301's would you remove pages from the old site's sitemap?

Once you 301 redirect the pages, nofollow and noindex won't have a role. The 301 redirect happens at the server level so the HTML page is never rendered to the search bot.

To put it simply, once you have a 301 redirect in place for a page, you can delete the old page. What happens is a request for the page comes into the server. The server has instructions to send a 301 redirect in place of the HTML file. At that point the robot sees the 301 redirect and proceeds to the new location without even seeing the old HTML file.

This goes doubly for robots.txt. You still want the bot to find the URL of the old page and then get redirected. If you block the robot via the robots.txt then you're essentially telling the robot to go away and not pass the link value. Remember.... even though you've told the robot the first time to 301 redirect... the robot may still arrive at those pages through other crawling entry points. So you don't want to block them via robots.txt

This post has been edited by phaithful: May 15 2007, 10:19 PM

[AbleReach]

To put it simply, once you have a 301 redirect in place for a page, you can delete the old page. What happens is a request for the page comes into the server. The server has instructions to send a 301 redirect in place of the HTML file. At that point the robot sees the 301 redirect and proceeds to the new location without even seeing the old HTML file.

Sometimes, after a 301, the old page stays around as supplemental, or the new page is supplemental for a while. My thought is that if both pages are out there and the new page is showing up pretty well, found a stable place in the SERPs, noindex in robots.txt might help a SE drop the old pages faster -- at least if I do that I feel better because I've given it a shot!

Agreed that a bot-related meta makes no sense on a page that has already been 301-ed to a new one.

You still want the bot to find the URL of the old page and then get redirected. If you block the robot via the robots.txt then you're essentially telling the robot to go away and not pass the link value.

Good point.

[DianeV]

They've owned the new domain for a couple of years and have currently got it parked to the same IP address as the existing site
...
What they want to do is swap the set-up so that the new domain is the main one and the old domain is parked to the same IP as the new one.

You've mentioned "moving" the website, although I'm not sure whether I'm taking the word "moving" too literally.

But if they share an IP address, then they're on the same server, so I'm wondering why the website has to be moved at all. If you move the "desirable" domain to a new server and copy the website there, you'd be giving it a new IP address (which some search engines don't always pick up on for some time but which may signal a big change to any or all of them — and that may not be what you want) and would be dealing with two websites, two web hosting accounts, etc. Instead, depending on the server setup, it may be possible to simply leave the website where it is, make the new domain the "main" domain, and redirect the old one to the new one. Then -presto!- even though you'd be changing the main domain name, there would be no change of IP, no additional web hosting account, no move, no duplicate pages, no loss of links, no loss of traffic.

Essentially, you'd just be making the desirable domain the "main" one, and parking the old one on top of that, which is done in the server's configuration file (httpd.conf -- see Ron Carnell's writeup at HR). However, you'd still have to deal with the one-for-one redirection of old to new URLs. What you don't want is to write out 10,000 301s in *any* file (because not only would that be what I'd call a mind-blowingly excruciating duty, but you'd likely slow down the server noticeably).

Instead, the redirection is best done with mod_rewrite, as you can generically make site1.com/whatever redirect to site2.com/whatever without having to specify the "whatever" — and mod_rewrite can be done in an .htaccess file.

<sigh> I believe there's a way to do mod_rewrite in the httpd.conf, but I've never been able to pull that off. (Ron?)

And, no matter which way you do it (.htaccess or httpd.conf), you ought also to deal with the www/non-www issues, etc.

If you're looking for a "least pain" situation, that's what I'd do. If there are other reasons why you need to move the website physically elsewhere (to a new IP or new server), then you'd need to take those into consideration.

<Sorry; I've just gotten a dedicated server and have moved any number of websites in the last few weeks to that and to VPSes (virtual private servers) at the same host, so I've kind of turned into server girl here. And now off I go to set up a new one.>

[phaithful]

Sometimes, after a 301, the old page stays around as supplemental, or the new page is supplemental for a while. My thought is that if both pages are out there and the new page is showing up pretty well, found a stable place in the SERPs, noindex in robots.txt might help a SE drop the old pages faster -- at least if I do that I feel better because I've given it a shot!

I'm not necessarily sure this is a bad thing to have both pages still existing in the index. Maybe from a branding point of view, if you wanted to get rid of the old domain because you didn't want to be associated with that brand. But many times when I've done this type of move, I'll see both sites start to rank on the first page which actually helps recoup some of that traffic I lose when I 301 redirect. Obviously you don't want them to rank very well for too long, or the engines will start noticing something fishy and supplemental both pages.

Another thing you might want to consider, Why you don't want to "hard-remove" pages from the index:

Even if the old pages are in supplemental, it shouldn't affect your new domain if it's ranking well. After you've done a site wide 301 redirect you should be receiving the same amount of traffic that you received on the old domain.

Now you have a perfectly good, well ranked old domain in your back pocket. Since all there are lots of pages still in the index (supplemental or not) this proves to increase the value of the domain, over a site that has no pages in the index. At some future time, you might want to resurrect the domain for another purpose, and you can then leverage some the value that still remains in the search engine as well as capture legacy links still pointing to those old pages (that's a whole other topic).

What you don't want is to write out 10,000 301s in *any* file (because not only would that be what I'd call a mind-blowingly excruciating duty, but you'd likely slow down the server noticeably).

Agreed... if you're still wanting to move the site in phases and not simply a site wide approach, you may want to look at how your site is currently structured. If it's product based you might just want to start with a category set (hopefully they are separated at the directory level example.com/product-category/) and write a regular expression moving that category and all subsequent pages within that directory. This will help you consolidate you 301 redirection list. If you're running Apache, mod_rewrite is appropriate and if you're running IIS you might want to look into an ISAPI filter.

[DianeV]

Good points, phaithful. What we don't know here is whether they want to disassociate the old domain from the new one.

[phaithful]

DianeV. I like your point about simply making a change to the httpd.conf and then modifing the .htaccess. It's a much simpler method than having to setup two sites on two different servers and then go through the hassle of "javascript cloaking".

Each time I've made a similar move, it was for branding purposes and the SAs always took it as an opportunity to improve the infrastructure. Hence, new servers and no option of httpd.conf

I guess in that case, if you wanted to 301 the site in chunks/phases you would have the old domain set to be a DNS alias so that both domains could render the same pages. But, you'll still have that UED disconnect where a portion of the pages would render as the old domain name until the user hit a page with an actual 301.

I guess that wouldn't be so terrible, but I'm sure marketing would have a fit

This post has been edited by phaithful: May 16 2007, 02:25 AM

[DianeV]

You're right -- unless they could do all the pages at once, marketing would indeed have a fit.

I also liked your idea of retaining the old domain for other purposes (unfortunately, we still don't know enough about what Kal's client wants to do to make completely informed suggestions).

[Kal]

This is really great feedback, thanks all. But my head hurts from thinking through the concepts!

[Kal]

At Search Summit in March this year, I asked Adam Lasnik from Google about the best way to switch domains and it was he that suggested doing it with 301s and in stages. But I wanted to hear from the men in the trenches, so to speak, to be aware of any potential problems with a large site. Anyway, I was pleased to find Adam's suggestions backed up in writing today on the Google Webmaster Blog: Musings on Down Under. The important bits:

* For each page on domain X, have it 301-redirect to the corresponding page on Y. (How? Typically through .htaccess, but check with your hosting provider).
* You might want to stagger the move, and redirect sub-sections of your site over time. This gives you the chance to keep an eye on the effects, and also gives search engines' crawl/indexing pipelines time to cover the space of redirected URLs.
* http://www.google.com/webmasters is your friend. Keep an eye on it during the transition to make sure that the redirects are having the effect you want.
* Give it time. How quickly the transition is reflected in the results depends on how quickly we recrawl your site and see those redirects, which depends on a lot of factors including the current reputation of your site's pages.
* Don't forget to update your Sitemap. (You are using Sitemaps, aren't you?)
* If possible, don't substantially change the content of your pages at the same time you make the move. Otherwise, it will be difficult to tell if ranking changes are due to the change of content or incorrectly implemented redirects.

Between you guys and this Google post, I can now advise the client with confidence!

[DianeV]

I note that he mentions "the move". Here, you may not actually be "moving" anything nor changing the IP address.

However, we've posted some questions above regarding why you might want to move the site elsewhere (or not), so it's difficult to answer your questions without knowing what you intend.

[A.N.Onym]

From what I have gathered, Kal simply is switching domains without actually moving files. It can very well be handled by 301 domain direct (any page from the old domain is redirected to the same URL on the new domain) either through httpd.conf settings (on a dedicated server) or through .htaccess, as mentioned above numerously.

[DianeV]

You're right. Issues get more complicated when you're actually moving a site from its present location.

[Kal]

Sorry Diane, must've missed your questions. Yes, Anonym is right - client is just switching domains on the same IP address. Making the parked one live and the live one parked. Looks like mod_rewrite in .htaccess is the go.