Robert Kern » Hosting

Move data from one web server to another using wget

Robert Kern — Thu, 28 Aug 2008 16:00:01 +0000

Recently I needed to move a large amount of data from one web server to another. I was moving about 50 websites to a new server. I didnt want to download all the data via FTP and then upload it to the new server (especially not from here in Honduras where the Internet isnt as fast as I would have liked). I didnt have SSH access to the old server due to shared hosting security issues. How did I move it all in only a few minutes?

I had asked the host for a backup of my home directory which contained the data for all the sites. Due to some annoying permissions problems, the backup they provided me didnt contain all the files (which I discovered as I begun doing the scheduled transfer). The timezone differences meant that while I was working at a comfortable 8am, the time in NZ was around 2am. I couldnt ask for another backup and expect to have it in time, so I had to think of a better way.
Then I remembered wget. An amazing Unix tool that I use very often. I usually only use it for http requests, and had forgotten it could do ftp as well. So while I didnt have SSH access to the old server, I did have FTP access. I used the following wget command to log into FTP and download every single file under my home directory to the new server (which I did have SSH access to as it is a VPS that I own):

wget -r -N -l inf ftp://user:pwd@ftp.example.com/foldername

Within a few minutes (the two servers were both based in Auckland and the transfer speed was incredibly fast) all the data had been transferred across. I only had to re-organise the location of the data to the appropriate places, backup and import the MySQL databases to the new server and wa-la it was done. Thankfully, the 50-odd sites are running on the Frondiz CMS which means that the installation on the new server was very quick (ie, I didnt have to setup 50 different sites in apache, just a few, one that contains lots of domain names).

Frondiz CMS – Move to new server

Robert Kern — Tue, 29 Jul 2008 17:00:09 +0000

This is just a quick update regarding the move of the Frondiz CMS to the new server.

Unfortunately, the backup provided by the old host was not complete! A few sub-folders down there were no files. I ended up downloading all the files by using wget to log into the old server via ftp which proved to be a fast and reliable way of moving the files.

I placed a maintenance page up on the old server so that if anyone tries to access a site that is still running off the old server they will receive the maintenance message. This will only apply if a DNS lookup has been cached and is still pointing to the old server.

I have changed the DNS for all Frondiz CMS sites to point to the new server (excluding emails which will remain on the old server for now). I also setup a DNS monitor for every domain to notify me when the changes had taken place. They have all successfully rolled over now (except for one which is out of my control and I will be contacting them to remind them to make the change at their end).

Everything seems to be working just fine on the new server now and I dont anticipate any issues. If you do find any issues then please contact me ASAP.

Redirect to maintenance page

Robert Kern — Tue, 29 Jul 2008 16:53:08 +0000

I recently had to move a website to a new server. While the DNS changes rolled over, I didnt want people to be able to access the site on the old server as I had already taken a backup and didnt want any more changes to be made.

I used a .htaccess file to redirect any request on the old site to a maintenance page:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} !^/maintenance\.php$
RewriteRule ^(.*)$ /maintenance.php [R=307,L]

The above code redirects everything except the maintenance.php page to the maintenance.php page.

Its very useful to use scripts like this even when performing general maintenance on your site. You could get even trickier by allowing your IP address to access the site while still blocking everyone else. That way you can make sure your updates are working properly before ‘turning the site back on’.