Move data from one web server to another using wget

Robert Kern — Thu, 28 Aug 2008 16:00:01 +0000

Recently I needed to move a large amount of data from one web server to another. I was moving about 50 websites to a new server. I didnt want to download all the data via FTP and then upload it to the new server (especially not from here in Honduras where the Internet isnt as fast as I would have liked). I didnt have SSH access to the old server due to shared hosting security issues. How did I move it all in only a few minutes?

I had asked the host for a backup of my home directory which contained the data for all the sites. Due to some annoying permissions problems, the backup they provided me didnt contain all the files (which I discovered as I begun doing the scheduled transfer). The timezone differences meant that while I was working at a comfortable 8am, the time in NZ was around 2am. I couldnt ask for another backup and expect to have it in time, so I had to think of a better way.
Then I remembered wget. An amazing Unix tool that I use very often. I usually only use it for http requests, and had forgotten it could do ftp as well. So while I didnt have SSH access to the old server, I did have FTP access. I used the following wget command to log into FTP and download every single file under my home directory to the new server (which I did have SSH access to as it is a VPS that I own):

wget -r -N -l inf ftp://user:pwd@ftp.example.com/foldername

Within a few minutes (the two servers were both based in Auckland and the transfer speed was incredibly fast) all the data had been transferred across. I only had to re-organise the location of the data to the appropriate places, backup and import the MySQL databases to the new server and wa-la it was done. Thankfully, the 50-odd sites are running on the Frondiz CMS which means that the installation on the new server was very quick (ie, I didnt have to setup 50 different sites in apache, just a few, one that contains lots of domain names).

Finding objects close to a location with MySQL

Robert Kern — Wed, 07 May 2008 01:38:10 +0000

For a project I’m currently working on, I needed to be able to find people that a within a certain distance from a particular point.

I have a large database of people that each have a Longitude and Latitude of their location. One of the searches in the app I am building needs to find people that are close to the location of the Job. So if the Job is in Kumeu (NZ), I want to see the people that are close (lets say 25km).

This is made easy with the Haversine formula. Google has a good article that gives a MySQL query that will produce a list of rows from the database where the locations are within a certain number of kilometres/miles.

The MySQL query looks like this:
SELECT *, (6371 * acos(cos(radians(174.556107)) * cos(radians(latitude)) * cos(radians(longitude) - radians(-36.775700)) + sin(radians(174.556107)) * sin(radians(latitude)))) AS distance FROM Locations HAVING distance < 25 ORDER BY distance
The location in the example above (-36.775700, 174.556107) is of Kumeu (NZ). The table currently has 4,500 records and the query takes about 0.0063 seconds (actually it is usually 0.0001 seconds).

Note that the above query calculates kilometres. If you are wanting miles, change the figure of 6371 to 3959.

The HAVING distance < 25 means that we are only getting the records who are within 25km of the original location.

Robert Kern » MySQL

Move data from one web server to another using wget

Finding objects close to a location with MySQL