How we do migrations

One of the tasks we, as a hosting company has to do every now and then, is to migrate accounts between machines. This can either be due to balancing resource utilization between systems or things such as hardware or operating system upgrades.

At Hosting4Real we use cPanel/WHM extensively for our control panel offering, cPanel/WHM comes with a tool called "Transfer Tool" which makes transfers between servers easier, but there's still a few problems when using transfer tool, one of these problems are the speed of which the transfers happen. The actual downtime per site is rather minimal, but when having to migrate a whole server, we ideally want to minimize the time spent on migrations.

This post is a broad overview of how we handle account migrations between servers and how we minimize the time between accounts when we have to do bulk migrations.

How "Transfer Tool" in cPanel/WHM Works

Transfer Tool in cPanel/WHM has two main steps when transferring an account:

Package the account on the source server

cPanel uses a tool called pkgacct to generate an so-called cpmove file, this file is an tar.gz archive containing a basic file structure with the following information:

• Reseller configuration
• SSL Certificates and Keys
• Domain Keys (DKIM)
• Bandwidth data
• DNS zones
• Mail configuration (users and passwords)
• FTP configuration (users and passwords)
• virtualhost templates
• mailman lists
• Database users
• Cronjobs, quotas, API Tokens, DNSSEC information

After the cpmove file has been generated, it will then sync the cpmove file to the destination server.

Extract the account on the destination server

On the destination the cpmove file will get restored using restorepkg, which restores the cPanel account with the above information.

What then happens next, is streaming the home directory (web files and emails) from the source server to the destination using rsync.

After the home directory has been streamed, it will create a websocket stream to dump the databases over the network (this happens in a secure tunnel), the databases are then restored.

Eventually the account gets fully restored, it updates a few settings on the source server such as the IP address on the DNS configuration to point to the new server.

It then performs yet another resync of the files for anything that might have updated in the meantime and then in the end triggers restarts of various software components.

So what's the problem here?

The main problem in the above is the initial synchronization of the home directory, because the time we spend on this step in the migration heavily depends on the number of files, the size of files and also the network speed between two systems.

For accounts that are less than 5 gigabyte, such migrations can often be done at the speed of roughly 80 megabytes per second, so a synchronization will take about 1-2 minutes, which is completely acceptable - however, the issue happens when we either have millions of files for an account (Looking at you Prestashop), or the size of an account are 20, 50, 80 or even 120 gigabyte in size, this suddenly increases the migration time a fair amount per account, slowing down the overall process when migrating accounts in bulk.

If we take a 50 gigabyte account as an example, this will most likely take 10-15 minutes to synchronize, if we just have a few of these accounts to migrate, we quickly end up wasting a lot of time just waiting for the initial synchronization to happen - so we worked out a different way of migrating things!

How we speed up the final migrations

We still use Transfer Tool to complete the actual migration, however, prior to the migration taking place, we're syncing all accounts above 5 gigabyte across.

For bulk migrations, we add a file in /home called users with the content as such:

user1
user2
user3
user4
...

We then perform the following command:

$ cat users | while read cpuser; do rsync --exclude */files/tmp/sess_* \
--exclude lscache/* \
--exclude */cache/smarty/cache/* \
--exclude .trash/* \
--exclude softaculous_backups/* \
--exclude */cache/smarty/compile/* \
-e 'ssh -p831' \
-av --delete \
--no-owner --no-group $cpuser/ \
wheeluser@nlcpXX.hosting4real.net:/home/$cpuser-tmp; done

The above command does the following:

• We read each line in the file users as cpuser
• Then execute rsync over SSH port 831 (the SSH port we use)
• Exclude a few folders containing useless data for a migration ta
• Delete files on destination that does not exist on the source using --delete
• We do not transfer the owner and group
• Accounts are moved to $cpuser-tmp in the home directory on the destination server

The above command will synchronize all accounts in the users file one by one, but we can put it in a screen session, so we don't have to monitor it.

The initial synchronization usually takes 1-2 hours depending on the number of accounts and the amount of data that has to be synchronized - however, doing a new synchronization of all accounts afterwards will usually only take a few minutes, since rsync will simply has to calculate the difference between the source and destination.

Doing the actual migration:

The actual migration boils down to a few actions:

• Do the synchronize command only targeting a single account
• Use "Transfer Tool" to transfer the account but without the home directory (Since it's already in /home/$cpuser-tmp)
• Execute a few commands on the destination:

$ mv /home/$cpuser /home/$cpuser-bk
$ mv /home/$cpuser-tmp $cpuser
$ bash ~/scripts/fixperms.sh $cpuser
$ cagefsctl --disable $cpuser
$ cagefsctl --enable $cpuser

We move the empty home directory that Transfer Tool created on restoration, we then move our $cpuser-tmp into $cpuser, we use our fixperms.sh script to fix the file permissions based on what cPanel/WHM expects.

The very last step we take is to disable and enable cagefs for the cPanel user that has just been migrated, what this does, is simply resetting anything related to cagefs.

So what did we achieve?

While the migration process takes slightly longer for us, due to a bit of extra planning required for this migration, we lower the amount of time spent during actual migrations to a few minutes per account (The biggest time spent is usually database restorations, so accounts with bigger databases will usually take a bit longer).

This benefits our customers since we're able to push migrations through faster, while it does require some manual intervention for us, we continue to work towards getting the majority of it automated, while keeping it safe!

We simply wanted to give a bit of insight into how we do migrations.