XtraBackup Manager - A couple of little features...

Just a quick check in… I just added a couple of things that I have found necessary as I'm testing out XtraBackup Manager.

You can now configure globally whether or not to automatically cleanup failed backups. Previously, XBM would always cleanup after itself on a failure.

I am finding that when things fail, that I would like a chance to investigate and troubleshoot why and perhaps open an XtraBackup bug or try some experimentation to see what might get around the problem.

I have been finding it particularly frustrating when waiting 9 hours for a multi-terabyte system to backup and then have some failure occur right at the end -- with the previous auto-cleanup I was left with nothing to troubleshoot with! Now I can turn off the cleanup and have a chance to do some forensics myself.

At the moment this is a quick and dirty feature -- it is only configurable at the global level, not per backup host.

The option is cleanup_on_failure and is found in /includes/config.php

The other feature that I added was the ability to configure how much memory XtraBackup Manager will tell XtraBackup to use when applying logs or merging incremental snapshots onto a full backup.

The default being used so far is 1G - so be careful about how much memory you have and consider how many possible concurrent backups you have configured.

This option is xtrabackup_use_memory and is also found in the config.php file as well as being a global feature only, not configurable per host.

These are not big features by any means, but they are certainly helpful for me and hopefully for others in the future.

That's it for today!

Lachlan

XtraBackup Manager - Movement on the home front...

It has been a while since I have posted any updates on the XtraBackup Manager front and I apologise for that. Between taking some time off for vacation (how dare I!?) and various different tasks at work snagging my focus away from XBM, I really haven't had much time to work on it.

(Un)fortunately last week we encountered a DB failure that would have been much faster and less painful to recover from had we had XtraBackup Manager finished and in place. While it was a pretty rough week for us DBAs working on addressing the failure, the silver lining is that we now have a concrete example to point to for the importance of the XtraBackup Manager project.

The silver lining in the long story cut short is that I now have the support I need to focus most of my time on XBM again.

So what have I been working on?

I have added support for materialzed backups to the "Continuous Incremental" backup strategy.

I have proceeded with actually running XBM against a few sample hosts with various schedules/settings to see what issues I may encounter.

I have posted a rough design outline in the Google Code wiki for the command-line interface for configuration and started on coding it.

My plan is to follow a similar design to the way the "zfs" command works on (Open)Solaris/Nexenta. 

You can see the design doc here:

http://code.google.com/p/xtrabackup-manager/wiki/DesignDoc

Once the CLI configurator is done, I'll proceed with some heavy documentation. After that point XBM should be pretty much ready for mass consumption in an evaluation capacity.

I have learned a lot about PHP and OO in the process of developing XBM, which has been fun, but I know code wise it isn't as elegant as it could be.

As I said when I started the project, I am not really a developer, so the internals of XBM probably aren't the cleanest code ever, but I'm doing my best while focussing on actually forging ahead to get it functional rather than getting too bogged down in how well the internals adhere to best OO design practise.

I'm hoping to get some more folks trying it out once the configurator and docs are up to snuff.

Stay tuned!

Lachlan

NFS Slowness Weirdness

We recently deployed a new NFS filer running on top of Nexenta using ZFS and noticed that some of our systems were having performance issues. Writes to the NFS system on most hosts were snappy, but reads from NFS were capping out at around 3M per second on a Gig-E network interface on a handful of hosts.

Systems that were identically configured in every way - kernel version, nfs package version, hardware, mount options, etc. were behaving differently. One would read and write at around 100M yet another would cap out at 3M.

Our systems guys did some troubleshooting and diagnosis on the network and could not find any issue there. So we went ahead and tested an scp from the NFS server to a problematic host.

The scp would run closer to 80M per second, so it seems that the problem was NFS itself.

We checked and double checked our configs, settings, sysctl.conf, versions and could not find anything different between a host where throughput was fine and one where it was horribly slow.

In the end we decided to umount all the NFS mounts, remove the "nfs" kernel module (rmmod) and reload it with modprobe and then remount the NFS mounts.

Lo and behold the throughput was back up to around 100M per second. This approach to fix the problem worked on all the problematic hosts we have tried it on so far.

Still,.. we are left scratching our head as to the real cause of the issue here as we basically have "jiggled the cable" or given NFS the "three fingered salute", if you will.

So though we now have a (rather intrusive) fix, we still don't know how to prevent the issue, if or when it will happen again, etc.

Has anyone out there seen anything similar to this before? Any ideas on what could be the issue?

Hmmm...

Lachlan

XtraBackup Manager - Backup Strategies and Materialized Snapshots

Hi Folks,

I have now committed the changes for the new Backup Strategies feature to trunk! In addition, I'm pretty much finished on implementing the majority of the Materialized Snapshot feature/option.

So let me talk a little bit about those features...

Enabling the "maintain_materialized_copy" feature for a backup will mean that while XBM takes FULL backups and INCREMENTAL backups and keeps them separately, it will maintain an additional directory that contains a complete backup with the latest deltas applied to it.

We only keep a materialized copy of the latest backup, not for each and every possible restore point as that would take up more space than most people can afford ( or at least more than we can afford ).

One benefit here is that if some problem should occur applying the latest set of deltas, you do not risk completely voiding your backup, you can always restore from the seed and deltas that are stored separately, up until the snapshot before the problem, and then perhaps use binary logs to roll forward from there.

Using materialized snapshots also means that you are constantly testing the process of actually applying your deltas, so if something was wrong with that step, you will learn about it quickly, not later on when you are desperately trying to restore from your backups.

Another great thing about materialized snapshots is that there is no waiting around for multiple sets of deltas to apply in order to restore your latest backup. Simply copy the materialized snapshot to the restore location and fire up MySQL -- InnoDB will of course take the usual time to do final crash recovery steps, but it should be much faster to get back up and running.

Now a little on Backup Strategies. There are three major strategies available and I'll talk a little on each below.

Full Backup Only

This is fairly cut and dry. XtraBackup Manager only takes full backups. You can configure how many is the maximum number of these snapshots to keep. After each backup is complete, the retention policy will be applied and any number of backups beyond the maximum will be deleted, counting from the latest to oldest. There is no need or option for materialized snapshots, since in this case all backups are always fully materialized.

Continuous Incremental

Take a full backup (aka seed) first and then after that only take incremental backups. The seed and deltas are all stored separately. Again you can configure the maximum number of snapshots to maintain (retention policy), however, in this case, we apply the oldest set of deltas onto the seed and repeat that process until we have no more than the maximum number of snapshots configured. The retention policy is always applied after a successful backup.

Rotating Groups


This is the most complex backup strategy, but it allows a great deal of flexibility. The concept here is that we consider a backup group as a full backup with corresponding sets of deltas. You may configure the number of groups you keep, as well as when new groups should be created in a variety of ways.

The benefit of keeping more than one group, is that should one seed or set of deltas be bad or broken in any way, you have another option -- in addition, you may more frequently take full backups, which means that the number of sets of deltas to be applied to reach a particular restore point will be less.

When using rotating groups, you must select a rotation method - there are two options: DAY_OF_WEEK and AFTER_SNAPSHOT_COUNT.

With the snapshot count rotation method, the first backup will be a FULL backup, after which incremental backups are taken until a total number of backups equals the number you configure. The next backup after that will be a full backup in a new group and the backups after that will be incrementals, based on the newly taken full backup. This cycle just repeats, however, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.

With day of week, you simply select which day(s) of week you would like your FULL backups to be taken on -- XBM will "rotate" on the first snapshot taken for that day. "Rotate" essentially means it will create a new group with its own full backup and then proceed to take deltas until a "rotate_day_of_week" is encountered again. You can also configure a maximum number of deltas allowed, in case for some reason the backup is never running on the day of the week that it should. In that case it will not backup - You may configure if you consider that a fatal error that should be alerted, or if it should just silently do nothing/skip that backup.

The benefit of using day of week over snapshot count is that it allows you to firmly control which days your full backups should happen on.

Eg. If I deploy backups on a new host and I configure to take full backups on Sunday. I might kick off the first backup on a Wednesday -- in this case because it is the first backup ever for the host, it will take a full backup and then continue to take deltas until Sunday, when it will take a full backup again and then continue to rotate every Sunday from then on.

Again for day of week rotation, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.

Now with all of these complex behaviours and options to configure and close to zero up-to-date documentation, I am about the only person who could make use of XBM, so the next steps are a better configuration tool/interface and documentation.

In addition, I'm also planning to add support for backing up the MySQL binary logs.

Once again, if you're interested in contributing in any way or just checking out the project, it is hosted on Google Code here:

  http://code.google.com/p/xtrabackup-manager/

Comments and feedback are welcome!

Cheers,
Lachlan

XtraBackup Manager - Wheels in motion!

Hi Folks,

I just realised that it has now been just a little over a month since I have posted anything regarding XtraBackup Manager!

Fear not friends, I have been working on the most significant changes and additions in XBM yet -- the addition of backup strategies.

With backup strategies you can get better control over when you want to take full backups and when you wish to take incremental backups.

I'll be making a more detailed post once I finish and push the code, but here is a little sneak preview of the kind of things it will do:

* Take full backups only, maintaining up to the last X backups.
* Take a full backup and then incrementals only, maintaining up to the last X backups
* Maintain N sets of backups, where each set has a full backup, followed by X incrementals.
* Rotate backup sets based on different rules like day of the week or after N successful backups.
* Choose which days of the week you want to take your full backups on and take incrementals on the rest!
* Control the number of sets of backups to keep -- old ones will be deleted.
* Keep a separate materialized copy of the most recent backup you took - No waiting to apply days of diffs in the event you need to restore!


The internal refactoring to support these features has made XBM much easier to build on, which is great news for those who may wish to either create their own patches or request features be added.

That's it for now, but keep your eyes peeled for more updates as I push towards a first official release!

Lachlan

XtraBackup Manager - Email Alerts and More Nexenta/ZFS Testing

Just a quick note - I have added support for email alerts on failed backups into XtraBackup Manager.

Now if something goes awry, XtraBackup Manager can optionally send some detailed information to the email address(es) of your choice!

This should allow you to easily hook to SMS gateways, NOC alert lists, etc.

Additionally, we have been doing some preliminary testing of XtraBackup Manager on a Nexenta machine with 8 Xeon CPUs using ZFS with lzjb compression.

We are getting around 3.8x compression and the Gigabit NICs seem to be the bottleneck in the speed here. Both CPU and disk utilisation look very low.

Our plan is to use a 10 Gigabit NIC in the Nexenta Backup host and stack it with storage - it should make for a very cost effective and space efficient backup host.

Meanwhile, development work continues towards a 1.0 !

Happy Friday!

Cheers,
Lachlan

XtraBackup Manager - Support for Nexenta (OpenSolaris)!

Hi Folks,

This is just a quick update to let you know that, after much cursing and frustration with my lack of Solaris experience, I have managed to make XtraBackup Manager work on Nexenta (NCP3).

So why would you care?

The answer is because Nexenta is OpenSolaris based and therefore has support for ZFS. I think ZFS is an awesome filesystem to combine with XtraBackup Manager, because you can benefit from transparent compression at the filesystem level.

This means you can store a whole lot more on less disk and you don't have to deal with compressing and uncompressing your backups all the time.

Sure, you could always use Nexenta Community or Enterprise appliances as a NFS mounted filer, but why would you want to stream all of your backup data into one Linux based server to run XtraBackup Manager just so that it all goes out an interface via NFS to the real storage?

This way you can run XtraBackup Manager right on the file server and save yourself network bandwidth and rackspace at the same time. Huzzah!

So far the feature set and documentation remain basic, but I will be continuing development as we want to get this tested so that we can use it ourselves in Production ASAP.

Once again, if you are interested in checking out the project you can find more at:

http://code.google.com/p/xtrabackup-manager

Feedback and comments are welcomed!

Lachlan