Howdy all,
Just a quick update in the world of XtraBackup Manager development. In the last couple of weeks I have not been doing a great deal of work on XtraBackup Manager itself, but rather doing a lot of testing of XtraBackup Manager and implicitly XtraBackup along with it.
I hit upon some bugs that were basically roadblocks in the way that we intend to use XtraBackup and I'm sure issues that other folks will run into eventually once adoption of XtraBackup increases even more...
I have been working with Percona and SkySQL Support, as well as dabbling in some of the code myself to get fixes for these issues.
The main issues we hit were:
* tar4ibd crashes on certain InnoDB data files (unable to use streaming backups at all) - This was a regression in pre-release build of xtrabackup-1.6.3, For now "fixed" by using an older tar4ibd binary from stable 1.6.2 release.
* innobackupex would not capture MySQL slave position unless using FLUSH TABLES WITH READ LOCK and performing a full backup. Now slave position can be captured in incrementals or full backups without locks, provided that --safe-slave-backup is specified.
* Tables getting both DROP/CREATE or TRUNCATE during backup can cause assertion failure - SkySQL contributed a fix via Monty Program which I am ready to test now.
* Xtrabackup apply-log can crash when attempting to create temporary tables if the temp dir does not exist - Should be fixed very soon in xtrabackup-1.6.3 release.
When I decided to embark on the project for XtraBackup Manager, I was happy to think that finally I'll be able to give something back to the community in the tool that I make. It seems what I didn't consider was that in being such a heavy integrator with XtraBackup that I'd also be contributing some good QA and perhaps improvements to XtraBackup itself.
As an aside, I also found a couple of little issues in Shlomi's online alter table in the OpenArk toolkit and submitted patches for that -- so I've felt very contributive lately.
So what lies ahead?
More testing of XtraBackup and XtraBackup Manager as well as finishing off the command-line configurator.
Then we will be preparing to start eating my dog food and run XBM in production!
That's it for now...
Have a great weekend all!
Cheers,
Lachlan
Friday, September 23, 2011
Thursday, September 1, 2011
XtraBackup Manager - Command-line Configurator Preview!
Over the past two weeks I have been working on XtraBackup Manager as much as I can and I'm pleased to say that the command-line configurator is coming along very nicely!
There is now a generic "xbm" command that will be the way to manage hosts, storage volumes and backup schedules as well as doing restores, etc.
So far I have built the volume and host management in and will begin work on the backup schedules next!
Once this command-line interface is complete, I'll work to document it on the project wiki on Google Code and it should be ready for public consumption.
Here is a little preview of how it looks in action - forgive the ugly wrapping…
There is now a generic "xbm" command that will be the way to manage hosts, storage volumes and backup schedules as well as doing restores, etc.
So far I have built the volume and host management in and will begin work on the backup schedules next!
Once this command-line interface is complete, I'll work to document it on the project wiki on Google Code and it should be ready for public consumption.
Here is a little preview of how it looks in action - forgive the ugly wrapping…
xbm@mslvlxbp01:~/xtrabackup-manager$ ./xbm XtraBackup Manager v0.5 - Copyright 2011 Marin Software Error: Context missing. Usage: xbm <context> <action> <args> ... Contexts and actions may be one of the following: volume [add|list|edit|delete] <args> -- Manage Backup Volumes host [add|list|edit|delete] <args> -- Manage Hosts to Backup backup [add|list|edit|delete] <args> -- Manage Scheduled Backup Tasks snapshot [list|delete] -- Manage Backup Snapshots restore [local|remote] <args> -- Restore Backups You may specify only a context, or a context and action to get help on its relevant arguments. xbm@mslvlxbp01:~/xtrabackup-manager$ ./xbm volumes XtraBackup Manager v0.5 - Copyright 2011 Marin Software Error: Action missing. Usage: xbm volumes <action> <args> ... Actions may be one of the following: add <name> <path> -- Add a New Backup Volume list -- List available Backup Volumes edit <name> <parameter> <value> -- Edit a Backup Volume to set <parameter> to <value> delete <name> -- Delete a Backup Volume You may specify an action without parameters to get help on its relevant arguments. xbm@mslvlxbp01:~/xtrabackup-manager$ ./xbm volumes list XtraBackup Manager v0.5 - Copyright 2011 Marin Software -- Listing all Backup Volumes -- Name: Storage Array 1 Path: /backup/xbm Name: Test /backup Path: /backup xbm@mslvlxbp01:~/xtrabackup-manager$ ./xbm volumes add MyVolume /tmp XtraBackup Manager v0.5 - Copyright 2011 Marin Software Action: New volume created with name/path: MyVolume -- /tmp
Friday, August 26, 2011
XtraBackup Manager - A couple of little features...
Just a quick check in… I just added a couple of things that I have found necessary as I'm testing out XtraBackup Manager.
You can now configure globally whether or not to automatically cleanup failed backups. Previously, XBM would always cleanup after itself on a failure.
I am finding that when things fail, that I would like a chance to investigate and troubleshoot why and perhaps open an XtraBackup bug or try some experimentation to see what might get around the problem.
I have been finding it particularly frustrating when waiting 9 hours for a multi-terabyte system to backup and then have some failure occur right at the end -- with the previous auto-cleanup I was left with nothing to troubleshoot with! Now I can turn off the cleanup and have a chance to do some forensics myself.
At the moment this is a quick and dirty feature -- it is only configurable at the global level, not per backup host.
The option is cleanup_on_failure and is found in/includes/config.php
The other feature that I added was the ability to configure how much memory XtraBackup Manager will tell XtraBackup to use when applying logs or merging incremental snapshots onto a full backup.
The default being used so far is 1G - so be careful about how much memory you have and consider how many possible concurrent backups you have configured.
This option is xtrabackup_use_memory and is also found in the config.php file as well as being a global feature only, not configurable per host.
These are not big features by any means, but they are certainly helpful for me and hopefully for others in the future.
That's it for today!
Lachlan
You can now configure globally whether or not to automatically cleanup failed backups. Previously, XBM would always cleanup after itself on a failure.
I am finding that when things fail, that I would like a chance to investigate and troubleshoot why and perhaps open an XtraBackup bug or try some experimentation to see what might get around the problem.
I have been finding it particularly frustrating when waiting 9 hours for a multi-terabyte system to backup and then have some failure occur right at the end -- with the previous auto-cleanup I was left with nothing to troubleshoot with! Now I can turn off the cleanup and have a chance to do some forensics myself.
At the moment this is a quick and dirty feature -- it is only configurable at the global level, not per backup host.
The option is cleanup_on_failure and is found in
The other feature that I added was the ability to configure how much memory XtraBackup Manager will tell XtraBackup to use when applying logs or merging incremental snapshots onto a full backup.
The default being used so far is 1G - so be careful about how much memory you have and consider how many possible concurrent backups you have configured.
This option is xtrabackup_use_memory and is also found in the config.php file as well as being a global feature only, not configurable per host.
These are not big features by any means, but they are certainly helpful for me and hopefully for others in the future.
That's it for today!
Lachlan
Tuesday, August 23, 2011
XtraBackup Manager - Movement on the home front...
It has been a while since I have posted any updates on the XtraBackup Manager front and I apologise for that. Between taking some time off for vacation (how dare I!?) and various different tasks at work snagging my focus away from XBM, I really haven't had much time to work on it.
(Un)fortunately last week we encountered a DB failure that would have been much faster and less painful to recover from had we had XtraBackup Manager finished and in place. While it was a pretty rough week for us DBAs working on addressing the failure, the silver lining is that we now have a concrete example to point to for the importance of the XtraBackup Manager project.
The silver lining in the long story cut short is that I now have the support I need to focus most of my time on XBM again.
So what have I been working on?
I have added support for materialzed backups to the "Continuous Incremental" backup strategy.
I have proceeded with actually running XBM against a few sample hosts with various schedules/settings to see what issues I may encounter.
I have posted a rough design outline in the Google Code wiki for the command-line interface for configuration and started on coding it.
My plan is to follow a similar design to the way the "zfs" command works on (Open)Solaris/Nexenta.
You can see the design doc here:
Once the CLI configurator is done, I'll proceed with some heavy documentation. After that point XBM should be pretty much ready for mass consumption in an evaluation capacity.
I have learned a lot about PHP and OO in the process of developing XBM, which has been fun, but I know code wise it isn't as elegant as it could be.
As I said when I started the project, I am not really a developer, so the internals of XBM probably aren't the cleanest code ever, but I'm doing my best while focussing on actually forging ahead to get it functional rather than getting too bogged down in how well the internals adhere to best OO design practise.
I'm hoping to get some more folks trying it out once the configurator and docs are up to snuff.
Stay tuned!
Lachlan
Thursday, July 7, 2011
NFS Slowness Weirdness
We recently deployed a new NFS filer running on top of Nexenta using ZFS and noticed that some of our systems were having performance issues. Writes to the NFS system on most hosts were snappy, but reads from NFS were capping out at around 3M per second on a Gig-E network interface on a handful of hosts.
Systems that were identically configured in every way - kernel version, nfs package version, hardware, mount options, etc. were behaving differently. One would read and write at around 100M yet another would cap out at 3M.
Our systems guys did some troubleshooting and diagnosis on the network and could not find any issue there. So we went ahead and tested an scp from the NFS server to a problematic host.
The scp would run closer to 80M per second, so it seems that the problem was NFS itself.
We checked and double checked our configs, settings, sysctl.conf, versions and could not find anything different between a host where throughput was fine and one where it was horribly slow.
In the end we decided to umount all the NFS mounts, remove the "nfs" kernel module (rmmod) and reload it with modprobe and then remount the NFS mounts.
Lo and behold the throughput was back up to around 100M per second. This approach to fix the problem worked on all the problematic hosts we have tried it on so far.
Still,.. we are left scratching our head as to the real cause of the issue here as we basically have "jiggled the cable" or given NFS the "three fingered salute", if you will.
So though we now have a (rather intrusive) fix, we still don't know how to prevent the issue, if or when it will happen again, etc.
Has anyone out there seen anything similar to this before? Any ideas on what could be the issue?
Hmmm...
Lachlan
Systems that were identically configured in every way - kernel version, nfs package version, hardware, mount options, etc. were behaving differently. One would read and write at around 100M yet another would cap out at 3M.
Our systems guys did some troubleshooting and diagnosis on the network and could not find any issue there. So we went ahead and tested an scp from the NFS server to a problematic host.
The scp would run closer to 80M per second, so it seems that the problem was NFS itself.
We checked and double checked our configs, settings, sysctl.conf, versions and could not find anything different between a host where throughput was fine and one where it was horribly slow.
In the end we decided to umount all the NFS mounts, remove the "nfs" kernel module (rmmod) and reload it with modprobe and then remount the NFS mounts.
Lo and behold the throughput was back up to around 100M per second. This approach to fix the problem worked on all the problematic hosts we have tried it on so far.
Still,.. we are left scratching our head as to the real cause of the issue here as we basically have "jiggled the cable" or given NFS the "three fingered salute", if you will.
So though we now have a (rather intrusive) fix, we still don't know how to prevent the issue, if or when it will happen again, etc.
Has anyone out there seen anything similar to this before? Any ideas on what could be the issue?
Hmmm...
Lachlan
Thursday, June 23, 2011
XtraBackup Manager - Backup Strategies and Materialized Snapshots
Hi Folks,
I have now committed the changes for the new Backup Strategies feature to trunk! In addition, I'm pretty much finished on implementing the majority of the Materialized Snapshot feature/option.
So let me talk a little bit about those features...
Enabling the "maintain_materialized_copy" feature for a backup will mean that while XBM takes FULL backups and INCREMENTAL backups and keeps them separately, it will maintain an additional directory that contains a complete backup with the latest deltas applied to it.
We only keep a materialized copy of the latest backup, not for each and every possible restore point as that would take up more space than most people can afford ( or at least more than we can afford ).
One benefit here is that if some problem should occur applying the latest set of deltas, you do not risk completely voiding your backup, you can always restore from the seed and deltas that are stored separately, up until the snapshot before the problem, and then perhaps use binary logs to roll forward from there.
Using materialized snapshots also means that you are constantly testing the process of actually applying your deltas, so if something was wrong with that step, you will learn about it quickly, not later on when you are desperately trying to restore from your backups.
Another great thing about materialized snapshots is that there is no waiting around for multiple sets of deltas to apply in order to restore your latest backup. Simply copy the materialized snapshot to the restore location and fire up MySQL -- InnoDB will of course take the usual time to do final crash recovery steps, but it should be much faster to get back up and running.
Now a little on Backup Strategies. There are three major strategies available and I'll talk a little on each below.
Full Backup Only
This is fairly cut and dry. XtraBackup Manager only takes full backups. You can configure how many is the maximum number of these snapshots to keep. After each backup is complete, the retention policy will be applied and any number of backups beyond the maximum will be deleted, counting from the latest to oldest. There is no need or option for materialized snapshots, since in this case all backups are always fully materialized.
Continuous Incremental
Take a full backup (aka seed) first and then after that only take incremental backups. The seed and deltas are all stored separately. Again you can configure the maximum number of snapshots to maintain (retention policy), however, in this case, we apply the oldest set of deltas onto the seed and repeat that process until we have no more than the maximum number of snapshots configured. The retention policy is always applied after a successful backup.
Rotating Groups
This is the most complex backup strategy, but it allows a great deal of flexibility. The concept here is that we consider a backup group as a full backup with corresponding sets of deltas. You may configure the number of groups you keep, as well as when new groups should be created in a variety of ways.
The benefit of keeping more than one group, is that should one seed or set of deltas be bad or broken in any way, you have another option -- in addition, you may more frequently take full backups, which means that the number of sets of deltas to be applied to reach a particular restore point will be less.
When using rotating groups, you must select a rotation method - there are two options: DAY_OF_WEEK and AFTER_SNAPSHOT_COUNT.
With the snapshot count rotation method, the first backup will be a FULL backup, after which incremental backups are taken until a total number of backups equals the number you configure. The next backup after that will be a full backup in a new group and the backups after that will be incrementals, based on the newly taken full backup. This cycle just repeats, however, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.
With day of week, you simply select which day(s) of week you would like your FULL backups to be taken on -- XBM will "rotate" on the first snapshot taken for that day. "Rotate" essentially means it will create a new group with its own full backup and then proceed to take deltas until a "rotate_day_of_week" is encountered again. You can also configure a maximum number of deltas allowed, in case for some reason the backup is never running on the day of the week that it should. In that case it will not backup - You may configure if you consider that a fatal error that should be alerted, or if it should just silently do nothing/skip that backup.
The benefit of using day of week over snapshot count is that it allows you to firmly control which days your full backups should happen on.
Eg. If I deploy backups on a new host and I configure to take full backups on Sunday. I might kick off the first backup on a Wednesday -- in this case because it is the first backup ever for the host, it will take a full backup and then continue to take deltas until Sunday, when it will take a full backup again and then continue to rotate every Sunday from then on.
Again for day of week rotation, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.
Now with all of these complex behaviours and options to configure and close to zero up-to-date documentation, I am about the only person who could make use of XBM, so the next steps are a better configuration tool/interface and documentation.
In addition, I'm also planning to add support for backing up the MySQL binary logs.
Once again, if you're interested in contributing in any way or just checking out the project, it is hosted on Google Code here:
http://code.google.com/p/xtrabackup-manager/
Comments and feedback are welcome!
Cheers,
Lachlan
I have now committed the changes for the new Backup Strategies feature to trunk! In addition, I'm pretty much finished on implementing the majority of the Materialized Snapshot feature/option.
So let me talk a little bit about those features...
Enabling the "maintain_materialized_copy" feature for a backup will mean that while XBM takes FULL backups and INCREMENTAL backups and keeps them separately, it will maintain an additional directory that contains a complete backup with the latest deltas applied to it.
We only keep a materialized copy of the latest backup, not for each and every possible restore point as that would take up more space than most people can afford ( or at least more than we can afford ).
One benefit here is that if some problem should occur applying the latest set of deltas, you do not risk completely voiding your backup, you can always restore from the seed and deltas that are stored separately, up until the snapshot before the problem, and then perhaps use binary logs to roll forward from there.
Using materialized snapshots also means that you are constantly testing the process of actually applying your deltas, so if something was wrong with that step, you will learn about it quickly, not later on when you are desperately trying to restore from your backups.
Another great thing about materialized snapshots is that there is no waiting around for multiple sets of deltas to apply in order to restore your latest backup. Simply copy the materialized snapshot to the restore location and fire up MySQL -- InnoDB will of course take the usual time to do final crash recovery steps, but it should be much faster to get back up and running.
Now a little on Backup Strategies. There are three major strategies available and I'll talk a little on each below.
Full Backup Only
This is fairly cut and dry. XtraBackup Manager only takes full backups. You can configure how many is the maximum number of these snapshots to keep. After each backup is complete, the retention policy will be applied and any number of backups beyond the maximum will be deleted, counting from the latest to oldest. There is no need or option for materialized snapshots, since in this case all backups are always fully materialized.
Continuous Incremental
Take a full backup (aka seed) first and then after that only take incremental backups. The seed and deltas are all stored separately. Again you can configure the maximum number of snapshots to maintain (retention policy), however, in this case, we apply the oldest set of deltas onto the seed and repeat that process until we have no more than the maximum number of snapshots configured. The retention policy is always applied after a successful backup.
Rotating Groups
This is the most complex backup strategy, but it allows a great deal of flexibility. The concept here is that we consider a backup group as a full backup with corresponding sets of deltas. You may configure the number of groups you keep, as well as when new groups should be created in a variety of ways.
The benefit of keeping more than one group, is that should one seed or set of deltas be bad or broken in any way, you have another option -- in addition, you may more frequently take full backups, which means that the number of sets of deltas to be applied to reach a particular restore point will be less.
When using rotating groups, you must select a rotation method - there are two options: DAY_OF_WEEK and AFTER_SNAPSHOT_COUNT.
With the snapshot count rotation method, the first backup will be a FULL backup, after which incremental backups are taken until a total number of backups equals the number you configure. The next backup after that will be a full backup in a new group and the backups after that will be incrementals, based on the newly taken full backup. This cycle just repeats, however, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.
With day of week, you simply select which day(s) of week you would like your FULL backups to be taken on -- XBM will "rotate" on the first snapshot taken for that day. "Rotate" essentially means it will create a new group with its own full backup and then proceed to take deltas until a "rotate_day_of_week" is encountered again. You can also configure a maximum number of deltas allowed, in case for some reason the backup is never running on the day of the week that it should. In that case it will not backup - You may configure if you consider that a fatal error that should be alerted, or if it should just silently do nothing/skip that backup.
The benefit of using day of week over snapshot count is that it allows you to firmly control which days your full backups should happen on.
Eg. If I deploy backups on a new host and I configure to take full backups on Sunday. I might kick off the first backup on a Wednesday -- in this case because it is the first backup ever for the host, it will take a full backup and then continue to take deltas until Sunday, when it will take a full backup again and then continue to rotate every Sunday from then on.
Again for day of week rotation, retention is controlled based on the maximum number of groups to maintain. Once a new group is created beyond the maximum allowed, the oldest group will be removed until there are no more than the maximum.
Now with all of these complex behaviours and options to configure and close to zero up-to-date documentation, I am about the only person who could make use of XBM, so the next steps are a better configuration tool/interface and documentation.
In addition, I'm also planning to add support for backing up the MySQL binary logs.
Once again, if you're interested in contributing in any way or just checking out the project, it is hosted on Google Code here:
http://code.google.com/p/xtrabackup-manager/
Comments and feedback are welcome!
Cheers,
Lachlan
Monday, June 13, 2011
XtraBackup Manager - Wheels in motion!
Hi Folks,
I just realised that it has now been just a little over a month since I have posted anything regarding XtraBackup Manager!
Fear not friends, I have been working on the most significant changes and additions in XBM yet -- the addition of backup strategies.
With backup strategies you can get better control over when you want to take full backups and when you wish to take incremental backups.
I'll be making a more detailed post once I finish and push the code, but here is a little sneak preview of the kind of things it will do:
* Take full backups only, maintaining up to the last X backups.
* Take a full backup and then incrementals only, maintaining up to the last X backups
* Maintain N sets of backups, where each set has a full backup, followed by X incrementals.
* Rotate backup sets based on different rules like day of the week or after N successful backups.
* Choose which days of the week you want to take your full backups on and take incrementals on the rest!
* Control the number of sets of backups to keep -- old ones will be deleted.
* Keep a separate materialized copy of the most recent backup you took - No waiting to apply days of diffs in the event you need to restore!
The internal refactoring to support these features has made XBM much easier to build on, which is great news for those who may wish to either create their own patches or request features be added.
That's it for now, but keep your eyes peeled for more updates as I push towards a first official release!
Lachlan
I just realised that it has now been just a little over a month since I have posted anything regarding XtraBackup Manager!
Fear not friends, I have been working on the most significant changes and additions in XBM yet -- the addition of backup strategies.
With backup strategies you can get better control over when you want to take full backups and when you wish to take incremental backups.
I'll be making a more detailed post once I finish and push the code, but here is a little sneak preview of the kind of things it will do:
* Take full backups only, maintaining up to the last X backups.
* Take a full backup and then incrementals only, maintaining up to the last X backups
* Maintain N sets of backups, where each set has a full backup, followed by X incrementals.
* Rotate backup sets based on different rules like day of the week or after N successful backups.
* Choose which days of the week you want to take your full backups on and take incrementals on the rest!
* Control the number of sets of backups to keep -- old ones will be deleted.
* Keep a separate materialized copy of the most recent backup you took - No waiting to apply days of diffs in the event you need to restore!
The internal refactoring to support these features has made XBM much easier to build on, which is great news for those who may wish to either create their own patches or request features be added.
That's it for now, but keep your eyes peeled for more updates as I push towards a first official release!
Lachlan
Subscribe to:
Posts (Atom)