The agile alternative to Ubuntu release upgrade

We had some Ubuntu 10.10 machines we wanted to upgrade to 12.04.1 LTS so I said let us give it a try in a clone to see how it goes before we work on the real servers.

After updating and upgrading 10.10 we proceed to a release upgrade:

do-release-upgrade

After running this command, going through some questions and restarting the box the system was upgraded to natty (11.04), not precise (12.04). Right there the problems started, mysql won't start, couchdb service would not listen and more. Running the command again did some progress and as you can see below after repeating the procedure 3 times I finally got the version I wanted (albeit several services will need to be tweaked to correctly work):

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.10
DISTRIB_CODENAME=maverick
DISTRIB_DESCRIPTION="Ubuntu 10.10"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS"

So this upgrade in phases, restarting again and again the machine, responding to questions etc demands too much human intervention for my taste, too much place for errors. Granted this would not be as bad if coming from 10.10LTS but yet ...

What is the alternative? Having your system scriptable, being able to reinstall the whole server from recipes. Plain Old Bash (POB) recipes would suffice and you can remotely deploy them using Remoto-IT.

Here are some advantages of rebuilding the box from scratch in comparison to upgrading the OS in the box:

1. Going beyond Disaster Recovery (DR): To recover even from total lost of snapshots (SQL data corruption in snapshots for example), or corrupted DR data. Just tape backups are enough
2. System Tests: As a real life exercise to test backup/restore procedure plus automated DevOps recipes for configuration and deployment management
3. Recipes update: Recipes might not actually work with newer versions of OS so with this process recipes are kept current or worst case scenario customized per OS release version
4. House keeping: From time to time IT guys will do manual stuff, keep big files where they should not, forget about cleaning up broken installations or simply can't afford the time it takes to correct broken dependencies. POB recipes on top of a brand new OS is a solution for that mess for sure
5. Performance: The bares minimum needed are guaranteed in the new fresh OS. That is not the situation for most of the servers that have seen several IT people tweaking them in many cases for several years
6. Reliability: Responding questions about the upgrade, manually hitting the button will always be less reliable than automated scripting. It is just the nature of manual intervention
7. Efficiency: Even today with virtualized environments you will spend more time upgrading the OS and dealing with potential problems than the time it will take to automatically deploy the necessary services and data in a brand new OS release
8. Security: Last but not least going with POB recipes makes sure there is no hidden malware (root-kit or not) deployed from an external or internal attacker. You simply get a cleaner, more reliable and secure system.

If you use an Ubuntu LTS distribution you need to do this every two years, not bad.