| Openstack compute node cleanup |
I've never used openstack before, which I imagine is similar to many other people out there. Its actually pretty cool, although I encountered a problem the other day that I think is worthy of some more documentation. Openstack runs virtual machines for users, in much the same manner as Amazon's EC2 system. These instances are started with a base image, and then copy on write is used to write differences for the instance as it changes stuff. This makes sense in a world where a given machine might be running more than one copy of the instance.
However, I encountered a compute node which was running low on disk. This is because there is currently nothing which cleans up these base images, so even if none of the instances on a machine require that image, and even if the machine is experiencing disk stress, the images still hang around. There are a few blog posts out there about this, but nothing really definitive that I could find. I've filed a bug asking for the Ubuntu package to include some sort of cleanup script, and interestingly that led me to learn that there are plans for a pretty comprehensive image management system. Unfortunately, it doesn't seem that anyone is working on this at the moment. I would offer to lend a hand, but its not clear to me as an openstack n00b where I should start. If you read this and have some pointers, feel free to contact me.
Anyways, we still need to cleanup that node experiencing disk stress. It turns out that nova uses qemu for its copy on write disk images. We can therefore ask qemu which are in use. It goes something like this:
$ cd /var/lib/nova/instances $ find -name "disk*" | xargs -n1 qemu-img info | grep backing | \ sed -e's/.*file: //' -e 's/ .*//' | sort | uniq > /tmp/inuse
/tmp/inuse will now contain a list of the images in _base that are in use at the moment. Now you can change to the base directory, which defaults to /var/lib/nova/instances/_base and do some cleanup. What I do is I look for large image files which are several days old. I then check if they appear in that temporary file I created, and if they don't I delete them.
I'm sure that this could be better automated by a simple python script, but I haven't gotten around to it yet. If I do, I will be sure to mention it here.
Tags for this post: openstack cloud computing nova glance qemu image management ubuntu canonical sre image_cache_management
Related posts: Reflecting on Essex; Folsom Dev Summit sessions; MySQL Workbench; Old ImageMagick packages in Debian and Ubuntu; Wow, qemu-img is fast; A first pass at glance replication; Further adventures with base images in OpenStack; Playing with StumbleUpon; Why document management is good; Managing MySQL the Slack Way: How Google Deploys New MySQL Servers; Being Geek; I won a radio shark and headphones!; Over-analysed Friday comments on Half Nekkid Thursday; Conference Wireless not working yet?; Nice touch; Taking over a launch pad project; Off to the MySQL tutorials; What packages are taking all my disk?; Perl sample source code; Links from Rasmus' PHP talk; Installing Linux on a Dell Dimension e310
posted at: 00:59 | path: /openstack | permanent link to this entry
Comment on this post.