Content here is by Michael Still All opinions are my own.
See recent comments. RSS feed of all comments.

Tue, 29 Dec 2015

A year of being more active

    A little over a year ago I helped host a panel session at the Paris OpenStack summit. The panel was three telco people talking about NFV, but that's not the bit I really remember. The bit that really stuck with me was how fat I looked in photos from the event. An example:

    I weighed just over 110 kilos (240 lbs). I decided something had to change -- I have a new daughter, and I want to be around to see her well into her life. So, I joined a gym and started bush walking. My first walk was documented here as a walk up Tuggeranong hill. That rapidly became an obsession with climbing hills to survey markers, which then started to include finding geocaches.

    I can't give you a full list of the tangents that one photo from Paris has caused, because the list isn't complete yet and may never be. I now run, swim, ride my bike, and generally sweat on things. Its all fun and has had the unexpected side effect that its helped me cope with work stress much more than I expected.

    I've lost about 15 kilos (30 lbs) so far. Weight loss isn't really the main goal now, but its something I continue to track.

    I thought it would be interesting to list all the places I've walked in Canberra this year, but a simple bullet point list is too long. So instead, here's an interactive map.

    There are a lot more walks I want to do around here. Its just a case of finding the time.

    Tags for this post: blog fitness health weight canberra bushwalk
    Related posts: Recumbent bikes; It hasn't been a very good week; Drugs; Bigger improvements; On running; Memorial service details

posted at: 17:34 | path: /diary | permanent link to this entry

Mon, 28 Dec 2015

Mount Tennent

      Ok, so I want to do Mount Tennent from the "wrong" side (Apollo Road up the fire trail). The reason for this is that there is a series of 20 something geocaches along the route that I'd like to tick off as well as walking to Tennent.

      I think the total route should be about 13kms, with about 800m of ascent. I propose we leave at least one car at the Apollo Road car park, and then drive the rest to the start point at the Namadgi visitor centre. We can then do the walk, and send one car back to collect the others while we wait under a tree (or something like that).

      Naismith's rule says this walk should take about 5 hours.

    I've been meaning to do this one for ages, but finally got around to doing the walk with a few friends. We went up Tennent from the Namadgi visitor's center, but walked back down on the far side of Tennent which seems less common. The far side is less scenic, but less steep as well I think. Along the way we collected 23 geocaches along the way, and a lovely walk was had by all.

    Just over 1,000 meters vertically, and around 16km horizontally.


    Tags for this post: blog pictures 20151228 photo canberra bushwalk

posted at: 12:28 | path: /diary/pictures/20151228 | permanent link to this entry

Wed, 23 Dec 2015

Mulligan's flat

    I went for a post lunch geocache walk in Mulligan's flat this afternoon as I wind down towards the Christmas break. A good time was had, and a cool wool shed found. I really like this area, perfect for cub walks!


    Tags for this post: blog pictures 20151223 photo canberra bushwalk

posted at: 12:43 | path: /diary/pictures/20151223 | permanent link to this entry

Tue, 22 Dec 2015

Street Orienteering

    I went street orienteering for the first time with my little brother the other day. It was super fun, and a good way to get a run in as well. We even came first in our division (team b grade)!

    Tags for this post: blog orienteering
    Related posts: Scout activity: orienteering at Mount Stranger

posted at: 15:13 | path: /diary | permanent link to this entry

Thu, 03 Dec 2015


posted at: 22:54 | path: /book/Kellie_Arrowsmith | permanent link to this entry

The Chronicles of Old Guy

    I found this e-book on Amazon while randomly poking around and read it on a recent set of flights. It was of interest because if looked like another bolo tank book, of which I have read many over the years. That said, its not in strictly the same universe as the other bolo books, and seems more like unofficial fan fiction than something which maps into the universe seamlessly.

    The book is competently written and readable. However, it regularly strays into what I would consider fantasy fiction (medieval warfare, vampires, battling Godzilla) in a way I found jarring and annoying. Overall I don't think I'll read the other books in this series.

    Tags for this post: book timothy_j_gawne bolo combat ai
    Related posts: Ancillary Justice; Bolos 3: The Triumphant; Consider Phlebas; Their Finest Hour; The Compleat Bolo; The Road to Damascus

posted at: 12:18 | path: /book/Timothy_J_Gawne | permanent link to this entry

Sun, 15 Nov 2015

Mount Stranger one last time

    This is the last walk in this series, which was just a pass through now that the rain has stopped to make sure that we hadn't left any markers or trash lying around after the Scout orienteering a week ago. This area has really grown on me -- I think most people stick to the path down by the river, whereas this whole area has nice terrain, plenty of gates through fences and is just fun to explore. I'm so lucky to have this so close to home.

    Tags for this post: blog canberra bushwalk

posted at: 12:20 | path: /diary | permanent link to this entry

Mon, 09 Nov 2015

A walk in the Orroral Valley

    Last weekend was a walk in the Orroral Valley with a group of scout leaders. Embarrassingly, I'd never been in this area before, and its lovely -- especially at the moment after all the rain we've had. Easy terrain, and a well marked path for this walk. The only catch is that there's either a car shuffle involved, or you need to do a 12km return walk.


    Tags for this post: blog pictures 20151107 photo canberra bushwalk

posted at: 19:13 | path: /diary/pictures/20151107 | permanent link to this entry

Sun, 08 Nov 2015

Scout activity: orienteering at Mount Stranger

    I've run scout activities before, but its always been relatively trivial things like arranging attendance at a Branch level event such as an astronomy night or an environment camp. They've involved consent forms and budgeting and so forth, but never the end to end creation of a thing from scratch. So, I was quite excited to be presented with an opportunity to take the scouts orienteering in an unfamiliar environment.

    I chose the area of nature reserve between Mount Stranger and the Murrumbidgee River because its nice terrain (no tea tree!), but big enough for us to be able to do some long distance bearing navigation, which is a badge requirement some of the scouts are working on at the moment.

    The first step was to scout out (pun intended) the area, and see what sort of options there are for controls and so forth. I'd walked through this area a bit before, as its close to my house, but I'd never bush bashed from the river to the trig before. The first attempt was a simple marking off of the gates along the bicentennial horse trail -- I knew we'd want to cross this somewhere for the long distance leg. That route looked like this:

    The next recce was a wander along a candidate route with some geocaching thrown in for good luck. The geocaching turned out to be quite useful, because on the actual night with the scouts it meant I had a better handle of what was in the area, so when a couple of girls started losing interest I could say stuff like "Did I forget to mention there's an awesome tree house just over there?".

    With that in mind, I then just started slogging out a route -- the long distance leg turned out to be the hardest part here. I wanted to avoid fence crossings as much as possible, and this whole area is littered with barbed wire fences. I think I redid that leg four times before I found a route that I was happy with, which was ironically the first one I'd tried.

    Job done! Now I only needed to walk this route three more times! The first walk was to lay out the orienteering markers before the scouts attacked the course:

    ...and then actually doing the course with some scouts...

    Comparing the two maps, I don't think they did too bad to be honest. There's definitely potential here for more navigation practise, but I think the key there is that practise makes perfect. There shall be more hiking and orienteering in our future! The final walk was just collecting the markers after the event, which I will skip here.

    I put a fair bit of effort into this course, so I'd like to see it used more than once. To that end, I am going to put the documentation online for others to see and use. If you'd like help running this course, drop me a line at and I'd be happy to help.

    Tags for this post: scouts orienteering navex
    Related posts: Light to Light, Day Three; Light to Light, Day One; Exploring the Jagungal; Street Orienteering; Light to Light, Day Two; Potato Point

posted at: 15:40 | path: /scouts | permanent link to this entry

Mon, 02 Nov 2015

Halo: The Fall of Reach

    ISBN: 0765367297
    As someone who doesn't play computer games and has never played a Halo game, I find myself in the strange position of having read a Halo book. This book is the first in the chronological lineage, and explains the history of the Spartan program which produced the Master Chief. I decided to read this after accidentally watching a Halo mini-movie on Netflix with a sick baby, and deciding it wasn't totally terrible.

    The book is actually ok to my surprise. Its competently written, and on par with much of the other combat fiction I've read. It certainly doesn't feel like its a tie in to a game. I would have liked this book to cover more of the moral issues around the back story to the Spartan program, but those were only briefly considered. Then again, I like a good shoot 'em up as much as the next guy and perhaps that would have been too boring. Overall I enjoyed it and think I might have to read more in this universe.

    Tags for this post: book eric_nylund combat halo engineered_human cranial_computer personal_ai aliens
    Related posts: Halo: The Flood; The Ghost Brigades (2); Old Man's War ; Old Man's War (2); The Ghost Brigades ; The End of All Things

posted at: 02:50 | path: /book/Eric_Nylund | permanent link to this entry

Fri, 16 Oct 2015

CBC Navigation Course

    This was the GPS followup to the previous map and compass navigation exercise. A really nice walk, apart from crazy horse lady. The walk also included another visit to Forster trig. I'm not sure if its the time of year or the direction of approach, but this ascent was much nicer than my previous one, we seemed to avoid most of the prickly things. It would be interesting to recce the other side of the hill and see if I just got unlucky last time, or misread the contours.


    Tags for this post: blog pictures 20151010 photo canberra bushwalk trig_point
    Related posts: One Tree and Painter; Harcourt and Rogers Trigs; Tuggeranong Hill (again); Wanniassa Trig; Quartz trig; Big Monks

posted at: 03:30 | path: /diary/pictures/20151010 | permanent link to this entry

Thu, 15 Oct 2015

Geocaching at the base of Mount Ainslie

    I have a bit of a backlog of posts about recent walks which I am working through. Last week I found myself in Ainslie for an appointment which ended at lunch time, so I figured I'd go for a walk. There is a series of geocaches near Campbell Park West (a set of office buildings for non-Canberrans), so off I went.

    The cache series was nice, but the most exciting part of the walking in this area was all the unexploded ordinance (UXO) warnings. I'm sure the area is totally safe, as many people walk through it each day, but it certainly adds an air of adventure to the walk.

    You can find a list of the UXO reports for the ACT on the Department of Defence website, I must say that its not very impressive that the Department has contaminated so many sites around Australia without remediating them -- there are heaps in New South Wales for example. The ACT gets off relatively lightly with only three contaminated sites.

    Its also interesting to note that a suburb very close to me was used as a bombing practise range in World War Two. I'm not aware of anyone in my circle who knew that.

    Anyway. Nice terrain, nice caches, lots of fun. I'd say this would be a good walk for cubs, but I am sure the risk management paperwork for a walk in a UXO are is complicated.


    Tags for this post: blog pictures 20151009 photo canberra bushwalk

posted at: 16:40 | path: /diary/pictures/20151009 | permanent link to this entry

Wed, 07 Oct 2015

Lunchtime geocaching

    So, I didn't get to sleep last night until 4:30am because of a combination of work meetings and small children, so today was a pretty weird day for me. I was having a lot of trouble concentrating at lunch time, so I decided a walk was the least worst thing I could do with my time. I decided to knock off some of the few remaining geocaches in southern Tuggeranong that i haven't found yet.

    This walk was odd -- it started and ended in a little bit of Theodore they never got around to actually building, and I can't find any documentation online about why. It then proceeded through a nice little green strip that has more than its share of rubbish dumped, Cleanup Australia needs to do a visit here! Then there were the Aboriginal axe grinding grooves (read more) just kind of in the middle of the green strip with no informational signage or anything. Finally, a geocache at an abandoned look out, which would have been much nicer if it wasn't being used as an unofficial dump now.

    That said, a nice little walk, but I have no real desire to revisit this one any time soon.


    Tags for this post: blog pictures 20151007 photo canberra bushwalk

posted at: 00:58 | path: /diary/pictures/20151007 | permanent link to this entry

Thu, 01 Oct 2015

A searchable database of walk waypoints

    Over the last year I've become increasingly interested in bush walking, especially around the ACT. It quickly became evident that John Evan's site is an incredibly valuable resource, especially if you're interested in trig points or border markers.

    However, I do most of my early walk planning and visualization in Google Earth before moving to Garmin Basecamp to generate walkable maps. I wanted a way to hook John's database of GPS logs into Google Earth, so that I could plan walks more effectively. For example, John often marks gates in fences, underpasses under major roads, and good routes through scrub in his GPS tracks.

    After a fair bit of playing, I ended up with this KML file which helps me do those things. Its basically magic -- the file is just a link to a search engine which has a database of GPS waypoints based off walks John and I have logged. These are then rendered in Google Earth as if they were in a static KML file. You can also download the search results as KML for editing and so forth as well.

    So, I'd be interested in other people's thoughts on if this is a useful thing. I'd also be very interested in other donated GPS logs of walks and bike rides around Canberra, especially if they have waypoints marked for interesting things. If you have any comments at all, please email me at

    Tags for this post: walks gps search google earth
    Related posts: A busy couple of days; Solar panel reflection effects in satellite imagery; Technorati search results polluted by porn blogs; Destinator 3 GPS navigation for the PocketPC; Google book search; Sensis Australian search

posted at: 14:59 | path: /walks | permanent link to this entry

Garran green strip

    When I was a teenager my best mate lived in a house which backs onto this smallish reserve and we used to walk his dog here heaps. I had a few spare moments yesterday, so I was keen to do a quick explore and see what its like now. The short answer is that its still nice -- good terrain, nice mature trees, and a few geocaches. I think this one would be a good walk for cubs.


    Tags for this post: blog pictures 20151001 photo canberra bushwalk

posted at: 14:35 | path: /diary/pictures/20151001 | permanent link to this entry

Tue, 29 Sep 2015

Wandering around Curtin

    I decided to go on a little walk on the way home from a work lunch and I don't regret it. This is a nice area, which I was exploring for geocaches. I probably wouldn't have come here at all, but it was the second part of the "Trees of Curtin" walk from Best Bush, Town and Village Walks in and around the ACT that I had done the first half of ages ago.

    I am glad I came back for the second half -- to be honest I was pretty bored with the first half (a bike path beside a major road mostly), whereas this is much more like walking around in nature. The terrain is nice, no thistles, and plenty of horses. A nice afternoon walk overall.

    Now back to reviewing Mitaka specs.


    Tags for this post: blog pictures 20150930 photo canberra bushwalk

posted at: 23:23 | path: /diary/pictures/20150930 | permanent link to this entry

Second trail run

    I went for my second trail run last night. This one was on much rockier terrain, and I ended up tweaking my right knee. I think that was related to the knee having to stabilize as I ran over uneven rocks. I'll experiment by finding a different less awkward trail to run and seeing what happens I suppose.

    Tags for this post: blog canberra trail run
    Related posts: First trail run; Chicken run; Boston; Update on the chickens; Random learning for the day

posted at: 15:06 | path: /diary | permanent link to this entry

Mon, 28 Sep 2015

Old Joe and Goorooyarroo

    Steve, Mel, Michael and I went for a walk to Old Joe trig yesterday. I hadn't been to Goorooyarroo at all before, and was quite impressed. The terrain is nice, with some steep bits as you get close to the border (its clear that the border follows the water catchment from a walk around here). Plenty of nice trees, not too many thistles, and good company. A nice morning walk.

    We bush bashed to the trig straight up the side of the hill, and I think there were gentler (but longer) approaches available -- like for instance how we walked down off the hill following the fence line. That said, the bush bash route wasn't terrible and its probably what I'd do again.

    I need to come back here and walk this border segment, that looks like fun. There are also heaps of geocaches in this area to collect.


    Tags for this post: blog pictures 20150928 photo canberra bushwalk

posted at: 15:34 | path: /diary/pictures/20150928 | permanent link to this entry

Wed, 23 Sep 2015

How we got to test_init_instance_retries_reboot_pending_soft_became_hard

    I've been asked some questions about a recent change to nova that I am responsible for, and I thought it would be easier to address those in this format than trying to explain what's happening in IRC. That way whenever someone compliments me on possibly the longest unit test name ever written, I can point them here.

    Let's start with some definitions. What is the difference between a soft reboot and a hard reboot in Nova? The short answer is that a soft reboot gives the operating system running in the instance an opportunity to respond to an ACPI power event gracefully before the rug is pulled out from under the instance, whereas a hard reboot just punches the instance in the face immediately.

    There is a bit more complexity than that of course, because this is OpenStack. A hard reboot also re-fetches image meta-data, and rebuilds the XML description of the instance that we hand to libvirt. It also re-populates any missing backing files. Finally it ensures that the networking is configured correctly and boots the instance again. In other words, a hard reboot is kind of like an initial instance boot, in that it makes fewer assumptions about how much you can trust the current state of the instance on the hypervisor node. Finally, a soft reboot which fails (probably because the instance operation system didn't respond to the ACPI event in a timely manner) is turned into a hard reboot after libvirt.wait_soft_reboot_seconds. So, we already perform hard reboots when a user asked for a soft reboot in certain error cases.

    Its important to note that the actual reboot mechanism is similar though -- its just how patient we are and what side effects we create that change -- in libvirt they both end up as a shutdown of the virtual machine and then a startup.

    Bug 1072751 reported an interesting edge case with a soft reboot though. If nova-compute crashes after shutting down the virtual machine, but before the virtual machine is started again, then the instance is left in an inconsistent state. We can demonstrate this with a devstack installation:

      Setup the right version of nova cd /opt/stack/nova git checkout dc6942c1218279097cda98bb5ebe4f273720115d Patch nova so it crashes on a soft reboot cat - > /tmp/patch <<EOF > diff --git a/nova/virt/libvirt/ b/nova/virt/libvirt/ > index ce19f22..6c565be 100644 > --- a/nova/virt/libvirt/ > +++ b/nova/virt/libvirt/ > @@ -34,6 +34,7 @@ import itertools > import mmap > import operator > import os > +import sys > import shutil > import tempfile > import time > @@ -2082,6 +2083,10 @@ class LibvirtDriver(driver.ComputeDriver): > # is already shutdown. > if state == power_state.RUNNING: > dom.shutdown() > + > + # NOTE(mikal): temporarily crash > + sys.exit(1) > + > # NOTE(vish): This actually could take slightly longer than the > # FLAG defines depending on how long the get_info > # call takes to return. > EOF patch -p1 < /tmp/patch restart nova-compute inside devstack to make sure you're running the patched version... Boot a victim instance cd ~/devstack source openrc admin glance image-list nova boot --image=cirros-0.3.4-x86_64-uec --flavor=1 foo Soft reboot, and verify its gone nova list nova reboot cacf99de-117d-4ab7-bd12-32cc2265e906 sudo virsh list ...virsh list should now show no virtual machines running as nova-compute crashed before it could start the instance again. However, nova-api knows that the instance should be rebooting... $ nova list +--------------------------------------+------+---------+----------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+---------+----------------+-------------+------------------+ | cacf99de-117d-4ab7-bd12-32cc2265e906 | foo | REBOOT | reboot_started | Running | private= | +--------------------------------------+------+---------+----------------+-------------+------------------+ start nova-compute again, nova-compute detects the missing instance on boot, and tries to start it up again... sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' \ > & echo $! >/opt/stack/status/stack/; fg || \ > echo "n-cpu failed to start" | tee "/opt/stack/status/stack/n-cpu.failure" [...snip...] Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/", line 444, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/", line 213, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/", line 728, in save columns_to_join=_expected_cols(expected_attrs)) File "/opt/stack/nova/nova/db/", line 764, in instance_update_and_get_original expected=expected) File "/opt/stack/nova/nova/db/sqlalchemy/", line 216, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 146, in wrapper ectxt.value = e.inner_exc File "/usr/local/lib/python2.7/dist-packages/oslo_utils/", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 136, in wrapper return f(*args, **kwargs) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2464, in instance_update_and_get_original expected, original=instance_ref)) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2602, in _instance_update raise exc(**exc_props) UnexpectedTaskStateError: Conflict updating instance cacf99de-117d-4ab7-bd12-32cc2265e906. Expected: {'task_state': [u'rebooting_hard', u'reboot_pending_hard', u'reboot_started_hard']}. Actual: {'task_state': u'reboot_started'}

    So what happened here? This is a bit confusing because we asked for a soft reboot of the instance, but the error we are seeing here is that a hard reboot was attempted -- specifically, we're trying to update an instance object but all the task states we expect the instance to be in are related to a hard reboot, but the task state we're actually in is for a soft reboot.

    We need to take a tour of the compute manager code to understand what happened here. nova-compute is implemented at nova/compute/ in the nova code base. Specifically, ComputeVirtAPI.init_host() sets up the service to start handling compute requests for a specific hypervisor node. As part of startup, this method calls ComputeVirtAPI._init_instance() once per instance on the hypervisor node. This method tries to do some sanity checking for each instance that nova thinks should be on the hypervisor:

    • Detecting if the instance was part of a failed evacuation.
    • Detecting instances that are soft deleted, deleting, or in an error state and ignoring them apart from a log message.
    • Detecting instances which we think are fully deleted but aren't in fact gone.
    • Moving instances we thought were booting, but which never completed into an error state. This happens if nova-compute crashes during the instance startup process.
    • Similarly, instances which were rebuilding are moved to an error state as well.
    • Clearing the task state for uncompleted tasks like snapshots or preparing for resize.
    • Finishes deleting instances which were partially deleted last time we saw them.
    • And finally, if the instance should be running but isn't, tries to reboot the instance to get it running.

    It is this final state which is relevant in this case -- we think the instance should be running and its not, so we're going to reboot it. We do that by calling ComputeVirtAPI.reboot_instance(). The code which does this work looks like this:

      try_reboot, reboot_type = self._retry_reboot(context, instance) current_power_state = self._get_power_state(context, instance) if try_reboot: LOG.debug("Instance in transitional state (%(task_state)s) at " "start-up and power state is (%(power_state)s), " "triggering reboot", {'task_state': instance.task_state, 'power_state': current_power_state}, instance=instance) self.reboot_instance(context, instance, block_device_info=None, reboot_type=reboot_type) return [...snip...] def _retry_reboot(self, context, instance): current_power_state = self._get_power_state(context, instance) current_task_state = instance.task_state retry_reboot = False reboot_type = compute_utils.get_reboot_type(current_task_state, current_power_state) pending_soft = (current_task_state == task_states.REBOOT_PENDING and instance.vm_state in vm_states.ALLOW_SOFT_REBOOT) pending_hard = (current_task_state == task_states.REBOOT_PENDING_HARD and instance.vm_state in vm_states.ALLOW_HARD_REBOOT) started_not_running = (current_task_state in [task_states.REBOOT_STARTED, task_states.REBOOT_STARTED_HARD] and current_power_state != power_state.RUNNING) if pending_soft or pending_hard or started_not_running: retry_reboot = True return retry_reboot, reboot_type

    So, we ask ComputeVirtAPI._retry_reboot() if a reboot is required, and if so what type. ComputeVirtAPI._retry_reboot() just uses nova.compute.utils.get_reboot_type() (aliased as compute_utils.get_reboot_type) to determine what type of reboot to use. This is the crux of the matter. Read on for a surprising discovery!

    nova.compute.utils.get_reboot_type() looks like this:

      def get_reboot_type(task_state, current_power_state): """Checks if the current instance state requires a HARD reboot.""" if current_power_state != power_state.RUNNING: return 'HARD' soft_types = [task_states.REBOOT_STARTED, task_states.REBOOT_PENDING, task_states.REBOOTING] reboot_type = 'SOFT' if task_state in soft_types else 'HARD' return reboot_type

    So, after all that it comes down to this. If the instance isn't running, then its a hard reboot. In our case, we shutdown the instance but haven't started it yet, so its not running. This will therefore be a hard reboot. This is where our problem lies -- we chose a hard reboot. The code doesn't blow up until later though -- when we try to do the reboot itself.

      @wrap_exception() @reverts_task_state @wrap_instance_event @wrap_instance_fault def reboot_instance(self, context, instance, block_device_info, reboot_type): """Reboot an instance on this host.""" # acknowledge the request made it to the manager if reboot_type == "SOFT": instance.task_state = task_states.REBOOT_PENDING expected_states = (task_states.REBOOTING, task_states.REBOOT_PENDING, task_states.REBOOT_STARTED) else: instance.task_state = task_states.REBOOT_PENDING_HARD expected_states = (task_states.REBOOTING_HARD, task_states.REBOOT_PENDING_HARD, task_states.REBOOT_STARTED_HARD) context = context.elevated()"Rebooting instance"), context=context, instance=instance) block_device_info = self._get_instance_block_device_info(context, instance) network_info = self.network_api.get_instance_nw_info(context, instance) self._notify_about_instance_usage(context, instance, "reboot.start") instance.power_state = self._get_power_state(context, instance) [...snip...]

    And there's our problem. We have a reboot_type of HARD, which means we set the expected_states to those matching a hard reboot. However, the state the instance is actually in will be one correlating to a soft reboot, because that's what the user requested. We therefore experience an exception when we try to save our changes to the instance. This is the exception we saw above.

    The fix in my patch is simply to change the current task state for an instance in this situation to one matching a hard reboot. It all just works then.

    So why do we decide to use a hard reboot if the current power state is not RUNNING? This code was introduced in this patch and there isn't much discussion in the review comments as to why a hard reboot is the right choice here. That said, we already fall back to a hard reboot in error cases of a soft reboot inside the libvirt driver, and a hard reboot requires less trust of the surrounding state for the instance (block device mappings, networks and all those side effects mentioned at the very beginning), so I think it is the right call.

    In conclusion, we use a hard reboot for soft reboots that fail, and a nova-compute crash during a soft reboot counts as one of those failure cases. So, when nova-compute detects a failed soft reboot, it converts it to a hard reboot and trys again.

    Tags for this post: openstack reboot nova nova-compute
    Related posts: Openstack compute node cleanup; Juno nova mid-cycle meetup summary: DB2 support; One week of Nova Kilo specifications; Juno nova mid-cycle meetup summary: bug management; Soft deleting instances and the reclaim_instance_interval in Nova; Further adventures with base images in OpenStack

posted at: 23:30 | path: /openstack | permanent link to this entry

Tue, 22 Sep 2015

First trail run

    So, now I trail run apparently. This was a test run for a hydration vest (thanks Steven Hanley for the loaner!). It was fun, but running up hills is evil.

    Tags for this post: blog canberra trail run
    Related posts: Second trail run; Boston; Chicken run; Random learning for the day; Update on the chickens

posted at: 17:26 | path: /diary | permanent link to this entry

Mon, 21 Sep 2015

Camp Cottermouth

    I spent the weekend at a Scout camp at Camp Cottermouth. The light on the hills here in the mornings is magic.


    Tags for this post: blog pictures 20150920 photo canberra bushwalk

posted at: 18:21 | path: /diary/pictures/20150920 | permanent link to this entry

Wed, 16 Sep 2015

Exploring for a navex

    I feel like I need more detailed maps of Mount Stranger than I currently have in order to layout a possible navex. I there spent a little time this afternoon wandering down the fire trail to mark all the gates in the fence. I need to do a little more of this before its ready for a navex.

    Tags for this post: blog canberra bush walk
    Related posts: First jog, and a walk to Los Altos; A Walk in the Woods; Walking to work

posted at: 15:15 | path: /diary | permanent link to this entry

Sun, 13 Sep 2015

On running

    I've been running for a little while now, but I don't mention it here much. I think I mostly don't mention it because I normally just post photos here, and I don't tend to stop and take happy snaps on my runs. The runs started off pretty modest -- initially I struggled with shin splints after more than a couple of minutes. I've worked through that and a couple of injuries along the way and am consistently doing 5km runs now.

    That said, my longest runs have been in the last week when I did a 7.5km and an 8.1km. I'm building up to 10km, mostly because its a nice round number. I think ultimately trail running might be the thing for me, I get quite bored running around suburbs over and over again.

    Its interesting that I come from an aggressively unsporting family, but yet all of my middle aged siblings and I have started running in the last year or two. Its a mid-life crisis thing perhaps?

    Tags for this post: blog running fitness sport
    Related posts: A year of being more active; First jog, and a walk to Los Altos; Martin retires from his work netball league

posted at: 13:46 | path: /diary | permanent link to this entry

Sat, 12 Sep 2015

CBC Navigation Course

    So today was the day long map and compass class with the Canberra Bushwalking Club. I liked this walk a lot. It was a good length at about 15km, and included a few things I'd wanted to do for a while like wander around McQuoids Hill and the northern crest of Urambi hills. Some nice terrain, and red rocks gorge clearly requires further exploration.


    See more thumbnails

    Tags for this post: blog pictures 20150912 photo canberra bushwalk

posted at: 00:32 | path: /diary/pictures/20150912 | permanent link to this entry

Tue, 08 Sep 2015

Lost and alone in the dark

    I've been doing the Canberra Bush Walking Club navigation course for the last couple of weeks, and last night's exercise was a night time dead reckoning navigation session. The course is really good by the way and I've been enjoying it a lot.

    It should be pointed out that I also wasn't lost, or alone, but it sure was dark.

    Anyway, the basic idea of the exercise is that you're given a hand drawn map, and a set of markers. You determine the bearing from each marker to the next, and the distance to walk. You then set off on your adventure. Getting a bearing or distance wrong matters, because you either need to stop and find the next way point, or carry the mistake on to the next marker. The markers were generally things like "gate in fence" or "two big trees".

    It turns out for me the hardest part is walking in a straight line when its dark. If you look at the GPS logged map below, you can see that the consistent error is that I tend to veer slowly to the right. That's a pretty useful thing to know, because it means I can correct a bit more for it next time. I got the line of march (not the bearing!) pretty badly wrong on the way to the dam, and that resulted in a bit of an adventure to find that way point. I think we missed the next way point as well because we carried the mistake on by setting off from the wrong point on the dam for the next leg.

    I really enjoyed this little walk, and I think I need to do a few more of these to get better at this skill. It seems arbitrary, but if my GPS ever fails and the weather is terrible it might come down to a skill like this keeping me moving in the right direction or not.

    Also, I think this would make a super good exercise for scouts. Now to try and convince them its fun...

    Tags for this post: blog canberra bushwalk navigation
    Related posts: Destinator 3 GPS navigation for the PocketPC; Outline mode numbering of headings

posted at: 16:17 | path: /diary | permanent link to this entry

Mon, 31 Aug 2015

Walk to the Southern Most Point

    I've just realized that I didn't post any pics of my walk to the most southern point of the ACT. The CBC had a planned walk to the southern most point on the ACT border and I was immediately intrigued. So, I took a day off work and gave it a go. It was well worth the experience, especially as Matthew the guide had a deep knowledge of the various huts and so forth in the area. A great day.


    See more thumbnails

    Tags for this post: blog pictures 20150818-southernmostpoint photo canberra bushwalk

posted at: 00:40 | path: /diary/pictures/20150818-southernmostpoint | permanent link to this entry

Thu, 06 Aug 2015

The Crossroad

posted at: 22:04 | path: /book/Mark_Donaldson | permanent link to this entry

Terrible pong

posted at: 17:21 | path: /coding_club | permanent link to this entry

Mon, 03 Aug 2015

Searching for open bugs in a launchpad project

    The launchpad API docs are OMG terrible, and it took me way too long to work out how to do this, so I thought I'd document it for later. Here's how you list all the open bugs in a launchpad project using the API:

      import argparse
      import os
      from launchpadlib import launchpad
      LP_INSTANCE = 'production'
      CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache/')
      def main(username, project):
          lp = launchpad.Launchpad.login_with(username, LP_INSTANCE, CACHE_DIR)
          for bug in lp.projects[project].searchTasks(status=["New",
                                                              "In Progress"]):
              print bug
      if __name__ == '__main__':
          parser = argparse.ArgumentParser(description='Fetch bugs from launchpad')
          args = parser.parse_args()
          main(args.username, args.project)

    Tags for this post: launchpad api
    Related posts: Juno nova mid-cycle meetup summary: the next generation Nova API; Taking over a launch pad project

posted at: 23:10 | path: /launchpad | permanent link to this entry

Sun, 02 Aug 2015

The End of All Things

posted at: 00:39 | path: /book/John_Scalzi | permanent link to this entry

Tue, 28 Jul 2015

Geocaching with a view

posted at: 16:38 | path: /diary/pictures/20150729 | permanent link to this entry

Mon, 27 Jul 2015

Chet and I went on an adventure to LA-96

    So, I've been fascinated with American nuclear history for ages, and Chet and I got talking about what if any nuclear launch facilities there were in LA. We found LA-96 online and set off on an expedition to explore. An interesting site, its a pity there are no radars left there. Apparently SF-88 is the place to go for tours from vets and radars.


    See more thumbnails

    I also made a quick and dirty 360 degree video of the view of LA from the top of the nike control radar tower:

    Tags for this post: blog pictures 20150727-nike_missile photo california
    Related posts: Packing; Jet lag; Views from a lookout on Mulholland Drive, Bel Air; Winchester Mystery House; Kern County oil wells on I-5; Hotel Avante

posted at: 17:07 | path: /diary/pictures/20150727-nike_missile | permanent link to this entry

Sun, 26 Jul 2015

Views from a lookout on Mulholland Drive, Bel Air

posted at: 20:37 | path: /diary/pictures/20150727-lookout | permanent link to this entry

Geocaching with TheDevilDuck

    In what amounts to possibly the longest LAX layover ever, I've been hanging out with Chet at his place in Altadena for a few days on the way home after the Nova mid-cycle meetup. We decided that being the dorks that we are we should do some geocaching. This is just some quick pics some unexpected bush land -- I never thought LA would be so close to nature, but this part certainly is.


    Tags for this post: blog pictures 20150727 photo california bushwalk
    Related posts: Summing up Santa Monica; All airline tickets now booked...; Hotel Avante; Andrew is a loony; Public transport to San Francisco from Santa Clara; When in San Diego, visit the aircraft carrier

posted at: 13:52 | path: /diary/pictures/20150727 | permanent link to this entry

Sat, 18 Jul 2015

Casuarina Sands to Kambah Pool

    I did a walk with the Canberra Bushwalking Club from Casuarina Sands (in the Cotter) to Kambah Pool (just near my house) yesterday. It was very enjoyable. I'm not going to pretend to be excellent at write ups for walks, but will note that the walk leader John Evans has a very detailed blog post about the walk up already. We found a bunch of geocaches along the way, with John doing most of the work and ChifleyGrrrl and I providing encouragement and scrambling skills. A very enjoyable day.


    See more thumbnails

    Tags for this post: blog pictures 20150718-casurina_sands_to_kambah_pool photo canberra bushwalk

posted at: 15:35 | path: /diary/pictures/20150718-casurina_sands_to_kambah_pool | permanent link to this entry

Wed, 15 Jul 2015


    I am on vacation this week, so I took this afternoon to do some walking and geocaching...

    That included a return visit to Narrabundah trig to clean up some geocaches I missed last visit:


    And exploring the Lindsay Pryor arboretum because I am trying to collect the complete set of arboretums in Canberra:


    And then finally the Majura trig, which was a new one for me:


    See more thumbnails

    I enjoyed the afternoon. I found a fair few geocaches, and walked for about five hours (not including driving between the locations). I would have spent more time geocaching at Majura, except I made it back to the car after sunset as it was.

    Tags for this post: blog pictures 20150715-wanderings photo canberra bushwalk trig_point
    Related posts: Melrose trig; CBC Navigation Course; A walk around Mount Stranger; Square Rock and Mount Franklin; Two trigs and a first attempt at finding Westlake; Forster trig

posted at: 14:30 | path: /diary/pictures/20150715-wanderings | permanent link to this entry

Mon, 13 Jul 2015

Quartz trig

    A morning of vacation geocaching, wandering, and walking to quartz trig. Quartz was a disappointment as its just a bolt in the ground, but this was a really nice area I am glad I wandered around in. This terrain would be very good for cubs and inexperienced scouts.


    See more thumbnails

    Tags for this post: blog pictures 20150713-quartz photo canberra bushwalk trig_point
    Related posts: Two trigs and a first attempt at finding Westlake; Forster trig; Big Monks; Percival trig; Wanderings; Melrose trig

posted at: 00:06 | path: /diary/pictures/20150713-quartz | permanent link to this entry

Wed, 01 Jul 2015

Hunting for GC1D1NB

    I went for an after work walk to try and find GC1D1NB on Tuggeranong Hill yesterday. It wasn't a great success. I was in the right area but I just couldn't find it. Eventually I ran out of time and had to turn back. I am sure I'll have another attempt at this one soon.


    Tags for this post: blog pictures 20150701-tuggeranong_hill photo canberra bushwalk

posted at: 15:52 | path: /diary/pictures/20150701-tuggeranong_hill | permanent link to this entry

Tue, 30 Jun 2015

Percival trig

    I had a pretty bad day, so I knocked off early and went for a walk before going off to the meeting at a charity I help out with. The walk was to Percival trig, which I have to say was one of the more boring trigs I've been to. Some of the forest nearly was nice enough, but the trig itself is stranded out in boring grasslands. Meh.


    Tags for this post: blog pictures 20150630-percival photo canberra bushwalk trig_point
    Related posts: Goodwin trig; Walk up Tuggeranong Hill; Oakey trig; Narrabundah trig and 16 geocaches; A team walk around Red Hill; Urambi Trig

posted at: 18:41 | path: /diary/pictures/20150630-percival | permanent link to this entry

A team walk around Red Hill

    My team at work is trying to get a bit more active, so a contingent from the Canberra portion of the team went for a walk around Red Hill. I managed to sneak in a side trip to Davidson trig, but it was cheating because it was from the car park at the top of the hill. A nice walk, with some cool geocaches along the way.


    Tags for this post: blog pictures 20150629-davidson photo canberra bushwalk trig_point
    Related posts: Harcourt and Rogers Trigs; One Tree and Painter; Wanniassa Trig; Quartz trig; Tuggeranong Hill (again); Two trigs and a first attempt at finding Westlake

posted at: 16:29 | path: /diary/pictures/20150629-davidson | permanent link to this entry

Sat, 20 Jun 2015

Yet another possible cub walk

    Jacqui and Catherine kindly agreed to come on another test walk for a possible cub walk. This one was the Sanctuary Loop at Tidbinbilla. To be honest this wasn't a great choice for cubs -- whilst being scenic and generally pleasant, the heavy use of black top paths and walkways made it feel like a walk in the Botanic Gardens, and the heavy fencing made it feel like an exhibit at a zoo. I'm sure its great for a weekend walk or for tourists, but if you're trying to have a cub adventure its not great.


    See more thumbnails

    Tags for this post: blog pictures 20150620-tidbinbilla photo canberra bushwalk

posted at: 02:20 | path: /diary/pictures/20150620-tidbinbilla | permanent link to this entry

Thu, 18 Jun 2015

Further adventures in the Jerrabomberra wetlands

    There was another walk option for cubs I wanted to explore at the wetlands, so I went back during lunch time yesterday. It was raining really quite heavily during this walk, but I still had fun. I think this route might be the winner -- its a bit longer, and a bit more interesting as well.


    See more thumbnails

    Tags for this post: blog pictures 20150618-jerrabomberra_wetlands photo canberra bushwalk

posted at: 16:01 | path: /diary/pictures/20150618-jerrabomberra_wetlands | permanent link to this entry

Tue, 16 Jun 2015

Exploring possible cub walks

    I've been exploring possible cub walks for a little while now, and decided that Jerrabomberra Wetlands might be an option. Most of these photos will seem a bit odd to readers, unless you realize I'm mostly interested in the terrain and its suitability for cubs...


    Tags for this post: blog pictures 20150617-jerrabomerra_wetlands photo canberra bushwalk

posted at: 21:35 | path: /diary/pictures/20150617-jerrabomerra_wetlands | permanent link to this entry

Thu, 04 Jun 2015

More coding club

    This is the second post about the coding club at my kid's school. I was away for four weeks travelling for work and then getting sick, so I am still getting back up to speed with what the kids have been up to while I've been away. This post is an attempt to gather some resources that I hope will be useful during the session today -- it remains to be seen how this maps to what the kids actually did while I was away.

    First off, the adults have decided to give Python for Kids a go as a teaching resource. The biggest catch with this book is that its kind of expensive -- at AUD $35 a copy, we can't just issue a copy to every kid in the room. That said, perhaps the kids don't each need a copy, as long as the adults are just using it as a guide for what things to cover.

    It appears that while I was away chapters 1 through 4 have been covered. 1 is about install python, and then 2-3 are language construct introductions. This is things like what a variable is, mathematical operators, strings, tuples and lists. So, that's all important but kind of dull. On the other hand, chapter 4 covers turtle graphics, which I didn't even realize that python had a module for.

    I have fond memories of doing logo graphics as a kid at school. Back in my day we'd sometimes even use actual robots to do some of the graphics, although most of it was simulated on Apple II machines of various forms. I think its important to let the kids of today know that these strange exercises they're doing used to relate to physical hardware that schools actually owned. Here are a couple of indicative pictures stolen from the Internet:

    So, I think that's what we'll keep going with this week -- I'll let the kids explain where they got to with turtle graphics and then we'll see how far we can take that without it becoming a chore.

    Tags for this post: coding_club kids coding python turtle graphics logo
    Related posts: Terrible pong; Coding club day one: a simple number guessing game in python; Graphics from the command line; Graphics programming with libtiff: black and white; Domain name lookup helper for python?; Python3 venvs for people who are old and grumpy

posted at: 19:35 | path: /coding_club | permanent link to this entry

Geocaching at the border

    Today's lunch walk was around Tuggeranong Pines again. At the back of the pine forest is the original train line from the 1880s which went down to Cooma. I walked as far as the old Tuggeranong siding before turning back. Its interesting, as there is evidence that there has been track work done here in the last ten years or so, even though the line hasn't been used since 1989.


    Tags for this post: blog pictures 20150604-geocaching photo canberra bushwalk

posted at: 02:49 | path: /diary/pictures/20150604-geocaching | permanent link to this entry

Mon, 01 Jun 2015

Melrose trig

    I went for a short geocaching walk at lunch today. Three geocaches in 45 minutes, so not too shabby. One of those caches was at the Melrose trig point, so bagged that too. There is some confusion here, as John Evans and I thought that Melrose was on private land. However, there is no signage to that effect in the area and the geocache owner asserts this is public land. ACTMAPi says the area is Tuggeranong Rural Block 35, but isn't clear on if the lease holder exists. Color me confused and possibly an accidental trespasser.


    Tags for this post: blog pictures 20150602-melrose photo canberra bushwalk trig_point
    Related posts: Tuggeranong Hill (again); Wanniassa Trig; Quartz trig; One Tree and Painter; Harcourt and Rogers Trigs; Square Rock and Mount Franklin

posted at: 22:55 | path: /diary/pictures/20150602-melrose | permanent link to this entry

In A Sunburned Country

    ISBN: 0965000281
    This is the first Bill Bryson book I've read, and I have to say I enjoyed it. Bill is hilarious and infuriating at the same time, which surprisingly to me makes for a very entertaining combination. I'm sure he's not telling the full story in this book -- its just not possible for someone so ill prepared to not just die in the outback somewhere. Take his visit to Canberra for example -- he drives down from Sydney, hits the first hotel he finds and then spends three days there. No wonder he's bored. Eventually he bothers to drive for another five minutes and finds there is more to the city than one hotel. On the other hand, he maligns my home town in such a hilarious manner I just can't be angry at him.

    I loved this book, highly recommended.

    Tags for this post: book bill_bryson travel australia
    Related posts: Getting ready to leave Sydney; Why Australia tops the piracy list; Australian public service announcement; Tech talk blogs; The Crossroad; We're getting there

posted at: 03:10 | path: /book/Bill_Bryson | permanent link to this entry

Sun, 31 May 2015

The 2016 Call For Proposals is open!

posted at: 22:44 | path: /openstack | permanent link to this entry

Thu, 28 May 2015

Square Rock and Mount Franklin

    I'm not really sure why it took me so long to write this set of walks up -- I think I just got lost in preparations for the most recent OpenStack summit and simply forgot. That said, here they are...

    Tony, Steven and I mounted an expedition to Mount Franklin, which is one of the trigs I hadn't been to yet. Its right on the ACT border with NSW, and despite not being a super long walk its verging of inaccessible in winter (think several feet of snow). So, we decided to get it done while we could.


    We also tacked on a trip to Square Rock based on the strong recommendation of a good friend. Square Rock has amazing views, highly recommended.


    Tags for this post: blog pictures 20150426-square_rock_franklin photo canberra bushwalk trig_point
    Related posts: Tuggeranong Hill (again); Quartz trig; Wanniassa Trig; One Tree and Painter; Harcourt and Rogers Trigs; A walk around Mount Stranger

posted at: 00:23 | path: /diary/pictures/20150426-square_rock_franklin | permanent link to this entry

Tue, 05 May 2015

Ancillary Justice

posted at: 20:48 | path: /book/Ann_Leckie | permanent link to this entry

Thu, 30 Apr 2015

Coding club day one: a simple number guessing game in python

    I've recently become involved in a new computer programming club at my kids' school. The club runs on Friday afternoons after school and is still very new so we're still working through exactly what it will look like long term. These are my thoughts on the content from this first session. The point of this first lesson was to approach a programming problem where every child stood a reasonable chance of finishing in the allotted 90 minutes. Many of the children had never programmed before, so the program had to be kept deliberately small. Additionally, this was a chance to demonstrate how literal computers are about the instructions they're given -- there is no room for intuition on the part of the machine here, it does exactly what you ask of it.

    The task: write a python program which picks a random number between zero and ten. Ask the user to guess the number the program has picked, with the program telling the user if they are high, low, or right.

    We then brainstormed the things we'd need to know how to do to make this program work. We came up with:
    • How do we get a random number?
    • What is a variable?
    • What are data types?
    • What is an integer? Why does that matter?
    • How do we get user input?
    • How do we do comparisons? What is a conditional?
    • What are the possible states for the game?
    • What is an exception? Why did I get one? How do I read it?

    With that done, we were ready to start programming. This was done with a series of steps that we walked through as a group -- let's all print hello work. Now let's generate a random number and print it. Ok, cool, now let's do input from a user. Now how do we compare that with the random number? Finally, how do we do a loop which keeps prompting until the user guesses the random number?

    For each of these a code snippet was written on the whiteboard and explained. It was up to the students to put them together into a program which actually works.

    Due to limitations in the school's operating environment (no local python installation and not working due to firewalling) we used for this exercise. The code that the kids ended up with looks like this:

      import random
      # Pick a random number
      number = random.randint(0, 10)
      # Now ask for guesses until the correct guess is made
      done = False
      while not done:
          guess = int(raw_input('What is your guess?'))
          print 'You guessed: %d' % guess
          if guess < number:
              print 'Higher!'
          elif guess > number:
              print 'Lower!'
              print 'Right!'
              done = True

    The plan for next session (tomorrow, in the first week of term two) is to recap what we did at the end of last term and explore this program to make sure everyone understands how it works.

    Tags for this post: coding_club teaching coding
    Related posts: More coding club; I'm glad I've turned on comments here; Terrible pong; Huffman coding

posted at: 01:18 | path: /coding_club | permanent link to this entry

Fri, 24 Apr 2015

Tuggeranong Trig (again)

    The cubs at my local scout group are interested in walking to a trig, but have some interesting constraints around mobility for a couple of their members. I therefore offered to re-walk Tuggeranong Trig in Oxley with an eye out for terrain. I think this walk would be very doable for cubs -- its 650 meters with only about 25 meters of vertical change. The path is also ok for a wheelchair I think.


    Tags for this post: blog pictures 20150415-tuggeranong_trig photo canberra bushwalk trig_point
    Related posts: Tuggeranong Hill (again); Wanniassa Trig; Quartz trig; One Tree and Painter; Harcourt and Rogers Trigs; A walk around Mount Stranger

posted at: 18:04 | path: /diary/pictures/20150415-tuggeranong_trig | permanent link to this entry

Sat, 11 Apr 2015

One Tree and Painter

    Paul and I set off to see two trigs today. One Tree is on the ACT border and is part of the centenary trail. Painter is a suburban trig in Belconnen. Much fun was had, I hope I didn't make Paul too late for the wedding he had to go to.


    Tags for this post: blog pictures 20150412-one_tree_painter photo canberra bushwalk trig_point
    Related posts: Harcourt and Rogers Trigs; Tuggeranong Hill (again); Quartz trig; Wanniassa Trig; Percival trig; Wanderings

posted at: 23:53 | path: /diary/pictures/20150412-one_tree_painter | permanent link to this entry

Thu, 09 Apr 2015

Thinking time

    I've had a lot of things to think about this week, so I've gone on a few walks. I found some geocaches along the way, but even better I think my head is a bit more sorted out now.

    Tags for this post: blog canberra bushwalk

posted at: 16:16 | path: /diary | permanent link to this entry

Sat, 04 Apr 2015

Bendora Arboretum and Bulls Head trig

    Prompted largely by a not very detailed entry in a book, a bunch of friends and I went to explore Bendora Arboretum. The arboretum was planted in the 1940's as scientific experiments exploring what soft woods would grow well in our climate -- this was prompted by the large amount of wood Australia was importing at the time. There were 34 Arboreta originally, but only this one remains. The last three other than this one were destroyed in the 2003 bush fires.

    This walk appears in Best Bush, Town and Village Walks in and around the ACT by Marion Stuart, which was the inspiration for this outing. The only thing to note with her description is that the walk is a fair bit longer than she describes -- its 2km from the locked gate to the hut, which means a 4km return walk before you explore the arboretum at all. The arboretum has received some attention from the ACT government recently, with new signage and a fresh gravel pass. Also please note this area might only be accessible by four wheel drive in winter, which is not mentioned in the book.

    We also did a side trip to Bulls Head trig, which was interesting as its not the traditional shape.


    See more thumbnails

    Tags for this post: blog pictures 20150404-bendora_bulls_head photo canberra bushwalk trig_point
    Related posts: One Tree and Painter; Harcourt and Rogers Trigs; Tuggeranong Hill (again); Quartz trig; Wanniassa Trig; Wanderings

posted at: 15:35 | path: /diary/pictures/20150404-bendora_bulls_head | permanent link to this entry

Mon, 23 Mar 2015

A quick walk through Curtin

    What do you do when you accidentally engaged a troll on twitter? You go for a walk of course.

    I didn't realize there had been a flash flood in Canberra in 1971 that killed seven people, probably because I wasn't born then. However, when I ask people who were around then, they don't remember without prompting either, which I think is sad. I only learnt about the flood because of the geocache I found hidden at the (not very well advertised) memorial today.

    This was walk inspired by one from Best Bush, Town and Village Walks in and around the ACT by Marion Stuart. I was disappointed that the guide book didn't mention the flash flood however, and skips the memorial.


    Tags for this post: blog pictures 20150323-curtin photo canberra bushwalk

posted at: 13:41 | path: /diary/pictures/20150323-curtin | permanent link to this entry

Sat, 21 Mar 2015

Narrabundah trig and 16 geocaches

    I walked to the Narrabundah trig yesterday, along the way collecting 15 of the 16 NRL themed caches in the area. It would have been all 16, except I can't find the last one for the life of me. I'm going to have to come back.

    I really like this area. Its scenic, has nice trails, and you can't tell you're in Canberra unless you really look for it. It seemed lightly used to be honest, I think I saw three other people the entire time I was there. I encountered more dogs off lead than people.


    Tags for this post: blog pictures 20150321-narrabundah photo canberra bushwalk trig_point
    Related posts: A team walk around Red Hill; Urambi Trig; Oakey trig; Goodwin trig; Walk up Tuggeranong Hill; Taylor Trig

posted at: 14:29 | path: /diary/pictures/20150321-narrabundah | permanent link to this entry

Thu, 19 Mar 2015

A quick trip to Namadgi

    I thought I'd drop down to the Namadgi visitors centre to have a look during lunch because I hadn't been there since being a teenager. I did a short walk to Gudgenby Hut, and on the way back discovered this original border blaze tree. Its stacked on pallets at the moment, but is apparently intended for display one day. This is how much of the ACT's boarder was marked originally -- blazes cut on trees.


    Tags for this post: blog pictures 20150320-namadgi photo canberra bushwalk namadgi border

posted at: 20:02 | path: /diary/pictures/20150320-namadgi | permanent link to this entry

Wed, 18 Mar 2015

Goodwin trig

posted at: 14:01 | path: /diary/pictures/20150318-goodwin | permanent link to this entry

Sun, 08 Mar 2015

Stromlo and Brown Trigs

    So, the Facebook group set off for our biggest Trig walk yet today -- Stromlo and Brown Trigs. This ended up being a 15km walk with a little bit of accidental trespass (sorry!) and about 600 meters of vertical rise. I was expecting Stromlo to be prettier to be honest, but it wasn't very foresty. Brown was in nicer territory, but its still not the nicest bit of Canberra I've been to. I really enjoyed this walk though, thanks to Simon, Tony and Jasmine for coming along!


    Tags for this post: blog pictures 20150309-stromlo_and_brown photo canberra bushwalk trig_point
    Related posts: Forster trig; Two trigs and a first attempt at finding Westlake; Percival trig; Wanderings; Big Monks; Melrose trig

posted at: 22:07 | path: /diary/pictures/20150309-stromlo_and_brown | permanent link to this entry

Wed, 25 Feb 2015

Tuggeranong Hill (again)

posted at: 16:06 | path: /diary/pictures/20150225-tuggeranong_hill | permanent link to this entry

Sun, 22 Feb 2015

Oakey trig

    I've got to say, this trig was disappointing. It was a lunch time walk, so a bit rushed, but the trig was just boring. Not particularly far, or particularly steep, or in a particularly interesting area. That said, it wasn't terrible. It just felt generic compared with other trigs I've walked to.


    Tags for this post: blog pictures 20150223-oakey_trig photo canberra tuggeranong bushwalk trig_point
    Related posts: A quick walk to Tuggeranong Trig; Taylor Trig; Walk up Tuggeranong Hill; Urambi Trig; Wanniassa Trig; Tuggeranong Hill (again)

posted at: 20:50 | path: /diary/pictures/20150223-oakey_trig | permanent link to this entry


posted at: 01:14 | path: /diary | permanent link to this entry

Fri, 20 Feb 2015

Command and Control

posted at: 20:27 | path: /book/Eric_Schlosser | permanent link to this entry

Forster trig

    Its been too long since I've attempted a trig walk -- 15 days to be exact. That's mostly because I've been really busy at work these last couple of weeks. That said, it was time for another trig, and this one was a bit of an adventure.

    Forster Trig is in the Bullen Nature Reserve and is one of the least urban trigs I've attempted so far, which is why this post is a bit more detailed than normal. Big Monks is probably the other trig walk most similar to this one. One of the challenges with this trig is that there is no track to the trig point. Reading John Evan's walk notes from his single assent of this trig, it seems that many people follow the 132kV power lines to the trig, but I consider this "cheating" as the power line is on private land and I didn't want to spend effort on getting permission to walk on someone's farm.

    Instead, I followed the Kambah Pool to Cassurina Sands track, and then turned right to bush bash to the trig when I got reasonably close. There wasn't any formed track this way, so I don't think this is a common approach. On the map you'll notice a fence marked -- that's where I had to jump a barbed wire fence, which wasn't the best plan ever. On the way back down from the summit I found a vehicle track, and I'd recommend that others follow that route (the one on the map with two gates marked and some stairs). The stairs are interesting -- a previous walker has mounded stones on both sides of the fence to make it easier to cross.

    Either way, its a bush bash up the hill itself, which is covered in reasonably dense spiky vegetation. You're going to want gaiters or long pants.


    Tags for this post: blog pictures 20150220-forster_trig photo canberra tuggeranong bushwalk trig_point
    Related posts: Tuggeranong Hill (again); Wanniassa Trig; Big Monks; Two trigs and a first attempt at finding Westlake; A walk around Mount Stranger; Taylor Trig

posted at: 16:39 | path: /diary/pictures/20150220-forster_trig | permanent link to this entry

Wed, 18 Feb 2015

Confessions of a middle aged orienteering marker

    I was an orienteering marker for the kid's scout troop tonight -- I guess it could have been a trick, but I think they were genuine. The basic idea was I went and stood at where the mark on the map was, and then noted which kids found me. Nice little hill in MacArthur, with pleasant views. I think I've found a good place for a geocache as well.

    Tags for this post: blog canberra bushwalk tuggeranong
    Related posts: A walk around Mount Stranger; Lunchtime geocaching; Big Monks; Two trigs and a first attempt at finding Westlake; Forster trig; Tuggeranong Hill (again)

posted at: 02:59 | path: /diary | permanent link to this entry

Tue, 17 Feb 2015

Little Black Mountain

    I went on a walk on Monday with the Canberra Bushwalking Club up Little Black Mountain. Its a nice area and I mostly enjoyed the walk. I say mostly because the walk leader was quite un-welcoming. There was the lecture about emergency beacons, and then the lecture about how he's never been bitten by a snake. It was quite an odd experience. I think I might avoid that leader in the future.

    Tags for this post: blog canberra bushwalk

posted at: 14:21 | path: /diary | permanent link to this entry

Wed, 11 Feb 2015

Geocaching in the evening, the second

    I went geocaching while the kids were at scouts. Overall, a very successful evening. The first hour was a phone call with Tristan Goode, and what I learnt is I should call Tristan more because every call (based on a data set of one walk) is 5km.

    Tags for this post: blog canberra tuggeranong geocaching
    Related posts: Another lunch time walk; Geocaching; Geocaching in the evening; Lunchtime geocaching; Walk up Tuggeranong Hill; Point Hut Cross to Pine Island

posted at: 15:32 | path: /diary | permanent link to this entry

Sun, 08 Feb 2015

Tuggeranong Stone Wall

posted at: 02:08 | path: /diary/pictures/20150208-tugg_stone_wall | permanent link to this entry

Sat, 07 Feb 2015

Point Hut Crossing to Pine Island

posted at: 14:26 | path: /diary/pictures/20150207-point_hut_pine_island | permanent link to this entry

Thu, 05 Feb 2015

Two trigs and a first attempt at finding Westlake

    I have to do HR paperwork for work at the moment, so I was looking for a quiet place to progress such things. I ended up beside Lake Burley Griffin working away on my laptop, which was quite pleasant. While in the area, it seemed like a good idea to have a quick first attempt at finding Westlake, which is the ruins of an abandoned builders camp from when Canberra was being constructed. I found some evidence, but I think I need to go further west in a subsequent wander.


    There was a planned walk to Davidson trig by the Facebook trig sweating group this evening, so I went along to that. Beforehand I was killing time in Yarralumla and bumped into the cub leader for my kids' troop. She's interested in trig walks suitable for cubs, so I am going to have to keep an eye out for some. Davidson was a nice on-track walk, probably suitable for cubs.


    Davidson finished about an hour earlier than expected, so I dropped in to walk to Mike on the way home in an entirely unplanned manner. This walk would be perfect for cubs if it wasn't almost entirely bush bashing. This is probably the lowest elevation trig I've been to so far.


    Tags for this post: blog pictures 20150205-westlake_davidson_and_mike photo canberra tuggeranong bushwalk trig_point
    Related posts: Forster trig; Big Monks; A walk around Mount Stranger; Wanniassa Trig; Tuggeranong Hill (again); Urambi Trig

posted at: 14:27 | path: /diary/pictures/20150205-westlake_davidson_and_mike | permanent link to this entry

Wed, 04 Feb 2015

Geocaching in the evening

posted at: 15:07 | path: /diary | permanent link to this entry

Sun, 01 Feb 2015

Big Monks

    I'm going to be honest here and say I got this one pretty wrong. Looking at Google Earth, Big Monks is about 1.7 km from the access road (which I got right), what I failed to notice is that it is super steep and that much of the walk is off the trail. So... A walk I thought would take 30 or 45 minutes took 2 hours. I didn't die though, so that's good.

    Fantastic views at the top, and found a couple of geocaches. There is obviously a lot of walking potential in Rob Roy Nature Park.


    Tags for this post: blog pictures 20150202-big_monks photo canberra tuggeranong bushwalk trig_point
    Related posts: Oakey trig; Walk up Tuggeranong Hill; Urambi Trig; A quick walk to Tuggeranong Trig; Taylor Trig; A walk around Mount Stranger

posted at: 21:25 | path: /diary/pictures/20150202-big_monks | permanent link to this entry

Sat, 31 Jan 2015

Cooleman and Arawang Trigs

    Doug and I went out to walk Doug's dog at short notice yesterday evening, and managed to sneak in two trigs while we were at it. A nice walk, although it took longer than I expected it to, with our average speed only being about 3.5 kilometers an hour. I wonder how much of that was the two peaks to climb, versus the puppy in tow.

    Along the way we found this super cool telegraph line, which appears to still have an active Telstra service on it. I wonder if we'll one day see fiber strung on these telegraph poles?


    Tags for this post: blog pictures 20150131-cooleman_and_arawang photo canberra western_creek bushwalk trig_point urban_trig
    Related posts: A quick walk to Tuggeranong Trig; Harcourt and Rogers Trigs; Bendora Arboretum and Bulls Head trig; Taylor Trig; Stromlo and Brown Trigs; Tuggeranong Trig (again)

posted at: 16:15 | path: /diary/pictures/20150131-cooleman_and_arawang | permanent link to this entry

Sun, 25 Jan 2015

First jog, and a walk to Los Altos

    Today was a busy day, not only did I foolishly go for a jog 5 minutes after sunrise...

    ...but then I went for a walk with James in the afternoon as well.

    Let's just say my fitbit is very impressed with me.

    Tags for this post: blog walk california running
    Related posts: Santa Monica; Noisy neighbours at Central Park in Mountain View; A walk in the San Mateo historic red woods; What are we doing with the pets?; Proposition 82; Light rail, oh how I love thee

posted at: 21:16 | path: /diary | permanent link to this entry

A walk in the San Mateo historic red woods

posted at: 06:48 | path: /diary/pictures/20150124-san_mateo | permanent link to this entry

Thu, 22 Jan 2015

Harcourt and Rogers Trigs

    I needed to visit someone in deepest darkest North Canberra yesterday, and there was an hour to kill between that meeting and the local Linux User's Group meeting. It seemed silly to have driven all that way and to not see a couple of trigs, so I visited these two. Both these trigs were easy to get to and urban. Frankly a little boring.

    Harcourt trig is in what I will call a cow paddock -- it doesn't have a lot of trees happening and feels a bit like left over land. Access to the nature reserve wasn't very obvious to me from the suburban streets, but the KML file below might help others to work it out. It wasn't too bad once I'd navigated the maze of streets and weird paved areas.


    Rogers was similar, except access was more obvious because it is in an older suburb. This is a nicer reserve than Harcourt's, with a nice peak and some walking opportunities around the base of the hill. I think I'll probably end up coming back to this one as my wife is nostalgic about growing up backing on to this reserve.


    Tags for this post: blog pictures 20150122-harcourt_and_rogers photo canberra gungahlin belconnen bushwalk trig_point urban_trig
    Related posts: A quick walk to Tuggeranong Trig; Cooleman and Arawang Trigs; Goodwin trig; Bendora Arboretum and Bulls Head trig; Taylor Trig; Tuggeranong Trig (again)

posted at: 13:10 | path: /diary/pictures/20150122-harcourt_and_rogers | permanent link to this entry

Tue, 20 Jan 2015

Another lunch time walk

    My arm still hurts, so no gym again. Instead, another lunch time walk although this one was shorter. The skies were dramatic, but no rain unfortunately. I found GC1DEFB during this walk.

    This walk appears in Walking Canberra by Graeme Barrow although I found it separately. The description in the book is accurate and helpful if you happen to own a copy.


    Tags for this post: blog pictures 20150120-geocaching photo canberra tuggeranong bushwalk geocaching
    Related posts: Geocaching in the evening, the second; Lunchtime geocaching; Geocaching in the evening; Geocaching; Wanniassa Trig; Tuggeranong Hill (again)

posted at: 01:57 | path: /diary/pictures/20150120-geocaching | permanent link to this entry

Sun, 18 Jan 2015

Lunchtime geocaching

posted at: 20:30 | path: /diary/pictures/20150119-geocaching | permanent link to this entry

Sat, 17 Jan 2015

Taylor Trig

    At the top of Mount Taylor lies the first trig point defeated by a group walk I've been on. Steve * 3, Erin, Michael *2, Andrew, Cadell, Maddie, Mel, Neill and Jenny all made it to the top of this one, so I'm super proud of us as a group. A nice walk and Mount Taylor clearly has potential for other walks as well, so I am sure I'll return here again.


    Tags for this post: blog pictures 20150118-mount_taylor photo canberra tuggeranong bushwalk trig_point
    Related posts: A quick walk to Tuggeranong Trig; Walk up Tuggeranong Hill; Oakey trig; Urambi Trig; Tuggeranong Hill (again); Wanniassa Trig

posted at: 23:50 | path: /diary/pictures/20150118-mount_taylor | permanent link to this entry

Thu, 15 Jan 2015

Another Nova spec update

    I started chasing down the list of spec freeze exceptions that had been requested, and that resulted in the list of specs for Kilo being updated. That updated list is below, but I'll do a separate post with the exception requests highlighted soon as well.


    • Add more detailed network information to the metadata server: review 85673 (approved).
    • Add separated policy rule for each v2.1 api: review 127863 (requested a spec exception).
    • Add user limits to the limits API (as well as project limits): review 127094.
    • Allow all printable characters in resource names: review 126696 (approved).
    • Consolidate all console access APIs into one: review 141065 (approved).
    • Expose the lock status of an instance as a queryable item: review 127139 (abandoned); review 85928 (approved).
    • Extend api to allow specifying vnic_type: review 138808 (requested a spec exception).
    • Implement instance tagging: review 127281 (fast tracked, approved).
    • Implement the v2.1 API: review 126452 (fast tracked, approved).
    • Improve the return codes for the instance lock APIs: review 135506.
    • Microversion support: review 127127 (approved).
    • Move policy validation to just the API layer: review 127160 (approved).
    • Nova Server Count API Extension: review 134279 (fast tracked).
    • Provide a policy statement on the goals of our API policies: review 128560 (abandoned).
    • Sorting enhancements: review 131868 (fast tracked, approved, implemented).
    • Support JSON-Home for API extension discovery: review 130715 (requested a spec exception).
    • Support X509 keypairs: review 105034 (approved).

    API (EC2)

    • Expand support for volume filtering in the EC2 API: review 104450.
    • Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).


    • Actively hunt for orphan instances and remove them: review 137996 (abandoned); review 138627.
    • Add totalSecurityGroupRulesUsed to the quota limits: review 145689.
    • Check that a service isn't running before deleting it: review 131633.
    • Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
    • Implement a daemon version of rootwrap: review 105404 (requested a spec exception).
    • Log request id mappings: review 132819 (fast tracked).
    • Monitor the health of hypervisor hosts: review 137768.
    • Remove the assumption that there is a single endpoint for services that nova talks to: review 132623.

    Block Storage

    • Allow direct access to LVM volumes if supported by Cinder: review 127318.
    • Cache data from volumes on local disk: review 138292 (abandoned); review 138619.
    • Enhance iSCSI volume multipath support: review 134299 (requested a spec exception).
    • Failover to alternative iSCSI portals on login failure: review 137468 (requested a spec exception).
    • Give additional info in BDM when source type is "blank": review 140133.
    • Implement support for a DRBD driver for Cinder block device access: review 134153 (requested a spec exception).
    • Poll volume status: review 142828 (abandoned).
    • Refactor ISCSIDriver to support other iSCSI transports besides TCP: review 130721 (approved).
    • StorPool volume attachment support: review 115716 (approved, requested a spec exception).
    • Support Cinder Volume Multi-attach: review 139580 (approved).
    • Support iSCSI live migration for different iSCSI target: review 132323 (approved).


    Containers Service


    • Develop and implement a profiler for SQL requests: review 142078 (abandoned).
    • Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved, implemented).
    • Nova db purge utility: review 132656.
    • Online schema change options: review 102545 (approved).
    • Support DB2 as a SQL database: review 141097 (fast tracked, approved).
    • Validate database migrations and model': review 134984 (approved).

    Hypervisor: Docker

    Hypervisor: FreeBSD

    • Implement support for FreeBSD networking in nova-network: review 127827.

    Hypervisor: Hyper-V

    • Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved, implemented).
    • Instance hot resize: review 141219.

    Hypervisor: Ironic

    Hypervisor: VMWare

    • Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
    • Add support for the HTML5 console: review 127283 (requested a spec exception).
    • Allow Nova to access a VMWare image store over NFS: review 126866.
    • Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
    • Enable the mapping of raw cinder devices to instances: review 128697.
    • Implement vSAN support: review 128600 (fast tracked, approved).
    • Support multiple disks inside a single OVA file: review 128691.
    • Support the OVA image format: review 127054 (fast tracked, approved).

    Hypervisor: libvirt

    Instance features


    • A lock-free quota implementation: review 135296 (approved).
    • Automate the documentation of the virtual machine state transition graph: review 94835.
    • Fake Libvirt driver for simulating HW testing: review 139927 (abandoned).
    • Flatten Aggregate Metadata in the DB: review 134573 (abandoned).
    • Flatten Instance Metadata in the DB: review 134945 (abandoned).
    • Implement a new code coverage API extension: review 130855.
    • Move flavor data out of the system_metadata table in the SQL database: review 126620 (approved).
    • Move to polling for cinder operations: review 135367.
    • PCI test cases for third party CI: review 141270.
    • Transition Nova to using the Glance v2 API: review 84887 (abandoned).
    • Transition to using glanceclient instead of our own home grown wrapper: review 133485 (approved).


    • Enable lazy translations of strings: review 126717 (fast tracked, approved).


    • Add a new linuxbridge VIF type, macvtap: review 117465 (abandoned).
    • Add a plugin mechanism for VIF drivers: review 136827 (abandoned).
    • Add support for InfiniBand SR-IOV VIF Driver: review 131729 (requested a spec exception).
    • Neutron DNS Using Nova Hostname: review 90150 (abandoned).
    • New VIF type to allow routing VM data instead of bridging it: review 130732 (approved, requested a spec exception).
    • Nova Plugin for OpenContrail: review 126446 (approved).
    • Refactor of the Neutron network adapter to be more maintainable: review 131413.
    • Use the Nova hostname in Neutron DNS: review 137669.
    • Wrap the Python NeutronClient: review 141108.


    • Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.


    • A nested quota driver API: review 129420.
    • Add a filter to take into account hypervisor type and version when scheduling: review 137714.
    • Add an IOPS weigher: review 127123 (approved, implemented); review 132614.
    • Add instance count on the hypervisor as a weight: review 127871 (abandoned).
    • Add soft affinity support for server group: review 140017 (approved).
    • Allow extra spec to match all values in a list by adding the ALL-IN operator: review 138698 (fast tracked, approved).
    • Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
    • Allow the remove of servers from server groups: review 136487.
    • Cache aggregate metadata: review 141846.
    • Convert get_available_resources to use an object instead of dict: review 133728 (abandoned).
    • Convert the resource tracker to objects: review 128964 (fast tracked, approved).
    • Create an object model to represent a request to boot an instance: review 127610 (approved).
    • Decouple services and compute nodes in the SQL database: review 126895 (approved).
    • Distribute PCI Requests Across Multiple Devices: review 142094.
    • Enable adding new scheduler hints to already booted instances: review 134746.
    • Fix the race conditions when migration with server-group: review 135527 (abandoned).
    • Implement resource objects in the resource tracker: review 127609 (approved, requested a spec exception).
    • Improve the ComputeCapabilities filter: review 133534 (requested a spec exception).
    • Isolate Scheduler DB for Filters: review 138444 (requested a spec exception).
    • Isolate the scheduler's use of the Nova SQL database: review 89893 (approved).
    • Let schedulers reuse filter and weigher objects: review 134506 (abandoned).
    • Move select_destinations() to using a request object: review 127612 (approved).
    • Persist scheduler hints: review 88983.
    • Refactor allocate_for_instance: review 141129.
    • Stop direct lookup for host aggregates in the Nova database: review 132065 (abandoned).
    • Stop direct lookup for instance groups in the Nova database: review 131553 (abandoned).
    • Support scheduling based on more image properties: review 138937.
    • Trusted computing support: review 133106.



    • Make key manager interface interoperable with Barbican: review 140144 (fast tracked, approved).
    • Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked, approved).
    • Strongly validate the tenant and user for quota consuming requests with keystone: review 92507 (approved).

    Service Groups

posted at: 19:16 | path: /openstack/kilo | permanent link to this entry

Mon, 12 Jan 2015

Kilo Nova deploy recommendations

    What would a Nova developer tell a deployer to think about before their first OpenStack install? This was the question I wanted to answer for my OpenStack miniconf talk, and writing this essay seemed like a reasonable way to take the bullet point list of ideas we generated and turn it into something that was a cohesive story. Hopefully this essay is also useful to people who couldn't make the conference talk.

    Please understand that none of these are hard rules -- what I seek is for you to consider your options and make informed decisions. Its really up to you how you deploy Nova.

    Operating environment

    • Consider what base OS you use for your hypervisor nodes if you're using Linux. I know that many environments have standardized on a given distribution, and that many have a preference for a long term supported release. However, Nova is at its most basic level a way of orchestrating tools packaged by your distribution via APIs. If those underlying tools are buggy, then your Nova experience will suffer as well. Sometimes we can work around known issues in older versions of our dependencies, but often those work-arounds are hard to implement (and therefore likely to be less than perfect) or have performance impacts. There are many examples of the problems you can encounter, but hypervisor kernel panics, and disk image corruption are just two examples. We are trying to work with distributions on ensuring they back port fixes, but the distributions might not be always willing to do that. Sometimes upgrading the base OS on your hypervisor nodes might be a better call.
    • The version of Python you use matters. The OpenStack project only tests with specific versions of Python, and there can be bugs between releases. This is especially true for very old versions of Python (anything older than 2.7) and new versions of Python (Python 3 is not supported for example). Your choice of base OS will affect the versions of Python available, so this is related to the previous point.
    • There are existing configuration management recipes for most configuration management systems. I'd avoid reinventing the wheel here and use the community supported recipes. There are definitely resources available for chef, puppet, juju, ansible and salt. If you're building a very large deployment from scratch consider triple-o as well. Please please please don't fork the community recipes. I know its tempting, but contribute to upstream instead. Invariably upstream will continue developing their stuff, and if you fork you'll spend a lot of effort keeping in sync.
    • Have a good plan for log collection and retention at your intended scale. The hard reality at the moment is that diagnosing Nova often requires that you turn on debug logging, which is very chatty. Whilst we're happy to take bug reports where we've gotten the log level wrong, we haven't had a lot of success at systematically fixing this issue. Your log infrastructure therefore needs to be able to handle the demands of debug logging when its turned on. If you're using central log servers think seriously about how much disks they require. If you're not doing centralized syslog logging, perhaps consider something like logstash.
    • Pay attention to memory usage on your controller nodes. OpenStack python processes can often consume hundreds of megabytes of virtual memory space. If you run many controller services on the same node, make sure you have enough RAM to deal with the number of processes that will, by default, be spawned for the many service endpoints. After a day or so of running a controller node, check in on the VMM used for python processes and make any adjustments needed to your "workers" configuration settings.

    • Estimate your final scale now. Sure, you're building a proof of concept, but these things have a habit of becoming entrenched. If you are planning a deployment that is likely to end up being thousands of nodes, then you are going to need to deploy with cells. This is also possibly true if you're going to have more than one hypervisor or hardware platform in your deployment -- its very common to have a cell per hypervisor type or per hardware platform. Cells is relatively cheap to deploy for your proof of concept, and it helps when that initial deploy grows into a bigger thing. Should you be deploying cells from the beginning? It should be noted however that not all features are currently implemented in cells. We are working on this at the moment though.
    • Consider carefully what SQL database to use. Nova supports many SQL databases via sqlalchemy, but are some are better tested and more widely deployed than others. For example, the Postgres back end is rarely deployed and is less tested. I'd recommend a variant of MySQL for your deployment. Personally I've seen good performance on Percona, but I know that many use the stock MySQL as well. There are known issues at the moment with Galera as well, so show caution there. There is active development happening on the select-for-update problems with Galera at the moment, so that might change by the time you get around to deploying in production. You can read more about our current Galera problems on Jay Pipe's blog .
    • We support read only replicas of the SQL database. Nova supports offloading read only SQL traffic to read only replicas of the main SQL database, but I do no believe this is widely deployed. It might be of interest to you though.
    • Expect a lot of SQL database connections. While Nova has the nova-conductor service to control the number of connections to the database server, other OpenStack services do not, and you will quickly out pace the number of default connections allowed, at least for a MySQL deployment. Actively monitor your SQL database connection counts so you know before you run out. Additionally, there are many places in Nova where a user request will block on a database query, so if your SQL back end isn't keeping up this will affect performance of your entire Nova deployment.
    • There are options with message queues as well. We currently support rabbitmq, zeromq and qpid. However, rabbitmq is the original and by far the most widely deployed. rabbitmq is therefore a reasonable default choice for deployment.

    • Not all hypervisor drivers are created equal. Let's be frank here -- some hypervisor drivers just aren't as actively developed as others. This is especially true for drivers which aren't in the Nova code base -- at least the ones the Nova team manage are updated when we change the internals of Nova. I'm not a hypervisor bigot -- there is a place in the world for many different hypervisor options. However, the start of a Nova deploy might be the right time to consider what hypervisor you want to use. I'd personally recommend drivers in the Nova code base with active development teams and good continuous integration, but ultimately you have to select a driver based on its merits in your situation. I've included some more detailed thoughts on how to evaluate hypervisor drivers later in this post, as I don't want to go off on a big tangent during my nicely formatted bullet list.
    • Remember that the hypervisor state is interesting debugging information. For example with the libvirt hypervisor, the contents on /var/lib/instances is super useful for debugging misbehaving instances. Additionally, all of the existing libvirt tools work, so you can use those to investigate as well. However, I strongly recommend you only change instance state via Nova, and not go directly to the hypervisor.

    • Avoid new deployments of nova-network. nova-network has been on the deprecation path for a very long time now, and we're currently working on the final steps of a migration plan for nova-network users to neutron. If you're a new deployment of Nova and therefore don't yet depend on any of the features of nova-network, I'd start with Neutron from the beginning. This will save you a possible troublesome migration to Neutron later.

    Testing and upgrades
    • You need a test lab. For a non-trivial deployment, you need a realistic test environment. Its expected that you test all upgrades before you do them in production, and rollbacks can sometimes be problematic. For example, some database migrations are very hard to roll back, especially if new instances have been created in the time it took you to decide to roll back. Perhaps consider turning off API access (or putting the API into a read only state) while you are validating a production deploy post upgrade, that way you can restore a database snapshot if you need to undo the upgrade. We know this isn't perfect and are working on a better upgrade strategy for information stored in the database, but we will always expect you to test upgrades before deploying them.
    • Test database migrations on a copy of your production database before doing them for real. Another reason to test upgrades before doing them in production is because some database migrations can be very slow. Its hard for the Nova developers to predict which migrations will be slow, but we do try to test for this and minimize the pain. However, aspects of your deployment can affect this in ways we don't expect -- for example if you have large numbers of volumes per instance, then that could result in database tables being larger than we expect. You should always test database migrations in a lab and report any problems you see.
    • Think about your upgrade strategy in general. While we now support having the control infrastructure running a newer release than the services on hypervisor nodes, we only support that for one release (so you could have your control plane running Kilo for example while you are still running Juno on your hypervisors, you couldn't run Icehouse on the hypervisors though). Are you going to upgrade every six months? Or are you going to do it less frequently but step through a series of upgrades in one session? I suspect the latter option is more risky -- if you encounter a bug in a previous release we would need to back port a fix, which is a much slower process than fixing the most recent release. There are also deployments which choose to "continuously deploy" from trunk. This gets the access to features as they're added, but means that the deployments need to have more operational skill and a closer association with the upstream developers. In general continuous deployers are larger public clouds as best as I can tell.

    libvirt specific considerations
    • For those intending to run the libvirt hypervisor driver, not all libvirt hypervisors are created equal. libvirt implements pluggable hypervisors, so if you select the Nova libvirt hypervisor driver, you then need to select what hypervisor to use with libvirt as well. It should be noted however that some hypervisors work better than others, with kvm being the most widely deployed.
    • There are two types of storage for instances. There is "instance storage", which is block devices that exist for the life of the instance and are then cleaned up when the instance is destroyed. There is also block storage provided Cinder, which is persistent and arguably easier to manage than instance storage. I won't discuss storage provided by Cinder any further however, because it is outside the scope of this post. Instance storage is provided by a plug in layer in the libvirt hypervisor driver, which presents you with another set of deployment decisions.
    • Shared instance storage is attractive, but it comes at a cost. Shared instance storage is an attractive option, but isn't required for live migration of instances using the libvirt hypervisor. Think about the costs of shared storage though -- for example putting everything on network attached storage is likely to be expensive, especially if most of your instances don't need the facility. There are other options such as Ceph, but the storage interface layer in libvirt is one of the areas of code where we need to improve testing so be wary of bugs before relying on those storage back ends.

    Thoughts on how to evaluate hypervisor drivers

    As promised, I also have some thoughts on how to evaluate which hypervisor driver is the right choice for you. First off, if your organization has a lot of experience with a particular hypervisor, then there is always value in that. If that is the case, then you should seriously consider running the hypervisor you already have experience with, as long as that hypervisor has a driver for Nova which meets the criteria below.

    What's important is to be looking for a driver which works well with Nova, and a good measure of that is how well the driver development team works with the Nova development team. The obvious best case here is where both teams are the same people -- which is true for drivers that are in the Nova code base. I am aware there are drivers that live outside of Nova's code repository, but you need to remember that the interface these drivers plug into isn't a stable or versioned interface. The risk of those drivers being broken by the ongoing development of Nova is very high. Additionally, only a very small number of those "out of tree" drivers contribute to our continuous integration testing. That means that the Nova team also doesn't know when those drivers are broken. The breakages can also be subtle, so if your vendor isn't at the very least doing tempest runs against their out of tree driver before shipping it to you then I'd be very worried.

    You should also check out how many bugs are open in LaunchPad for your chosen driver (this assumes the Nova team is aware of the existence of the driver I suppose). Here's an example link to the libvirt driver bugs currently open. As well as total bug count, I'd be looking for bug close activity -- its nice if there is a very small number of bugs filed, but perhaps that's because there aren't many users. It doesn't necessarily mean the team for that driver is super awesome at closing bugs. The easiest way to look into bug close rates (and general code activity) would be to checkout the code for Nova and then look at the log for your chosen driver. For example for the libvirt driver again:

    $ git clone
    $ cd nova/nova/virt/driver/libvirt
    $ git log .

    That will give you a report on all the commits ever for that driver. You don't need to read the entire report, but it will give you an idea of what the driver authors have recently been thinking about.

    Another good metric is the specification activity for your driver. Specifications are the formal design documents that Nova adopted for the Juno release, and they document all the features that we're currently working on. I write summaries of the current state of Nova specs regularly, which you can see posted at with this being the most recent summary at the time of writing this post. You should also check how much your driver authors interact with the core Nova team. The easiest way to do that is probably to keep an eye on the Nova team meeting minutes, which are posted online.

    Finally, the OpenStack project believes strongly in continuous integration testing. It (s/It/Testing) has clear value in the number of bugs it finds in code before our users experience them, and I would be very wary of driver code which isn't continuously integrated with Nova. Thus, you need to ensure that your driver has well maintained continuous integration testing. This is easy for "in tree" drivers, as we do that for all of them. For out of tree drivers, continuous integration testing is done with a thing called "third party CI".

    How do you determine if a third party CI system is well maintained? First off, I'd start by determining if a third party CI system actually exists by looking at OpenStack's list of known third party CI systems. If the third party isn't listed on that page, then that's a very big warning sign. Next you can use Joe Gordon's lastcomment tool to see when a given CI system last reported a result:

    $ git clone
    $ ./ --name "DB Datasets CI"
    last 5 comments from 'DB Datasets CI'
    [0] 2015-01-07 00:46:33 (1:35:13 old) 'Ignore 'dynamic' addr flag on gateway initialization' 
    [1] 2015-01-07 00:37:24 (1:44:22 old) 'Use session with neutronclient' 
    [2] 2015-01-07 00:35:33 (1:46:13 old) 'libvirt: Expanded test libvirt driver' 
    [3] 2015-01-07 00:29:50 (1:51:56 old) 'ephemeral file names should reflect fs type and mkfs command' 
    [4] 2015-01-07 00:15:59 (2:05:47 old) 'Support for ext4 as default filesystem for ephemeral disks' 

    You can see here that the most recent run is 1 hour 35 minutes old when I ran this command. That's actually pretty good given that I wrote this while most of America was asleep. If the most recent run is days old, that's another warning sign. If you're left in doubt, then I'd recommend appearing in the OpenStack IRC channels on freenode and asking for advice. OpenStack has a number of requirements for third party CI systems, and I haven't discussed many of them here. There is more detail on what OpenStack considers a "well run CI system" on the OpenStack Infrastructure documentation page.

    General operational advice

    Finally, I have some general advice for operators of OpenStack. There is an active community of operators who discuss their use of the various OpenStack components at the openstack-operators mailing list, if you're deploying Nova you should consider joining that mailing list. While you're welcome to ask questions about deploying OpenStack at that list, you can also ask questions at the more general OpenStack mailing list if you want to.

    There are also many companies now which will offer to operate an OpenStack cloud for you. For some organizations engaging a subject matter expert will be the right decision. Probably the most obvious way to evaluate which of those companies to use is to look at their track record of successful deployments, as well as their overall involvement in the OpenStack community. You need a partner who can advocate for you with the OpenStack developers, as well as keeping an eye on what's happening upstream to ensure it meets your needs.


    Thanks for reading so far! I hope this document is useful to someone out there. I'd love to hear your feedback -- are there other things we wished deployers considered before committing to a plan? Am I simply wrong somewhere? Finally, this is the first time that I've posted an essay form of a conference talk instead of just the slide deck, and I'd be interested in if people find this format more useful than a YouTube video post conference. Please drop me a line and let me know if you find this useful!

    Tags for this post: openstack nova
    Related posts: Juno nova mid-cycle meetup summary: bug management; Soft deleting instances and the reclaim_instance_interval in Nova; Further adventures with base images in OpenStack; One week of Nova Kilo specifications; Juno nova mid-cycle meetup summary: DB2 support; Openstack compute node cleanup

posted at: 14:11 | path: /openstack | permanent link to this entry

Wed, 07 Jan 2015

A quick walk to William Farrer's grave

posted at: 01:53 | path: /diary/pictures/20150107-william_farrers_grave | permanent link to this entry

Sun, 04 Jan 2015

Wanniassa Trig

    I walked up to Wanniassa Trig this afternoon. It was a nice walk, the nature park is in the middle of suburban Canberra, but you couldn't tell that from within much of the park. The nature park also has excellently marked fire trails. There were really cool thunderstorms on the ranges as I walked, whilst I managed to avoid getting rained on while walking.


    Tags for this post: blog pictures 20150104-wanniassa_trig photo canberra tuggeranong bushwalk trig_point
    Related posts: Tuggeranong Hill (again); Big Monks; Forster trig; Two trigs and a first attempt at finding Westlake; A walk around Mount Stranger; Taylor Trig

posted at: 02:00 | path: /diary/pictures/20150104-wanniassa_trig | permanent link to this entry