Content here is by Michael Still All opinions are my own.
See recent comments. RSS feed of all comments.

Thu, 15 Oct 2015

Geocaching at the base of Mount Ainslie

    I have a bit of a backlog of posts about recent walks which I am working through. Last week I found myself in Ainslie for an appointment which ended at lunch time, so I figured I'd go for a walk. There is a series of geocaches near Campbell Park West (a set of office buildings for non-Canberrans), so off I went.

    The cache series was nice, but the most exciting part of the walking in this area was all the unexploded ordinance (UXO) warnings. I'm sure the area is totally safe, as many people walk through it each day, but it certainly adds an air of adventure to the walk.

    You can find a list of the UXO reports for the ACT on the Department of Defence website, I must say that its not very impressive that the Department has contaminated so many sites around Australia without remediating them -- there are heaps in New South Wales for example. The ACT gets off relatively lightly with only three contaminated sites.

    Its also interesting to note that a suburb very close to me was used as a bombing practise range in World War Two. I'm not aware of anyone in my circle who knew that.

    Anyway. Nice terrain, nice caches, lots of fun. I'd say this would be a good walk for cubs, but I am sure the risk management paperwork for a walk in a UXO are is complicated.


    Tags for this post: blog pictures 20151009 photo canberra bushwalk

posted at: 16:40 | path: /diary/pictures/20151009 | permanent link to this entry

Wed, 07 Oct 2015

Lunchtime geocaching

    So, I didn't get to sleep last night until 4:30am because of a combination of work meetings and small children, so today was a pretty weird day for me. I was having a lot of trouble concentrating at lunch time, so I decided a walk was the least worst thing I could do with my time. I decided to knock off some of the few remaining geocaches in southern Tuggeranong that i haven't found yet.

    This walk was odd -- it started and ended in a little bit of Theodore they never got around to actually building, and I can't find any documentation online about why. It then proceeded through a nice little green strip that has more than its share of rubbish dumped, Cleanup Australia needs to do a visit here! Then there were the Aboriginal axe grinding grooves (read more) just kind of in the middle of the green strip with no informational signage or anything. Finally, a geocache at an abandoned look out, which would have been much nicer if it wasn't being used as an unofficial dump now.

    That said, a nice little walk, but I have no real desire to revisit this one any time soon.


    Tags for this post: blog pictures 20151007 photo canberra bushwalk

posted at: 00:58 | path: /diary/pictures/20151007 | permanent link to this entry

Thu, 01 Oct 2015

A searchable database of walk waypoints

    Over the last year I've become increasingly interested in bush walking, especially around the ACT. It quickly became evident that John Evan's site is an incredibly valuable resource, especially if you're interested in trig points or border markers.

    However, I do most of my early walk planning and visualization in Google Earth before moving to Garmin Basecamp to generate walkable maps. I wanted a way to hook John's database of GPS logs into Google Earth, so that I could plan walks more effectively. For example, John often marks gates in fences, underpasses under major roads, and good routes through scrub in his GPS tracks.

    After a fair bit of playing, I ended up with this KML file which helps me do those things. Its basically magic -- the file is just a link to a search engine which has a database of GPS waypoints based off walks John and I have logged. These are then rendered in Google Earth as if they were in a static KML file. You can also download the search results as KML for editing and so forth as well.

    So, I'd be interested in other people's thoughts on if this is a useful thing. I'd also be very interested in other donated GPS logs of walks and bike rides around Canberra, especially if they have waypoints marked for interesting things. If you have any comments at all, please email me at

    Tags for this post: walks gps search google earth
    Related posts: A busy couple of days; HP iPaq GPS FA256A; Well, that's Google blog search live then; MelbourneIT are into search engine optimisation?; Google book search; Historical revisionism

posted at: 14:59 | path: /walks | permanent link to this entry

Garran green strip

    When I was a teenager my best mate lived in a house which backs onto this smallish reserve and we used to walk his dog here heaps. I had a few spare moments yesterday, so I was keen to do a quick explore and see what its like now. The short answer is that its still nice -- good terrain, nice mature trees, and a few geocaches. I think this one would be a good walk for cubs.


    Tags for this post: blog pictures 20151001 photo canberra bushwalk

posted at: 14:35 | path: /diary/pictures/20151001 | permanent link to this entry

Tue, 29 Sep 2015

Wandering around Curtin

    I decided to go on a little walk on the way home from a work lunch and I don't regret it. This is a nice area, which I was exploring for geocaches. I probably wouldn't have come here at all, but it was the second part of the "Trees of Curtin" walk from Best Bush, Town and Village Walks in and around the ACT that I had done the first half of ages ago.

    I am glad I came back for the second half -- to be honest I was pretty bored with the first half (a bike path beside a major road mostly), whereas this is much more like walking around in nature. The terrain is nice, no thistles, and plenty of horses. A nice afternoon walk overall.

    Now back to reviewing Mitaka specs.


    Tags for this post: blog pictures 20150930 photo canberra bushwalk

posted at: 23:23 | path: /diary/pictures/20150930 | permanent link to this entry

Second trail run

    I went for my second trail run last night. This one was on much rockier terrain, and I ended up tweaking my right knee. I think that was related to the knee having to stabilize as I ran over uneven rocks. I'll experiment by finding a different less awkward trail to run and seeing what happens I suppose.

    Tags for this post: blog canberra trail run
    Related posts: First trail run; Chicken run; Update on the chickens; Random learning for the day; Boston

posted at: 15:06 | path: /diary | permanent link to this entry

Mon, 28 Sep 2015

Old Joe and Goorooyarroo

    Steve, Mel, Michael and I went for a walk to Old Joe trig yesterday. I hadn't been to Goorooyarroo at all before, and was quite impressed. The terrain is nice, with some steep bits as you get close to the border (its clear that the border follows the water catchment from a walk around here). Plenty of nice trees, not too many thistles, and good company. A nice morning walk.

    We bush bashed to the trig straight up the side of the hill, and I think there were gentler (but longer) approaches available -- like for instance how we walked down off the hill following the fence line. That said, the bush bash route wasn't terrible and its probably what I'd do again.

    I need to come back here and walk this border segment, that looks like fun. There are also heaps of geocaches in this area to collect.


    Tags for this post: blog pictures 20150928 photo canberra bushwalk

posted at: 15:34 | path: /diary/pictures/20150928 | permanent link to this entry

Wed, 23 Sep 2015

How we got to test_init_instance_retries_reboot_pending_soft_became_hard

    I've been asked some questions about a recent change to nova that I am responsible for, and I thought it would be easier to address those in this format than trying to explain what's happening in IRC. That way whenever someone compliments me on possibly the longest unit test name ever written, I can point them here.

    Let's start with some definitions. What is the difference between a soft reboot and a hard reboot in Nova? The short answer is that a soft reboot gives the operating system running in the instance an opportunity to respond to an ACPI power event gracefully before the rug is pulled out from under the instance, whereas a hard reboot just punches the instance in the face immediately.

    There is a bit more complexity than that of course, because this is OpenStack. A hard reboot also re-fetches image meta-data, and rebuilds the XML description of the instance that we hand to libvirt. It also re-populates any missing backing files. Finally it ensures that the networking is configured correctly and boots the instance again. In other words, a hard reboot is kind of like an initial instance boot, in that it makes fewer assumptions about how much you can trust the current state of the instance on the hypervisor node. Finally, a soft reboot which fails (probably because the instance operation system didn't respond to the ACPI event in a timely manner) is turned into a hard reboot after libvirt.wait_soft_reboot_seconds. So, we already perform hard reboots when a user asked for a soft reboot in certain error cases.

    Its important to note that the actual reboot mechanism is similar though -- its just how patient we are and what side effects we create that change -- in libvirt they both end up as a shutdown of the virtual machine and then a startup.

    Bug 1072751 reported an interesting edge case with a soft reboot though. If nova-compute crashes after shutting down the virtual machine, but before the virtual machine is started again, then the instance is left in an inconsistent state. We can demonstrate this with a devstack installation:

      Setup the right version of nova cd /opt/stack/nova git checkout dc6942c1218279097cda98bb5ebe4f273720115d Patch nova so it crashes on a soft reboot cat - > /tmp/patch <<EOF > diff --git a/nova/virt/libvirt/ b/nova/virt/libvirt/ > index ce19f22..6c565be 100644 > --- a/nova/virt/libvirt/ > +++ b/nova/virt/libvirt/ > @@ -34,6 +34,7 @@ import itertools > import mmap > import operator > import os > +import sys > import shutil > import tempfile > import time > @@ -2082,6 +2083,10 @@ class LibvirtDriver(driver.ComputeDriver): > # is already shutdown. > if state == power_state.RUNNING: > dom.shutdown() > + > + # NOTE(mikal): temporarily crash > + sys.exit(1) > + > # NOTE(vish): This actually could take slightly longer than the > # FLAG defines depending on how long the get_info > # call takes to return. > EOF patch -p1 < /tmp/patch restart nova-compute inside devstack to make sure you're running the patched version... Boot a victim instance cd ~/devstack source openrc admin glance image-list nova boot --image=cirros-0.3.4-x86_64-uec --flavor=1 foo Soft reboot, and verify its gone nova list nova reboot cacf99de-117d-4ab7-bd12-32cc2265e906 sudo virsh list ...virsh list should now show no virtual machines running as nova-compute crashed before it could start the instance again. However, nova-api knows that the instance should be rebooting... $ nova list +--------------------------------------+------+---------+----------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+---------+----------------+-------------+------------------+ | cacf99de-117d-4ab7-bd12-32cc2265e906 | foo | REBOOT | reboot_started | Running | private= | +--------------------------------------+------+---------+----------------+-------------+------------------+ start nova-compute again, nova-compute detects the missing instance on boot, and tries to start it up again... sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' \ > & echo $! >/opt/stack/status/stack/; fg || \ > echo "n-cpu failed to start" | tee "/opt/stack/status/stack/n-cpu.failure" [...snip...] Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/", line 444, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/", line 213, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/", line 728, in save columns_to_join=_expected_cols(expected_attrs)) File "/opt/stack/nova/nova/db/", line 764, in instance_update_and_get_original expected=expected) File "/opt/stack/nova/nova/db/sqlalchemy/", line 216, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 146, in wrapper ectxt.value = e.inner_exc File "/usr/local/lib/python2.7/dist-packages/oslo_utils/", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 136, in wrapper return f(*args, **kwargs) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2464, in instance_update_and_get_original expected, original=instance_ref)) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2602, in _instance_update raise exc(**exc_props) UnexpectedTaskStateError: Conflict updating instance cacf99de-117d-4ab7-bd12-32cc2265e906. Expected: {'task_state': [u'rebooting_hard', u'reboot_pending_hard', u'reboot_started_hard']}. Actual: {'task_state': u'reboot_started'}

    So what happened here? This is a bit confusing because we asked for a soft reboot of the instance, but the error we are seeing here is that a hard reboot was attempted -- specifically, we're trying to update an instance object but all the task states we expect the instance to be in are related to a hard reboot, but the task state we're actually in is for a soft reboot.

    We need to take a tour of the compute manager code to understand what happened here. nova-compute is implemented at nova/compute/ in the nova code base. Specifically, ComputeVirtAPI.init_host() sets up the service to start handling compute requests for a specific hypervisor node. As part of startup, this method calls ComputeVirtAPI._init_instance() once per instance on the hypervisor node. This method tries to do some sanity checking for each instance that nova thinks should be on the hypervisor:

    • Detecting if the instance was part of a failed evacuation.
    • Detecting instances that are soft deleted, deleting, or in an error state and ignoring them apart from a log message.
    • Detecting instances which we think are fully deleted but aren't in fact gone.
    • Moving instances we thought were booting, but which never completed into an error state. This happens if nova-compute crashes during the instance startup process.
    • Similarly, instances which were rebuilding are moved to an error state as well.
    • Clearing the task state for uncompleted tasks like snapshots or preparing for resize.
    • Finishes deleting instances which were partially deleted last time we saw them.
    • And finally, if the instance should be running but isn't, tries to reboot the instance to get it running.

    It is this final state which is relevant in this case -- we think the instance should be running and its not, so we're going to reboot it. We do that by calling ComputeVirtAPI.reboot_instance(). The code which does this work looks like this:

      try_reboot, reboot_type = self._retry_reboot(context, instance) current_power_state = self._get_power_state(context, instance) if try_reboot: LOG.debug("Instance in transitional state (%(task_state)s) at " "start-up and power state is (%(power_state)s), " "triggering reboot", {'task_state': instance.task_state, 'power_state': current_power_state}, instance=instance) self.reboot_instance(context, instance, block_device_info=None, reboot_type=reboot_type) return [...snip...] def _retry_reboot(self, context, instance): current_power_state = self._get_power_state(context, instance) current_task_state = instance.task_state retry_reboot = False reboot_type = compute_utils.get_reboot_type(current_task_state, current_power_state) pending_soft = (current_task_state == task_states.REBOOT_PENDING and instance.vm_state in vm_states.ALLOW_SOFT_REBOOT) pending_hard = (current_task_state == task_states.REBOOT_PENDING_HARD and instance.vm_state in vm_states.ALLOW_HARD_REBOOT) started_not_running = (current_task_state in [task_states.REBOOT_STARTED, task_states.REBOOT_STARTED_HARD] and current_power_state != power_state.RUNNING) if pending_soft or pending_hard or started_not_running: retry_reboot = True return retry_reboot, reboot_type

    So, we ask ComputeVirtAPI._retry_reboot() if a reboot is required, and if so what type. ComputeVirtAPI._retry_reboot() just uses nova.compute.utils.get_reboot_type() (aliased as compute_utils.get_reboot_type) to determine what type of reboot to use. This is the crux of the matter. Read on for a surprising discovery!

    nova.compute.utils.get_reboot_type() looks like this:

      def get_reboot_type(task_state, current_power_state): """Checks if the current instance state requires a HARD reboot.""" if current_power_state != power_state.RUNNING: return 'HARD' soft_types = [task_states.REBOOT_STARTED, task_states.REBOOT_PENDING, task_states.REBOOTING] reboot_type = 'SOFT' if task_state in soft_types else 'HARD' return reboot_type

    So, after all that it comes down to this. If the instance isn't running, then its a hard reboot. In our case, we shutdown the instance but haven't started it yet, so its not running. This will therefore be a hard reboot. This is where our problem lies -- we chose a hard reboot. The code doesn't blow up until later though -- when we try to do the reboot itself.

      @wrap_exception() @reverts_task_state @wrap_instance_event @wrap_instance_fault def reboot_instance(self, context, instance, block_device_info, reboot_type): """Reboot an instance on this host.""" # acknowledge the request made it to the manager if reboot_type == "SOFT": instance.task_state = task_states.REBOOT_PENDING expected_states = (task_states.REBOOTING, task_states.REBOOT_PENDING, task_states.REBOOT_STARTED) else: instance.task_state = task_states.REBOOT_PENDING_HARD expected_states = (task_states.REBOOTING_HARD, task_states.REBOOT_PENDING_HARD, task_states.REBOOT_STARTED_HARD) context = context.elevated()"Rebooting instance"), context=context, instance=instance) block_device_info = self._get_instance_block_device_info(context, instance) network_info = self.network_api.get_instance_nw_info(context, instance) self._notify_about_instance_usage(context, instance, "reboot.start") instance.power_state = self._get_power_state(context, instance) [...snip...]

    And there's our problem. We have a reboot_type of HARD, which means we set the expected_states to those matching a hard reboot. However, the state the instance is actually in will be one correlating to a soft reboot, because that's what the user requested. We therefore experience an exception when we try to save our changes to the instance. This is the exception we saw above.

    The fix in my patch is simply to change the current task state for an instance in this situation to one matching a hard reboot. It all just works then.

    So why do we decide to use a hard reboot if the current power state is not RUNNING? This code was introduced in this patch and there isn't much discussion in the review comments as to why a hard reboot is the right choice here. That said, we already fall back to a hard reboot in error cases of a soft reboot inside the libvirt driver, and a hard reboot requires less trust of the surrounding state for the instance (block device mappings, networks and all those side effects mentioned at the very beginning), so I think it is the right call.

    In conclusion, we use a hard reboot for soft reboots that fail, and a nova-compute crash during a soft reboot counts as one of those failure cases. So, when nova-compute detects a failed soft reboot, it converts it to a hard reboot and trys again.

    Tags for this post: openstack reboot nova nova-compute
    Related posts: One week of Nova Kilo specifications; How are we going with Nova Kilo specs after our review day?; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Thoughts from the PTL; Chronological list of Juno Nova mid-cycle meetup posts

posted at: 23:30 | path: /openstack | permanent link to this entry

Tue, 22 Sep 2015

First trail run

    So, now I trail run apparently. This was a test run for a hydration vest (thanks Steven Hanley for the loaner!). It was fun, but running up hills is evil.

    Tags for this post: blog canberra trail run
    Related posts: Second trail run; Chicken run; Update on the chickens; Random learning for the day; Boston

posted at: 17:26 | path: /diary | permanent link to this entry

Mon, 21 Sep 2015

Camp Cottermouth

    I spent the weekend at a Scout camp at Camp Cottermouth. The light on the hills here in the mornings is magic.


    Tags for this post: blog pictures 20150920 photo canberra bushwalk

posted at: 18:21 | path: /diary/pictures/20150920 | permanent link to this entry

Wed, 16 Sep 2015

Exploring for a navex

    I feel like I need more detailed maps of Mount Stranger than I currently have in order to layout a possible navex. I there spent a little time this afternoon wandering down the fire trail to mark all the gates in the fence. I need to do a little more of this before its ready for a navex.

    Tags for this post: blog canberra bush walk
    Related posts: Walking to work; First jog, and a walk to Los Altos; A Walk in the Woods

posted at: 15:15 | path: /diary | permanent link to this entry

Sun, 13 Sep 2015

On running

    I've been running for a little while now, but I don't mention it here much. I think I mostly don't mention it because I normally just post photos here, and I don't tend to stop and take happy snaps on my runs. The runs started off pretty modest -- initially I struggled with shin splints after more than a couple of minutes. I've worked through that and a couple of injuries along the way and am consistently doing 5km runs now.

    That said, my longest runs have been in the last week when I did a 7.5km and an 8.1km. I'm building up to 10km, mostly because its a nice round number. I think ultimately trail running might be the thing for me, I get quite bored running around suburbs over and over again.

    Its interesting that I come from an aggressively unsporting family, but yet all of my middle aged siblings and I have started running in the last year or two. Its a mid-life crisis thing perhaps?

    Tags for this post: blog running fitness sport
    Related posts: First jog, and a walk to Los Altos; Martin retires from his work netball league; A year of being more active

posted at: 13:46 | path: /diary | permanent link to this entry

Sat, 12 Sep 2015

CBC Navigation Course

    So today was the day long map and compass class with the Canberra Bushwalking Club. I liked this walk a lot. It was a good length at about 15km, and included a few things I'd wanted to do for a while like wander around McQuoids Hill and the northern crest of Urambi hills. Some nice terrain, and red rocks gorge clearly requires further exploration.


    See more thumbnails

    Tags for this post: blog pictures 20150912 photo canberra bushwalk

posted at: 00:32 | path: /diary/pictures/20150912 | permanent link to this entry

Tue, 08 Sep 2015

Lost and alone in the dark

    I've been doing the Canberra Bush Walking Club navigation course for the last couple of weeks, and last night's exercise was a night time dead reckoning navigation session. The course is really good by the way and I've been enjoying it a lot.

    It should be pointed out that I also wasn't lost, or alone, but it sure was dark.

    Anyway, the basic idea of the exercise is that you're given a hand drawn map, and a set of markers. You determine the bearing from each marker to the next, and the distance to walk. You then set off on your adventure. Getting a bearing or distance wrong matters, because you either need to stop and find the next way point, or carry the mistake on to the next marker. The markers were generally things like "gate in fence" or "two big trees".

    It turns out for me the hardest part is walking in a straight line when its dark. If you look at the GPS logged map below, you can see that the consistent error is that I tend to veer slowly to the right. That's a pretty useful thing to know, because it means I can correct a bit more for it next time. I got the line of march (not the bearing!) pretty badly wrong on the way to the dam, and that resulted in a bit of an adventure to find that way point. I think we missed the next way point as well because we carried the mistake on by setting off from the wrong point on the dam for the next leg.

    I really enjoyed this little walk, and I think I need to do a few more of these to get better at this skill. It seems arbitrary, but if my GPS ever fails and the weather is terrible it might come down to a skill like this keeping me moving in the right direction or not.

    Also, I think this would make a super good exercise for scouts. Now to try and convince them its fun...

    Tags for this post: blog canberra bushwalk navigation
    Related posts: Outline mode numbering of headings; Destinator 3 GPS navigation for the PocketPC

posted at: 16:17 | path: /diary | permanent link to this entry

Mon, 31 Aug 2015

Walk to the Southern Most Point

    I've just realized that I didn't post any pics of my walk to the most southern point of the ACT. The CBC had a planned walk to the southern most point on the ACT border and I was immediately intrigued. So, I took a day off work and gave it a go. It was well worth the experience, especially as Matthew the guide had a deep knowledge of the various huts and so forth in the area. A great day.


    See more thumbnails

    Tags for this post: blog pictures 20150818-southernmostpoint photo canberra bushwalk

posted at: 00:40 | path: /diary/pictures/20150818-southernmostpoint | permanent link to this entry

Thu, 06 Aug 2015

The Crossroad

    ISBN: 9781743519103
    Written by a Victoria Cross recipient, this is the true story of a messed up kid who made something of himself. Mark's dad died of cancer when he was young, and his mum was murdered. Mark then went through a period of being a burden on society, breaking windows for fun and generally being a pain in the butt. But then one day he decided to join the army...

    This book is very well written, and super readable. I enjoyed it a lot, and I think its an important lesson about how troubled teenagers are sometimes that way because of pain in their past, and can often still end up being a valued contributor to society. I have been recommending this book to pretty much everyone I meet since I started reading it.

    Tags for this post: book mark_donaldson combat sas army afghanistan biography australia
    Related posts: Don't Tell Mum I Work On The Rigs; Skimpy; Qantas site; We're getting there; More on the new E-3 Visa; Thanks for the kind word Pia

posted at: 22:04 | path: /book/Mark_Donaldson | permanent link to this entry

Terrible pong

posted at: 17:21 | path: /coding_club | permanent link to this entry

Mon, 03 Aug 2015

Searching for open bugs in a launchpad project

    The launchpad API docs are OMG terrible, and it took me way too long to work out how to do this, so I thought I'd document it for later. Here's how you list all the open bugs in a launchpad project using the API:

      import argparse
      import os
      from launchpadlib import launchpad
      LP_INSTANCE = 'production'
      CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache/')
      def main(username, project):
          lp = launchpad.Launchpad.login_with(username, LP_INSTANCE, CACHE_DIR)
          for bug in lp.projects[project].searchTasks(status=["New",
                                                              "In Progress"]):
              print bug
      if __name__ == '__main__':
          parser = argparse.ArgumentParser(description='Fetch bugs from launchpad')
          args = parser.parse_args()
          main(args.username, args.project)

    Tags for this post: launchpad api
    Related posts: Taking over a launch pad project; Juno nova mid-cycle meetup summary: the next generation Nova API

posted at: 23:10 | path: /launchpad | permanent link to this entry

Sun, 02 Aug 2015

The End of All Things

posted at: 00:39 | path: /book/John_Scalzi | permanent link to this entry

Tue, 28 Jul 2015

Geocaching with a view

    I went to find a couple of geocaches in a jet lag fuelled caching walk this morning. Quite scenic!


    Tags for this post: blog pictures 20150729 photo sydney
    Related posts: Weekend update; Sydney redeems itself, if only a little; Slack talk at SLUG; Bigger improvements; Google? Sydney?; In Sydney!

posted at: 16:38 | path: /diary/pictures/20150729 | permanent link to this entry

Previous page