Content here is by Michael Still All opinions are my own.
See recent comments. RSS feed of all comments.

Second trail run

    I went for my second trail run last night. This one was on much rockier terrain, and I ended up tweaking my right knee. I think that was related to the knee having to stabilize as I ran over uneven rocks. I'll experiment by finding a different less awkward trail to run and seeing what happens I suppose.

    Tags for this post: blog canberra trail run
    Related posts: First trail run; Chicken run; Update on the chickens; Random learning for the day; Boston

posted at: 15:06 | path: /diary | permanent link to this entry

Mon, 28 Sep 2015

Old Joe and Goorooyarroo

    Steve, Mel, Michael and I went for a walk to Old Joe trig yesterday. I hadn't been to Goorooyarroo at all before, and was quite impressed. The terrain is nice, with some steep bits as you get close to the border (its clear that the border follows the water catchment from a walk around here). Plenty of nice trees, not too many thistles, and good company. A nice morning walk.

    We bush bashed to the trig straight up the side of the hill, and I think there were gentler (but longer) approaches available -- like for instance how we walked down off the hill following the fence line. That said, the bush bash route wasn't terrible and its probably what I'd do again.

    I need to come back here and walk this border segment, that looks like fun. There are also heaps of geocaches in this area to collect.


    Tags for this post: blog pictures 20150928 photo canberra bushwalk

posted at: 15:34 | path: /diary/pictures/20150928 | permanent link to this entry

Wed, 23 Sep 2015

How we got to test_init_instance_retries_reboot_pending_soft_became_hard

    I've been asked some questions about a recent change to nova that I am responsible for, and I thought it would be easier to address those in this format than trying to explain what's happening in IRC. That way whenever someone compliments me on possibly the longest unit test name ever written, I can point them here.

    Let's start with some definitions. What is the difference between a soft reboot and a hard reboot in Nova? The short answer is that a soft reboot gives the operating system running in the instance an opportunity to respond to an ACPI power event gracefully before the rug is pulled out from under the instance, whereas a hard reboot just punches the instance in the face immediately.

    There is a bit more complexity than that of course, because this is OpenStack. A hard reboot also re-fetches image meta-data, and rebuilds the XML description of the instance that we hand to libvirt. It also re-populates any missing backing files. Finally it ensures that the networking is configured correctly and boots the instance again. In other words, a hard reboot is kind of like an initial instance boot, in that it makes fewer assumptions about how much you can trust the current state of the instance on the hypervisor node. Finally, a soft reboot which fails (probably because the instance operation system didn't respond to the ACPI event in a timely manner) is turned into a hard reboot after libvirt.wait_soft_reboot_seconds. So, we already perform hard reboots when a user asked for a soft reboot in certain error cases.

    Its important to note that the actual reboot mechanism is similar though -- its just how patient we are and what side effects we create that change -- in libvirt they both end up as a shutdown of the virtual machine and then a startup.

    Bug 1072751 reported an interesting edge case with a soft reboot though. If nova-compute crashes after shutting down the virtual machine, but before the virtual machine is started again, then the instance is left in an inconsistent state. We can demonstrate this with a devstack installation:

      Setup the right version of nova cd /opt/stack/nova git checkout dc6942c1218279097cda98bb5ebe4f273720115d Patch nova so it crashes on a soft reboot cat - > /tmp/patch <<EOF > diff --git a/nova/virt/libvirt/ b/nova/virt/libvirt/ > index ce19f22..6c565be 100644 > --- a/nova/virt/libvirt/ > +++ b/nova/virt/libvirt/ > @@ -34,6 +34,7 @@ import itertools > import mmap > import operator > import os > +import sys > import shutil > import tempfile > import time > @@ -2082,6 +2083,10 @@ class LibvirtDriver(driver.ComputeDriver): > # is already shutdown. > if state == power_state.RUNNING: > dom.shutdown() > + > + # NOTE(mikal): temporarily crash > + sys.exit(1) > + > # NOTE(vish): This actually could take slightly longer than the > # FLAG defines depending on how long the get_info > # call takes to return. > EOF patch -p1 < /tmp/patch restart nova-compute inside devstack to make sure you're running the patched version... Boot a victim instance cd ~/devstack source openrc admin glance image-list nova boot --image=cirros-0.3.4-x86_64-uec --flavor=1 foo Soft reboot, and verify its gone nova list nova reboot cacf99de-117d-4ab7-bd12-32cc2265e906 sudo virsh list ...virsh list should now show no virtual machines running as nova-compute crashed before it could start the instance again. However, nova-api knows that the instance should be rebooting... $ nova list +--------------------------------------+------+---------+----------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+---------+----------------+-------------+------------------+ | cacf99de-117d-4ab7-bd12-32cc2265e906 | foo | REBOOT | reboot_started | Running | private= | +--------------------------------------+------+---------+----------------+-------------+------------------+ start nova-compute again, nova-compute detects the missing instance on boot, and tries to start it up again... sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' \ > & echo $! >/opt/stack/status/stack/; fg || \ > echo "n-cpu failed to start" | tee "/opt/stack/status/stack/n-cpu.failure" [...snip...] Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/", line 444, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/", line 213, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/", line 728, in save columns_to_join=_expected_cols(expected_attrs)) File "/opt/stack/nova/nova/db/", line 764, in instance_update_and_get_original expected=expected) File "/opt/stack/nova/nova/db/sqlalchemy/", line 216, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 146, in wrapper ectxt.value = e.inner_exc File "/usr/local/lib/python2.7/dist-packages/oslo_utils/", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 136, in wrapper return f(*args, **kwargs) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2464, in instance_update_and_get_original expected, original=instance_ref)) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2602, in _instance_update raise exc(**exc_props) UnexpectedTaskStateError: Conflict updating instance cacf99de-117d-4ab7-bd12-32cc2265e906. Expected: {'task_state': [u'rebooting_hard', u'reboot_pending_hard', u'reboot_started_hard']}. Actual: {'task_state': u'reboot_started'}

    So what happened here? This is a bit confusing because we asked for a soft reboot of the instance, but the error we are seeing here is that a hard reboot was attempted -- specifically, we're trying to update an instance object but all the task states we expect the instance to be in are related to a hard reboot, but the task state we're actually in is for a soft reboot.

    We need to take a tour of the compute manager code to understand what happened here. nova-compute is implemented at nova/compute/ in the nova code base. Specifically, ComputeVirtAPI.init_host() sets up the service to start handling compute requests for a specific hypervisor node. As part of startup, this method calls ComputeVirtAPI._init_instance() once per instance on the hypervisor node. This method tries to do some sanity checking for each instance that nova thinks should be on the hypervisor:

    • Detecting if the instance was part of a failed evacuation.
    • Detecting instances that are soft deleted, deleting, or in an error state and ignoring them apart from a log message.
    • Detecting instances which we think are fully deleted but aren't in fact gone.
    • Moving instances we thought were booting, but which never completed into an error state. This happens if nova-compute crashes during the instance startup process.
    • Similarly, instances which were rebuilding are moved to an error state as well.
    • Clearing the task state for uncompleted tasks like snapshots or preparing for resize.
    • Finishes deleting instances which were partially deleted last time we saw them.
    • And finally, if the instance should be running but isn't, tries to reboot the instance to get it running.

    It is this final state which is relevant in this case -- we think the instance should be running and its not, so we're going to reboot it. We do that by calling ComputeVirtAPI.reboot_instance(). The code which does this work looks like this:

      try_reboot, reboot_type = self._retry_reboot(context, instance) current_power_state = self._get_power_state(context, instance) if try_reboot: LOG.debug("Instance in transitional state (%(task_state)s) at " "start-up and power state is (%(power_state)s), " "triggering reboot", {'task_state': instance.task_state, 'power_state': current_power_state}, instance=instance) self.reboot_instance(context, instance, block_device_info=None, reboot_type=reboot_type) return [...snip...] def _retry_reboot(self, context, instance): current_power_state = self._get_power_state(context, instance) current_task_state = instance.task_state retry_reboot = False reboot_type = compute_utils.get_reboot_type(current_task_state, current_power_state) pending_soft = (current_task_state == task_states.REBOOT_PENDING and instance.vm_state in vm_states.ALLOW_SOFT_REBOOT) pending_hard = (current_task_state == task_states.REBOOT_PENDING_HARD and instance.vm_state in vm_states.ALLOW_HARD_REBOOT) started_not_running = (current_task_state in [task_states.REBOOT_STARTED, task_states.REBOOT_STARTED_HARD] and current_power_state != power_state.RUNNING) if pending_soft or pending_hard or started_not_running: retry_reboot = True return retry_reboot, reboot_type

    So, we ask ComputeVirtAPI._retry_reboot() if a reboot is required, and if so what type. ComputeVirtAPI._retry_reboot() just uses nova.compute.utils.get_reboot_type() (aliased as compute_utils.get_reboot_type) to determine what type of reboot to use. This is the crux of the matter. Read on for a surprising discovery!

    nova.compute.utils.get_reboot_type() looks like this:

      def get_reboot_type(task_state, current_power_state): """Checks if the current instance state requires a HARD reboot.""" if current_power_state != power_state.RUNNING: return 'HARD' soft_types = [task_states.REBOOT_STARTED, task_states.REBOOT_PENDING, task_states.REBOOTING] reboot_type = 'SOFT' if task_state in soft_types else 'HARD' return reboot_type

    So, after all that it comes down to this. If the instance isn't running, then its a hard reboot. In our case, we shutdown the instance but haven't started it yet, so its not running. This will therefore be a hard reboot. This is where our problem lies -- we chose a hard reboot. The code doesn't blow up until later though -- when we try to do the reboot itself.

      @wrap_exception() @reverts_task_state @wrap_instance_event @wrap_instance_fault def reboot_instance(self, context, instance, block_device_info, reboot_type): """Reboot an instance on this host.""" # acknowledge the request made it to the manager if reboot_type == "SOFT": instance.task_state = task_states.REBOOT_PENDING expected_states = (task_states.REBOOTING, task_states.REBOOT_PENDING, task_states.REBOOT_STARTED) else: instance.task_state = task_states.REBOOT_PENDING_HARD expected_states = (task_states.REBOOTING_HARD, task_states.REBOOT_PENDING_HARD, task_states.REBOOT_STARTED_HARD) context = context.elevated()"Rebooting instance"), context=context, instance=instance) block_device_info = self._get_instance_block_device_info(context, instance) network_info = self.network_api.get_instance_nw_info(context, instance) self._notify_about_instance_usage(context, instance, "reboot.start") instance.power_state = self._get_power_state(context, instance) [...snip...]

    And there's our problem. We have a reboot_type of HARD, which means we set the expected_states to those matching a hard reboot. However, the state the instance is actually in will be one correlating to a soft reboot, because that's what the user requested. We therefore experience an exception when we try to save our changes to the instance. This is the exception we saw above.

    The fix in my patch is simply to change the current task state for an instance in this situation to one matching a hard reboot. It all just works then.

    So why do we decide to use a hard reboot if the current power state is not RUNNING? This code was introduced in this patch and there isn't much discussion in the review comments as to why a hard reboot is the right choice here. That said, we already fall back to a hard reboot in error cases of a soft reboot inside the libvirt driver, and a hard reboot requires less trust of the surrounding state for the instance (block device mappings, networks and all those side effects mentioned at the very beginning), so I think it is the right call.

    In conclusion, we use a hard reboot for soft reboots that fail, and a nova-compute crash during a soft reboot counts as one of those failure cases. So, when nova-compute detects a failed soft reboot, it converts it to a hard reboot and trys again.

    Tags for this post: openstack reboot nova nova-compute
    Related posts: One week of Nova Kilo specifications; How are we going with Nova Kilo specs after our review day?; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Thoughts from the PTL; Chronological list of Juno Nova mid-cycle meetup posts

posted at: 23:30 | path: /openstack | permanent link to this entry

Tue, 22 Sep 2015

First trail run

    So, now I trail run apparently. This was a test run for a hydration vest (thanks Steven Hanley for the loaner!). It was fun, but running up hills is evil.

    Tags for this post: blog canberra trail run
    Related posts: Second trail run; Chicken run; Update on the chickens; Random learning for the day; Boston

posted at: 17:26 | path: /diary | permanent link to this entry

Mon, 21 Sep 2015

Camp Cottermouth

    I spent the weekend at a Scout camp at Camp Cottermouth. The light on the hills here in the mornings is magic.


    Tags for this post: blog pictures 20150920 photo canberra bushwalk

posted at: 18:21 | path: /diary/pictures/20150920 | permanent link to this entry

Wed, 16 Sep 2015

Exploring for a navex

    I feel like I need more detailed maps of Mount Stranger than I currently have in order to layout a possible navex. I there spent a little time this afternoon wandering down the fire trail to mark all the gates in the fence. I need to do a little more of this before its ready for a navex.

    Tags for this post: blog canberra bush walk
    Related posts: Walking to work; First jog, and a walk to Los Altos; A Walk in the Woods

posted at: 15:15 | path: /diary | permanent link to this entry

Sun, 13 Sep 2015

On running

    I've been running for a little while now, but I don't mention it here much. I think I mostly don't mention it because I normally just post photos here, and I don't tend to stop and take happy snaps on my runs. The runs started off pretty modest -- initially I struggled with shin splints after more than a couple of minutes. I've worked through that and a couple of injuries along the way and am consistently doing 5km runs now.

    That said, my longest runs have been in the last week when I did a 7.5km and an 8.1km. I'm building up to 10km, mostly because its a nice round number. I think ultimately trail running might be the thing for me, I get quite bored running around suburbs over and over again.

    Its interesting that I come from an aggressively unsporting family, but yet all of my middle aged siblings and I have started running in the last year or two. Its a mid-life crisis thing perhaps?

    Tags for this post: blog running fitness sport
    Related posts: First jog, and a walk to Los Altos; Martin retires from his work netball league; A year of being more active

posted at: 13:46 | path: /diary | permanent link to this entry

Sat, 12 Sep 2015

CBC Navigation Course

    So today was the day long map and compass class with the Canberra Bushwalking Club. I liked this walk a lot. It was a good length at about 15km, and included a few things I'd wanted to do for a while like wander around McQuoids Hill and the northern crest of Urambi hills. Some nice terrain, and red rocks gorge clearly requires further exploration.


    See more thumbnails

    Tags for this post: blog pictures 20150912 photo canberra bushwalk

posted at: 00:32 | path: /diary/pictures/20150912 | permanent link to this entry

Tue, 08 Sep 2015

Lost and alone in the dark

    I've been doing the Canberra Bush Walking Club navigation course for the last couple of weeks, and last night's exercise was a night time dead reckoning navigation session. The course is really good by the way and I've been enjoying it a lot.

    It should be pointed out that I also wasn't lost, or alone, but it sure was dark.

    Anyway, the basic idea of the exercise is that you're given a hand drawn map, and a set of markers. You determine the bearing from each marker to the next, and the distance to walk. You then set off on your adventure. Getting a bearing or distance wrong matters, because you either need to stop and find the next way point, or carry the mistake on to the next marker. The markers were generally things like "gate in fence" or "two big trees".

    It turns out for me the hardest part is walking in a straight line when its dark. If you look at the GPS logged map below, you can see that the consistent error is that I tend to veer slowly to the right. That's a pretty useful thing to know, because it means I can correct a bit more for it next time. I got the line of march (not the bearing!) pretty badly wrong on the way to the dam, and that resulted in a bit of an adventure to find that way point. I think we missed the next way point as well because we carried the mistake on by setting off from the wrong point on the dam for the next leg.

    I really enjoyed this little walk, and I think I need to do a few more of these to get better at this skill. It seems arbitrary, but if my GPS ever fails and the weather is terrible it might come down to a skill like this keeping me moving in the right direction or not.

    Also, I think this would make a super good exercise for scouts. Now to try and convince them its fun...

    Tags for this post: blog canberra bushwalk navigation
    Related posts: Outline mode numbering of headings; Destinator 3 GPS navigation for the PocketPC

posted at: 16:17 | path: /diary | permanent link to this entry

Mon, 31 Aug 2015

Walk to the Southern Most Point

    I've just realized that I didn't post any pics of my walk to the most southern point of the ACT. The CBC had a planned walk to the southern most point on the ACT border and I was immediately intrigued. So, I took a day off work and gave it a go. It was well worth the experience, especially as Matthew the guide had a deep knowledge of the various huts and so forth in the area. A great day.


    See more thumbnails

    Tags for this post: blog pictures 20150818-southernmostpoint photo canberra bushwalk

posted at: 00:40 | path: /diary/pictures/20150818-southernmostpoint | permanent link to this entry

Thu, 06 Aug 2015

The Crossroad

    ISBN: 9781743519103
    Written by a Victoria Cross recipient, this is the true story of a messed up kid who made something of himself. Mark's dad died of cancer when he was young, and his mum was murdered. Mark then went through a period of being a burden on society, breaking windows for fun and generally being a pain in the butt. But then one day he decided to join the army...

    This book is very well written, and super readable. I enjoyed it a lot, and I think its an important lesson about how troubled teenagers are sometimes that way because of pain in their past, and can often still end up being a valued contributor to society. I have been recommending this book to pretty much everyone I meet since I started reading it.

    Tags for this post: book mark_donaldson combat sas army afghanistan biography australia
    Related posts: Don't Tell Mum I Work On The Rigs; Skimpy; Qantas site; We're getting there; More on the new E-3 Visa; Thanks for the kind word Pia

posted at: 22:04 | path: /book/Mark_Donaldson | permanent link to this entry

Terrible pong

posted at: 17:21 | path: /coding_club | permanent link to this entry

Mon, 03 Aug 2015

Searching for open bugs in a launchpad project

    The launchpad API docs are OMG terrible, and it took me way too long to work out how to do this, so I thought I'd document it for later. Here's how you list all the open bugs in a launchpad project using the API:

      import argparse
      import os
      from launchpadlib import launchpad
      LP_INSTANCE = 'production'
      CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache/')
      def main(username, project):
          lp = launchpad.Launchpad.login_with(username, LP_INSTANCE, CACHE_DIR)
          for bug in lp.projects[project].searchTasks(status=["New",
                                                              "In Progress"]):
              print bug
      if __name__ == '__main__':
          parser = argparse.ArgumentParser(description='Fetch bugs from launchpad')
          args = parser.parse_args()
          main(args.username, args.project)

    Tags for this post: launchpad api
    Related posts: Taking over a launch pad project; Juno nova mid-cycle meetup summary: the next generation Nova API

posted at: 23:10 | path: /launchpad | permanent link to this entry

Sun, 02 Aug 2015

The End of All Things

posted at: 00:39 | path: /book/John_Scalzi | permanent link to this entry

Tue, 28 Jul 2015

Geocaching with a view

    I went to find a couple of geocaches in a jet lag fuelled caching walk this morning. Quite scenic!


    Tags for this post: blog pictures 20150729 photo sydney
    Related posts: Weekend update; Sydney redeems itself, if only a little; Slack talk at SLUG; Bigger improvements; Google? Sydney?; In Sydney!

posted at: 16:38 | path: /diary/pictures/20150729 | permanent link to this entry

Mon, 27 Jul 2015

Chet and I went on an adventure to LA-96

    So, I've been fascinated with American nuclear history for ages, and Chet and I got talking about what if any nuclear launch facilities there were in LA. We found LA-96 online and set off on an expedition to explore. An interesting site, its a pity there are no radars left there. Apparently SF-88 is the place to go for tours from vets and radars.


    See more thumbnails

    I also made a quick and dirty 360 degree video of the view of LA from the top of the nike control radar tower:

    Tags for this post: blog pictures 20150727-nike_missile photo california
    Related posts: Arrived yesterday; Napa Valley; First jog, and a walk to Los Altos; Sitting in the California DMV office; VTA station for the Santa Clara Convention Center; Packing

posted at: 17:07 | path: /diary/pictures/20150727-nike_missile | permanent link to this entry

Sun, 26 Jul 2015

Views from a lookout on Mulholland Drive, Bel Air

posted at: 20:37 | path: /diary/pictures/20150727-lookout | permanent link to this entry

Geocaching with TheDevilDuck

    In what amounts to possibly the longest LAX layover ever, I've been hanging out with Chet at his place in Altadena for a few days on the way home after the Nova mid-cycle meetup. We decided that being the dorks that we are we should do some geocaching. This is just some quick pics some unexpected bush land -- I never thought LA would be so close to nature, but this part certainly is.


    Tags for this post: blog pictures 20150727 photo california bushwalk
    Related posts: Arrived yesterday; Napa Valley; First jog, and a walk to Los Altos; Sitting in the California DMV office; VTA station for the Santa Clara Convention Center; Packing

posted at: 13:52 | path: /diary/pictures/20150727 | permanent link to this entry

Sat, 18 Jul 2015

Casuarina Sands to Kambah Pool

    I did a walk with the Canberra Bushwalking Club from Casuarina Sands (in the Cotter) to Kambah Pool (just near my house) yesterday. It was very enjoyable. I'm not going to pretend to be excellent at write ups for walks, but will note that the walk leader John Evans has a very detailed blog post about the walk up already. We found a bunch of geocaches along the way, with John doing most of the work and ChifleyGrrrl and I providing encouragement and scrambling skills. A very enjoyable day.


    See more thumbnails

    Tags for this post: blog pictures 20150718-casurina_sands_to_kambah_pool photo canberra bushwalk

posted at: 15:35 | path: /diary/pictures/20150718-casurina_sands_to_kambah_pool | permanent link to this entry

Wed, 15 Jul 2015


    I am on vacation this week, so I took this afternoon to do some walking and geocaching...

    That included a return visit to Narrabundah trig to clean up some geocaches I missed last visit:


    And exploring the Lindsay Pryor arboretum because I am trying to collect the complete set of arboretums in Canberra:


    And then finally the Majura trig, which was a new one for me:


    See more thumbnails

    I enjoyed the afternoon. I found a fair few geocaches, and walked for about five hours (not including driving between the locations). I would have spent more time geocaching at Majura, except I made it back to the car after sunset as it was.

    Tags for this post: blog pictures 20150715-wanderings photo canberra bushwalk trig_point
    Related posts: Cooleman and Arawang Trigs; Square Rock and Mount Franklin; Quartz trig; Goodwin trig; Tuggeranong Trig (again); Stromlo and Brown Trigs

posted at: 14:30 | path: /diary/pictures/20150715-wanderings | permanent link to this entry

Previous page