Content here is by Michael Still All opinions are my own.
See recent comments. RSS feed of all comments.

Fri, 01 Jan 2016


      Ok, so now that I've rediscovered walking to trigs while pursuing my hobby of trying to kill Mr Carden, I am enthused to do some more urban walking.

      The only urban trig I haven't been to is YA-90 on Isaacs Ridge. There are also three geocaches along the way I'd like to collect to. I propose we stage a walk from Isaacs as there's plenty of parking and do this one some evening when its cooler.

      Total distance should be about 4km return, with about 150 meters of vertical ascent. It should take about an hour.

    We did this walk on a warm afternoon on new year's day. The walk is pleasant, with plenty of shade for most of the way. In the end we only collected two of the geocaches, as following the ridge line to the third was considered an exercise for another day. A very pleasant walk. A 4.2km walk with 169 meters of ascent.


    Tags for this post: blog pictures 20160101 photo canberra bushwalk

posted at: 11:43 | path: /diary/pictures/20160101 | permanent link to this entry

Tue, 29 Dec 2015

A year of being more active

    A little over a year ago I helped host a panel session at the Paris OpenStack summit. The panel was three telco people talking about NFV, but that's not the bit I really remember. The bit that really stuck with me was how fat I looked in photos from the event. An example:

    I weighed just over 110 kilos (240 lbs). I decided something had to change -- I have a new daughter, and I want to be around to see her well into her life. So, I joined a gym and started bush walking. My first walk was documented here as a walk up Tuggeranong hill. That rapidly became an obsession with climbing hills to survey markers, which then started to include finding geocaches.

    I can't give you a full list of the tangents that one photo from Paris has caused, because the list isn't complete yet and may never be. I now run, swim, ride my bike, and generally sweat on things. Its all fun and has had the unexpected side effect that its helped me cope with work stress much more than I expected.

    I've lost about 15 kilos (30 lbs) so far. Weight loss isn't really the main goal now, but its something I continue to track.

    I thought it would be interesting to list all the places I've walked in Canberra this year, but a simple bullet point list is too long. So instead, here's an interactive map.

    There are a lot more walks I want to do around here. Its just a case of finding the time.

    Tags for this post: blog fitness health weight canberra bushwalk
    Related posts: Weekend update; Bigger improvements; Australian Health Insurance; Memorial service details; Recumbent bikes; An update on Catherine's health

posted at: 17:34 | path: /diary | permanent link to this entry

Mon, 28 Dec 2015

Mount Tennent

      Ok, so I want to do Mount Tennent from the "wrong" side (Apollo Road up the fire trail). The reason for this is that there is a series of 20 something geocaches along the route that I'd like to tick off as well as walking to Tennent.

      I think the total route should be about 13kms, with about 800m of ascent. I propose we leave at least one car at the Apollo Road car park, and then drive the rest to the start point at the Namadgi visitor centre. We can then do the walk, and send one car back to collect the others while we wait under a tree (or something like that).

      Naismith's rule says this walk should take about 5 hours.

    I've been meaning to do this one for ages, but finally got around to doing the walk with a few friends. We went up Tennent from the Namadgi visitor's center, but walked back down on the far side of Tennent which seems less common. The far side is less scenic, but less steep as well I think. Along the way we collected 23 geocaches along the way, and a lovely walk was had by all.

    Just over 1,000 meters vertically, and around 16km horizontally.


    Tags for this post: blog pictures 20151228 photo canberra bushwalk

posted at: 12:28 | path: /diary/pictures/20151228 | permanent link to this entry

Wed, 23 Dec 2015

Mulligan's flat

    I went for a post lunch geocache walk in Mulligan's flat this afternoon as I wind down towards the Christmas break. A good time was had, and a cool wool shed found. I really like this area, perfect for cub walks!


    Tags for this post: blog pictures 20151223 photo canberra bushwalk

posted at: 12:43 | path: /diary/pictures/20151223 | permanent link to this entry

Tue, 22 Dec 2015

Street Orienteering

    I went street orienteering for the first time with my little brother the other day. It was super fun, and a good way to get a run in as well. We even came first in our division (team b grade)!

    Tags for this post: blog orienteering
    Related posts: Scout activity: orienteering at Mount Stranger

posted at: 15:13 | path: /diary | permanent link to this entry

Thu, 03 Dec 2015


    ISBN: 9780733634383
    I've had a bit of a thing about biographies recently, having just read the very good The Crossroad by Mark Donaldson. This book is a very different story, but I think still quite interesting. Kellie was a country girl with no real plans and an impulse control problem. While the book follows her formative years as she parties across Australia in a generally northern direction, I think the underlying story about growing up and finding your way in the world is quite interesting.

    Is this great literature while will enlighten the masses? Probably not. Was it a fun read on a flight and mostly about a teenager with no direction finding her place in the world? Yes.

    Tags for this post: book kellie_arrowsmith biography australia
    Related posts: Don't Tell Mum I Work On The Rigs; The Crossroad; Qantas site; We're getting there; More on the new E-3 Visa; Thanks for the kind word Pia

posted at: 22:54 | path: /book/Kellie_Arrowsmith | permanent link to this entry

The Chronicles of Old Guy

    I found this e-book on Amazon while randomly poking around and read it on a recent set of flights. It was of interest because if looked like another bolo tank book, of which I have read many over the years. That said, its not in strictly the same universe as the other bolo books, and seems more like unofficial fan fiction than something which maps into the universe seamlessly.

    The book is competently written and readable. However, it regularly strays into what I would consider fantasy fiction (medieval warfare, vampires, battling Godzilla) in a way I found jarring and annoying. Overall I don't think I'll read the other books in this series.

    Tags for this post: book timothy_j_gawne bolo combat ai
    Related posts: Against the Tide; The Stars Must Wait; Emerald Sea; All The Weyrs of Pern; Consider Phlebas; Bolos 2: The Unconquerable

posted at: 12:18 | path: /book/Timothy_J_Gawne | permanent link to this entry

Sun, 15 Nov 2015

Mount Stranger one last time

    This is the last walk in this series, which was just a pass through now that the rain has stopped to make sure that we hadn't left any markers or trash lying around after the Scout orienteering a week ago. This area has really grown on me -- I think most people stick to the path down by the river, whereas this whole area has nice terrain, plenty of gates through fences and is just fun to explore. I'm so lucky to have this so close to home.

    Tags for this post: blog canberra bushwalk

posted at: 12:20 | path: /diary | permanent link to this entry

Mon, 09 Nov 2015

A walk in the Orroral Valley

    Last weekend was a walk in the Orroral Valley with a group of scout leaders. Embarrassingly, I'd never been in this area before, and its lovely -- especially at the moment after all the rain we've had. Easy terrain, and a well marked path for this walk. The only catch is that there's either a car shuffle involved, or you need to do a 12km return walk.


    Tags for this post: blog pictures 20151107 photo canberra bushwalk

posted at: 19:13 | path: /diary/pictures/20151107 | permanent link to this entry

Sun, 08 Nov 2015

Scout activity: orienteering at Mount Stranger

    I've run scout activities before, but its always been relatively trivial things like arranging attendance at a Branch level event such as an astronomy night or an environment camp. They've involved consent forms and budgeting and so forth, but never the end to end creation of a thing from scratch. So, I was quite excited to be presented with an opportunity to take the scouts orienteering in an unfamiliar environment.

    I chose the area of nature reserve between Mount Stranger and the Murrumbidgee River because its nice terrain (no tea tree!), but big enough for us to be able to do some long distance bearing navigation, which is a badge requirement some of the scouts are working on at the moment.

    The first step was to scout out (pun intended) the area, and see what sort of options there are for controls and so forth. I'd walked through this area a bit before, as its close to my house, but I'd never bush bashed from the river to the trig before. The first attempt was a simple marking off of the gates along the bicentennial horse trail -- I knew we'd want to cross this somewhere for the long distance leg. That route looked like this:

    The next recce was a wander along a candidate route with some geocaching thrown in for good luck. The geocaching turned out to be quite useful, because on the actual night with the scouts it meant I had a better handle of what was in the area, so when a couple of girls started losing interest I could say stuff like "Did I forget to mention there's an awesome tree house just over there?".

    With that in mind, I then just started slogging out a route -- the long distance leg turned out to be the hardest part here. I wanted to avoid fence crossings as much as possible, and this whole area is littered with barbed wire fences. I think I redid that leg four times before I found a route that I was happy with, which was ironically the first one I'd tried.

    Job done! Now I only needed to walk this route three more times! The first walk was to lay out the orienteering markers before the scouts attacked the course:

    ...and then actually doing the course with some scouts...

    Comparing the two maps, I don't think they did too bad to be honest. There's definitely potential here for more navigation practise, but I think the key there is that practise makes perfect. There shall be more hiking and orienteering in our future! The final walk was just collecting the markers after the event, which I will skip here.

    I put a fair bit of effort into this course, so I'd like to see it used more than once. To that end, I am going to put the documentation online for others to see and use. If you'd like help running this course, drop me a line at and I'd be happy to help.

    Tags for this post: scouts orienteering navex
    Related posts: Street Orienteering; Exploring the Jagungal; Potato Point

posted at: 15:40 | path: /scouts | permanent link to this entry

Mon, 02 Nov 2015

Halo: The Fall of Reach

    ISBN: 0765367297
    As someone who doesn't play computer games and has never played a Halo game, I find myself in the strange position of having read a Halo book. This book is the first in the chronological lineage, and explains the history of the Spartan program which produced the Master Chief. I decided to read this after accidentally watching a Halo mini-movie on Netflix with a sick baby, and deciding it wasn't totally terrible.

    The book is actually ok to my surprise. Its competently written, and on par with much of the other combat fiction I've read. It certainly doesn't feel like its a tie in to a game. I would have liked this book to cover more of the moral issues around the back story to the Spartan program, but those were only briefly considered. Then again, I like a good shoot 'em up as much as the next guy and perhaps that would have been too boring. Overall I enjoyed it and think I might have to read more in this universe.

    Tags for this post: book eric_nylund combat halo engineered_human cranial_computer personal_ai aliens
    Related posts: Halo: The Flood; The Ghost Brigades (2); The Human Division; Old Man's War ; The Last Colony ; The End of All Things

posted at: 02:50 | path: /book/Eric_Nylund | permanent link to this entry

Fri, 16 Oct 2015

CBC Navigation Course

    This was the GPS followup to the previous map and compass navigation exercise. A really nice walk, apart from crazy horse lady. The walk also included another visit to Forster trig. I'm not sure if its the time of year or the direction of approach, but this ascent was much nicer than my previous one, we seemed to avoid most of the prickly things. It would be interesting to recce the other side of the hill and see if I just got unlucky last time, or misread the contours.


    Tags for this post: blog pictures 20151010 photo canberra bushwalk trig_point
    Related posts: Cooleman and Arawang Trigs; Square Rock and Mount Franklin; Quartz trig; Goodwin trig; Tuggeranong Trig (again); Stromlo and Brown Trigs

posted at: 03:30 | path: /diary/pictures/20151010 | permanent link to this entry

Thu, 15 Oct 2015

Geocaching at the base of Mount Ainslie

    I have a bit of a backlog of posts about recent walks which I am working through. Last week I found myself in Ainslie for an appointment which ended at lunch time, so I figured I'd go for a walk. There is a series of geocaches near Campbell Park West (a set of office buildings for non-Canberrans), so off I went.

    The cache series was nice, but the most exciting part of the walking in this area was all the unexploded ordinance (UXO) warnings. I'm sure the area is totally safe, as many people walk through it each day, but it certainly adds an air of adventure to the walk.

    You can find a list of the UXO reports for the ACT on the Department of Defence website, I must say that its not very impressive that the Department has contaminated so many sites around Australia without remediating them -- there are heaps in New South Wales for example. The ACT gets off relatively lightly with only three contaminated sites.

    Its also interesting to note that a suburb very close to me was used as a bombing practise range in World War Two. I'm not aware of anyone in my circle who knew that.

    Anyway. Nice terrain, nice caches, lots of fun. I'd say this would be a good walk for cubs, but I am sure the risk management paperwork for a walk in a UXO are is complicated.


    Tags for this post: blog pictures 20151009 photo canberra bushwalk

posted at: 16:40 | path: /diary/pictures/20151009 | permanent link to this entry

Wed, 07 Oct 2015

Lunchtime geocaching

    So, I didn't get to sleep last night until 4:30am because of a combination of work meetings and small children, so today was a pretty weird day for me. I was having a lot of trouble concentrating at lunch time, so I decided a walk was the least worst thing I could do with my time. I decided to knock off some of the few remaining geocaches in southern Tuggeranong that i haven't found yet.

    This walk was odd -- it started and ended in a little bit of Theodore they never got around to actually building, and I can't find any documentation online about why. It then proceeded through a nice little green strip that has more than its share of rubbish dumped, Cleanup Australia needs to do a visit here! Then there were the Aboriginal axe grinding grooves (read more) just kind of in the middle of the green strip with no informational signage or anything. Finally, a geocache at an abandoned look out, which would have been much nicer if it wasn't being used as an unofficial dump now.

    That said, a nice little walk, but I have no real desire to revisit this one any time soon.


    Tags for this post: blog pictures 20151007 photo canberra bushwalk

posted at: 00:58 | path: /diary/pictures/20151007 | permanent link to this entry

Thu, 01 Oct 2015

A searchable database of walk waypoints

    Over the last year I've become increasingly interested in bush walking, especially around the ACT. It quickly became evident that John Evan's site is an incredibly valuable resource, especially if you're interested in trig points or border markers.

    However, I do most of my early walk planning and visualization in Google Earth before moving to Garmin Basecamp to generate walkable maps. I wanted a way to hook John's database of GPS logs into Google Earth, so that I could plan walks more effectively. For example, John often marks gates in fences, underpasses under major roads, and good routes through scrub in his GPS tracks.

    After a fair bit of playing, I ended up with this KML file which helps me do those things. Its basically magic -- the file is just a link to a search engine which has a database of GPS waypoints based off walks John and I have logged. These are then rendered in Google Earth as if they were in a static KML file. You can also download the search results as KML for editing and so forth as well.

    So, I'd be interested in other people's thoughts on if this is a useful thing. I'd also be very interested in other donated GPS logs of walks and bike rides around Canberra, especially if they have waypoints marked for interesting things. If you have any comments at all, please email me at

    Tags for this post: walks gps search google earth
    Related posts: A busy couple of days; HP iPaq GPS FA256A; Well, that's Google blog search live then; MelbourneIT are into search engine optimisation?; Google book search; Historical revisionism

posted at: 14:59 | path: /walks | permanent link to this entry

Garran green strip

    When I was a teenager my best mate lived in a house which backs onto this smallish reserve and we used to walk his dog here heaps. I had a few spare moments yesterday, so I was keen to do a quick explore and see what its like now. The short answer is that its still nice -- good terrain, nice mature trees, and a few geocaches. I think this one would be a good walk for cubs.


    Tags for this post: blog pictures 20151001 photo canberra bushwalk

posted at: 14:35 | path: /diary/pictures/20151001 | permanent link to this entry

Tue, 29 Sep 2015

Wandering around Curtin

    I decided to go on a little walk on the way home from a work lunch and I don't regret it. This is a nice area, which I was exploring for geocaches. I probably wouldn't have come here at all, but it was the second part of the "Trees of Curtin" walk from Best Bush, Town and Village Walks in and around the ACT that I had done the first half of ages ago.

    I am glad I came back for the second half -- to be honest I was pretty bored with the first half (a bike path beside a major road mostly), whereas this is much more like walking around in nature. The terrain is nice, no thistles, and plenty of horses. A nice afternoon walk overall.

    Now back to reviewing Mitaka specs.


    Tags for this post: blog pictures 20150930 photo canberra bushwalk

posted at: 23:23 | path: /diary/pictures/20150930 | permanent link to this entry

Second trail run

    I went for my second trail run last night. This one was on much rockier terrain, and I ended up tweaking my right knee. I think that was related to the knee having to stabilize as I ran over uneven rocks. I'll experiment by finding a different less awkward trail to run and seeing what happens I suppose.

    Tags for this post: blog canberra trail run
    Related posts: First trail run; Chicken run; Update on the chickens; Random learning for the day; Boston

posted at: 15:06 | path: /diary | permanent link to this entry

Mon, 28 Sep 2015

Old Joe and Goorooyarroo

    Steve, Mel, Michael and I went for a walk to Old Joe trig yesterday. I hadn't been to Goorooyarroo at all before, and was quite impressed. The terrain is nice, with some steep bits as you get close to the border (its clear that the border follows the water catchment from a walk around here). Plenty of nice trees, not too many thistles, and good company. A nice morning walk.

    We bush bashed to the trig straight up the side of the hill, and I think there were gentler (but longer) approaches available -- like for instance how we walked down off the hill following the fence line. That said, the bush bash route wasn't terrible and its probably what I'd do again.

    I need to come back here and walk this border segment, that looks like fun. There are also heaps of geocaches in this area to collect.


    Tags for this post: blog pictures 20150928 photo canberra bushwalk

posted at: 15:34 | path: /diary/pictures/20150928 | permanent link to this entry

Wed, 23 Sep 2015

How we got to test_init_instance_retries_reboot_pending_soft_became_hard

    I've been asked some questions about a recent change to nova that I am responsible for, and I thought it would be easier to address those in this format than trying to explain what's happening in IRC. That way whenever someone compliments me on possibly the longest unit test name ever written, I can point them here.

    Let's start with some definitions. What is the difference between a soft reboot and a hard reboot in Nova? The short answer is that a soft reboot gives the operating system running in the instance an opportunity to respond to an ACPI power event gracefully before the rug is pulled out from under the instance, whereas a hard reboot just punches the instance in the face immediately.

    There is a bit more complexity than that of course, because this is OpenStack. A hard reboot also re-fetches image meta-data, and rebuilds the XML description of the instance that we hand to libvirt. It also re-populates any missing backing files. Finally it ensures that the networking is configured correctly and boots the instance again. In other words, a hard reboot is kind of like an initial instance boot, in that it makes fewer assumptions about how much you can trust the current state of the instance on the hypervisor node. Finally, a soft reboot which fails (probably because the instance operation system didn't respond to the ACPI event in a timely manner) is turned into a hard reboot after libvirt.wait_soft_reboot_seconds. So, we already perform hard reboots when a user asked for a soft reboot in certain error cases.

    Its important to note that the actual reboot mechanism is similar though -- its just how patient we are and what side effects we create that change -- in libvirt they both end up as a shutdown of the virtual machine and then a startup.

    Bug 1072751 reported an interesting edge case with a soft reboot though. If nova-compute crashes after shutting down the virtual machine, but before the virtual machine is started again, then the instance is left in an inconsistent state. We can demonstrate this with a devstack installation:

      Setup the right version of nova cd /opt/stack/nova git checkout dc6942c1218279097cda98bb5ebe4f273720115d Patch nova so it crashes on a soft reboot cat - > /tmp/patch <<EOF > diff --git a/nova/virt/libvirt/ b/nova/virt/libvirt/ > index ce19f22..6c565be 100644 > --- a/nova/virt/libvirt/ > +++ b/nova/virt/libvirt/ > @@ -34,6 +34,7 @@ import itertools > import mmap > import operator > import os > +import sys > import shutil > import tempfile > import time > @@ -2082,6 +2083,10 @@ class LibvirtDriver(driver.ComputeDriver): > # is already shutdown. > if state == power_state.RUNNING: > dom.shutdown() > + > + # NOTE(mikal): temporarily crash > + sys.exit(1) > + > # NOTE(vish): This actually could take slightly longer than the > # FLAG defines depending on how long the get_info > # call takes to return. > EOF patch -p1 < /tmp/patch restart nova-compute inside devstack to make sure you're running the patched version... Boot a victim instance cd ~/devstack source openrc admin glance image-list nova boot --image=cirros-0.3.4-x86_64-uec --flavor=1 foo Soft reboot, and verify its gone nova list nova reboot cacf99de-117d-4ab7-bd12-32cc2265e906 sudo virsh list ...virsh list should now show no virtual machines running as nova-compute crashed before it could start the instance again. However, nova-api knows that the instance should be rebooting... $ nova list +--------------------------------------+------+---------+----------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+---------+----------------+-------------+------------------+ | cacf99de-117d-4ab7-bd12-32cc2265e906 | foo | REBOOT | reboot_started | Running | private= | +--------------------------------------+------+---------+----------------+-------------+------------------+ start nova-compute again, nova-compute detects the missing instance on boot, and tries to start it up again... sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' \ > & echo $! >/opt/stack/status/stack/; fg || \ > echo "n-cpu failed to start" | tee "/opt/stack/status/stack/n-cpu.failure" [...snip...] Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/", line 444, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/", line 213, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/", line 728, in save columns_to_join=_expected_cols(expected_attrs)) File "/opt/stack/nova/nova/db/", line 764, in instance_update_and_get_original expected=expected) File "/opt/stack/nova/nova/db/sqlalchemy/", line 216, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 146, in wrapper ectxt.value = e.inner_exc File "/usr/local/lib/python2.7/dist-packages/oslo_utils/", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python2.7/dist-packages/oslo_db/", line 136, in wrapper return f(*args, **kwargs) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2464, in instance_update_and_get_original expected, original=instance_ref)) File "/opt/stack/nova/nova/db/sqlalchemy/", line 2602, in _instance_update raise exc(**exc_props) UnexpectedTaskStateError: Conflict updating instance cacf99de-117d-4ab7-bd12-32cc2265e906. Expected: {'task_state': [u'rebooting_hard', u'reboot_pending_hard', u'reboot_started_hard']}. Actual: {'task_state': u'reboot_started'}

    So what happened here? This is a bit confusing because we asked for a soft reboot of the instance, but the error we are seeing here is that a hard reboot was attempted -- specifically, we're trying to update an instance object but all the task states we expect the instance to be in are related to a hard reboot, but the task state we're actually in is for a soft reboot.

    We need to take a tour of the compute manager code to understand what happened here. nova-compute is implemented at nova/compute/ in the nova code base. Specifically, ComputeVirtAPI.init_host() sets up the service to start handling compute requests for a specific hypervisor node. As part of startup, this method calls ComputeVirtAPI._init_instance() once per instance on the hypervisor node. This method tries to do some sanity checking for each instance that nova thinks should be on the hypervisor:

    • Detecting if the instance was part of a failed evacuation.
    • Detecting instances that are soft deleted, deleting, or in an error state and ignoring them apart from a log message.
    • Detecting instances which we think are fully deleted but aren't in fact gone.
    • Moving instances we thought were booting, but which never completed into an error state. This happens if nova-compute crashes during the instance startup process.
    • Similarly, instances which were rebuilding are moved to an error state as well.
    • Clearing the task state for uncompleted tasks like snapshots or preparing for resize.
    • Finishes deleting instances which were partially deleted last time we saw them.
    • And finally, if the instance should be running but isn't, tries to reboot the instance to get it running.

    It is this final state which is relevant in this case -- we think the instance should be running and its not, so we're going to reboot it. We do that by calling ComputeVirtAPI.reboot_instance(). The code which does this work looks like this:

      try_reboot, reboot_type = self._retry_reboot(context, instance) current_power_state = self._get_power_state(context, instance) if try_reboot: LOG.debug("Instance in transitional state (%(task_state)s) at " "start-up and power state is (%(power_state)s), " "triggering reboot", {'task_state': instance.task_state, 'power_state': current_power_state}, instance=instance) self.reboot_instance(context, instance, block_device_info=None, reboot_type=reboot_type) return [...snip...] def _retry_reboot(self, context, instance): current_power_state = self._get_power_state(context, instance) current_task_state = instance.task_state retry_reboot = False reboot_type = compute_utils.get_reboot_type(current_task_state, current_power_state) pending_soft = (current_task_state == task_states.REBOOT_PENDING and instance.vm_state in vm_states.ALLOW_SOFT_REBOOT) pending_hard = (current_task_state == task_states.REBOOT_PENDING_HARD and instance.vm_state in vm_states.ALLOW_HARD_REBOOT) started_not_running = (current_task_state in [task_states.REBOOT_STARTED, task_states.REBOOT_STARTED_HARD] and current_power_state != power_state.RUNNING) if pending_soft or pending_hard or started_not_running: retry_reboot = True return retry_reboot, reboot_type

    So, we ask ComputeVirtAPI._retry_reboot() if a reboot is required, and if so what type. ComputeVirtAPI._retry_reboot() just uses nova.compute.utils.get_reboot_type() (aliased as compute_utils.get_reboot_type) to determine what type of reboot to use. This is the crux of the matter. Read on for a surprising discovery!

    nova.compute.utils.get_reboot_type() looks like this:

      def get_reboot_type(task_state, current_power_state): """Checks if the current instance state requires a HARD reboot.""" if current_power_state != power_state.RUNNING: return 'HARD' soft_types = [task_states.REBOOT_STARTED, task_states.REBOOT_PENDING, task_states.REBOOTING] reboot_type = 'SOFT' if task_state in soft_types else 'HARD' return reboot_type

    So, after all that it comes down to this. If the instance isn't running, then its a hard reboot. In our case, we shutdown the instance but haven't started it yet, so its not running. This will therefore be a hard reboot. This is where our problem lies -- we chose a hard reboot. The code doesn't blow up until later though -- when we try to do the reboot itself.

      @wrap_exception() @reverts_task_state @wrap_instance_event @wrap_instance_fault def reboot_instance(self, context, instance, block_device_info, reboot_type): """Reboot an instance on this host.""" # acknowledge the request made it to the manager if reboot_type == "SOFT": instance.task_state = task_states.REBOOT_PENDING expected_states = (task_states.REBOOTING, task_states.REBOOT_PENDING, task_states.REBOOT_STARTED) else: instance.task_state = task_states.REBOOT_PENDING_HARD expected_states = (task_states.REBOOTING_HARD, task_states.REBOOT_PENDING_HARD, task_states.REBOOT_STARTED_HARD) context = context.elevated()"Rebooting instance"), context=context, instance=instance) block_device_info = self._get_instance_block_device_info(context, instance) network_info = self.network_api.get_instance_nw_info(context, instance) self._notify_about_instance_usage(context, instance, "reboot.start") instance.power_state = self._get_power_state(context, instance) [...snip...]

    And there's our problem. We have a reboot_type of HARD, which means we set the expected_states to those matching a hard reboot. However, the state the instance is actually in will be one correlating to a soft reboot, because that's what the user requested. We therefore experience an exception when we try to save our changes to the instance. This is the exception we saw above.

    The fix in my patch is simply to change the current task state for an instance in this situation to one matching a hard reboot. It all just works then.

    So why do we decide to use a hard reboot if the current power state is not RUNNING? This code was introduced in this patch and there isn't much discussion in the review comments as to why a hard reboot is the right choice here. That said, we already fall back to a hard reboot in error cases of a soft reboot inside the libvirt driver, and a hard reboot requires less trust of the surrounding state for the instance (block device mappings, networks and all those side effects mentioned at the very beginning), so I think it is the right call.

    In conclusion, we use a hard reboot for soft reboots that fail, and a nova-compute crash during a soft reboot counts as one of those failure cases. So, when nova-compute detects a failed soft reboot, it converts it to a hard reboot and trys again.

    Tags for this post: openstack reboot nova nova-compute
    Related posts: One week of Nova Kilo specifications; How are we going with Nova Kilo specs after our review day?; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Thoughts from the PTL; Chronological list of Juno Nova mid-cycle meetup posts

posted at: 23:30 | path: /openstack | permanent link to this entry

Previous page