I think I found a bug in python's unittest.mock library

    Mocking is a pretty common thing to do in unit tests covering OpenStack Nova code. Over the years we've used various mock libraries to do that, with the flavor de jour being unittest.mock. I must say that I strongly prefer unittest.mock to the old mox code we used to write, but I think I just accidentally found a fairly big bug.

    The problem is that python mocks are magical. Its an object where you can call any method name, and the mock will happily pretend it has that method, and return None. You can then later ask what "methods" were called on the mock.

    However, you use the same mock object later to make assertions about what was called. Herein is the problem -- the mock object doesn't know if you're the code under test, or the code that's making assertions. So, if you fat finger the assertion in your test code, the assertion will just quietly map to a non-existent method which returns None, and your code will pass.

    Here's an example:

      #!/usr/bin/python3
      
      from unittest import mock
      
      
      class foo(object):
          def dummy(a, b):
              return a + b
      
      
      @mock.patch.object(foo, 'dummy')
      def call_dummy(mock_dummy):
          f = foo()
          f.dummy(1, 2)
      
          print('Asserting a call should work if the call was made')
          mock_dummy.assert_has_calls([mock.call(1, 2)])
          print('Assertion for expected call passed')
      
          print()
          print('Asserting a call should raise an exception if the call wasn\'t made')
          mock_worked = False
          try:
              mock_dummy.assert_has_calls([mock.call(3, 4)])
          except AssertionError as e:
              mock_worked = True
              print('Expected failure, %s' % e)
      
          if not mock_worked:
              print('*** Assertion should have failed ***')
      
          print()
          print('Asserting a call where the assertion has a typo should fail, but '
                'doesn\'t')
          mock_worked = False
          try:
              mock_dummy.typo_assert_has_calls([mock.call(3, 4)])
          except AssertionError as e:
              mock_worked = True
              print('Expected failure, %s' % e)
              print()
      
          if not mock_worked:
              print('*** Assertion should have failed ***')
              print(mock_dummy.mock_calls)
              print()
      
      
      if __name__ == '__main__':
          call_dummy()
      


    If I run that code, I get this:

      $ python3 mock_assert_errors.py 
      Asserting a call should work if the call was made
      Assertion for expected call passed
      
      Asserting a call should raise an exception if the call wasn't made
      Expected failure, Calls not found.
      Expected: [call(3, 4)]
      Actual: [call(1, 2)]
      
      Asserting a call where the assertion has a typo should fail, but doesn't
      *** Assertion should have failed ***
      [call(1, 2), call.typo_assert_has_calls([call(3, 4)])]
      


    So, we should have been told that typo_assert_has_calls isn't a thing, but we didn't notice because it silently failed. I discovered this when I noticed an assertion with a (smaller than this) typo in its call in a code review yesterday.

    I don't really have a solution to this right now (I'm home sick and not thinking straight), but it would be interesting to see what other people think.

posted at: 21:58 | path: /python | permanent link to this entry

    Add a comment to this post:

    Your name:

    Your email: Email me new comments on this post
      (Your email will not be published on this site, and will only be used to contact you directly with a reply to your comment if needed. Oh, and we'll use it to send you new comments on this post it you selected that checkbox.)


    Your website:

    Comments: