personal web log written by izabeera and dryobates

interfaces python zope duck-typing ABC django

Interfaces and ABCs in Python

by dryobates

Python is powerful dynamically typed language. We can benefit from it's dynamic nature doing a lot of things difficult to achieve in statically typed languages. But that nature has it's price which grows with a project. Unless you resolve into technics learned from statically typed languages.

What I talk about are of course formal interfaces. I see your grimace on your face. "What is he talking about? We have duck-typing" you'll say [1]. Yes, with all my heart I am for duck-typing! But at some complexity level duck-typing is not enough.

In company I work we have over 800 KLOC spread across 50 projects and 3 dozens of libraries. It's quite a lot of code to maintain. Now consider that one library is quite generic (as it should be :) ) and provides service that acts on any object that has given set of methods. That library is used in most of our projects.

For the purpose experiment let that object expected by library have given methods:

class ExpectedObject(object):

     def method1(self):
         pass

     def method2(self):
         pass

     def method3(self):
         pass

So any object with this 3 methods can be passed into our service. Now let our service looks like this:

class Service(object):

     def __init__(self, expected_object):
         self._obj = expected_object

     def do_something1(self):
         try:
             if self._obj.method1():
                 self.do_something2()
         except AttributeError:
             raise IncompatibleObjectError("Passed object should have method1")

     # other methods

And we use it this way:

expected_object = ExpectedObject()
service = Service(expected_object)
# some other code doing heavy preparation
service.do_something1()

If "method1" is absent in expected_object we would like to know it before we do those heavy things. Ideally at service creation time. So let check for "method1" in init part:

class Service(object):

     def __init__(self, expected_object):
         if not hasattr(expected_object, "method1"):
             raise IncompatibleObjectError("Passed object should have method1")
         self._obj = expected_object

     def do_something1(self):
         if self._obj.method1():
             self.do_something2()

     # other methods

We had to sacrifice duck typing. Holy Coding Purity please forgive us but "practicality beats purity" [2] Wait a minute. We need to check for more methods. We expect also "method2" and "method3" later in code. Let's check for that:

class Service(object):

     def __init__(self, expected_object):
         expected_methods = ("method1", "method2", "method3")
         for method in expected_methods:
             if not hasattr(expected_object, method):
                 raise IncompatibleObjectError("Passed object should have method1")
         self._obj = expected_object

     def do_something1(self):
         if self._obj.method1():
             self.do_something2()

     # other methods

Huh. That looks to become not quite practical...

Unit tests

OK, let's try something different. We have unit tests. Let us check correctness in unittests:

# library
class Service(object):

     def __init__(self, expected_object):
         self._obj = expected_object

     def do_something1(self):
         try:
             if self._obj.method1():
                 self.do_something2()
         except AttributeError:
             raise IncompatibleObjectError("Passed object should have method1")

     # other methods

# project tests
class TestExpectedObject(TestCase):

     def test_should_have_compatible_interface(self):
         try:
             expected_object = ExpectedObject()
             expected_object.method1
             expected_object.method2
             expected_object.method3
         except Exception as e:
             assert False, "Incompatible interface: %s" % e

OK, it works! So let us write similar tests for all other 50 projects...

Abstract Base Classes

When we'll finish the obvious thing will happen. "In this world nothing can be said to be certain, except death and taxes" once wrote Benjamin Franklin. Well in software's world nothing can be said to be certain, except change in functionality...

What if our library would expect one more method from passed object? So how to detect which objects needs to be modified and gain "method4"? If we ran tests it won't detect missing methods. Maybe we should use grep? But if method is created dynamically or aliased?

class SomeObject(object):

     def __init__(self):
         setattr(self, "method4", self._other_method)

Searching in 800 KLOC doesn't look to be funny thing :/

In order to cope with a problem of object behaving as we expect, in PEP 3119 appeared Abstract Base Classes [3]. I encourage you to read rationale in this PEP to get a feel why it can be helpful. Now, with a new tool, let us rewrite our library and tests:

# library
from abc import ABCMeta, abstractmethod
class ExpectedObjectBaseClass:

     __metaclass__ = ABCMeta

     @abstractmethod
     def method1(self):
         pass

     @abstractmethod
     def method2(self):
         pass

     @abstractmethod
     def method3(self):
         pass

     @abstractmethod
     def method4(self):
         pass

class Service(object):

     def __init__(self, expected_object):
         self._obj = expected_object

     def do_something1(self):
         try:
             if self._obj.method1():
                 self.do_something2()
         except AttributeError:
             raise IncompatibleObjectError("Passed object should have method1")

     # other methods

# project code
class ExpectedObject(ExpectedObjectBaseClass):

     def method1(self):
         pass
     # ...

# project tests
class TestExpectedObject(TestCase):

     def test_should_have_compatible_interface(self):
         try:
             expected_object = ExpectedObject()
         except Exception as e:
             assert False, "Incompatible interface: %s" % e

Now if ExpectedObject has missing any of methods marked as abstractmethod in parent class then we would get an exception at time of objects creation:

TypeError: Can't instantiate abstract class ExpectedObject with abstract methods method3

Note

One important thing you should note is that we still use duck typing. Abstract Base Classes help library clients to create compatible object interface, but they aren't enforced. You can still pass objects which do not inherit from ExpectedObjectBaseClass and it will still work unless you provide expected methods.

Abstract Base Classes and Django

OK, so we have ABCs and it looks like a great tool to keep our large code base coherent. Most of our projects are Django-based. Let's use ABCs on Django models:

from django.db import models
class SomeModel(ExpectedObjectBaseClass, models.Model):

    def method1(self):
        pass

Now if we create model:

TypeError: Error when calling the metaclass bases metaclass conflict: the
    metaclass of a derived class must be a (non-strict) subclass of the
    metaclasses of all its bases

Wow! Something went wrong. Django models use metaclasses too and it conflicts :(

So we could use adapters:

 from django.db import models

 class SomeModel(models.Model):

     def method1(self):
         pass

 class SomeModelAdapter(ExpectedObjectBaseClass):

     def __init__(self, obj):
         self._obj = obj

     def method1(self):
         return self._obj.method1()

expected_object = SomeModelAdapter(SomeModel())
service = Service(expected_object)
# some other code doing heavy preparation
service.do_something1()

OK that would work but... if I made a mistake and do something like this?

expected_object = SomeModelAdapter(SomeOtherIncompatibleModel())
service = Service(expected_object)
# some other code doing heavy preparation
service.do_something1()

Oh no... It's dead end :(

Zope Interface

Isn't there any other method to use with Django models? Well we always use veteran on the field of Python's interfaces: zope.interface [4].

If you think I'll encourage you to use them on runtime then don't be afraid. I won't. As I have written before: with all my heart I am for duck-typing. Saying that I consider zope.interface [4] as convenient piece of software that in conjunction with tests can help you keep your projects easier to maintain.

So let's try with it:

# library
class ExpectedObjectBaseClass(zope.interface.Interface):

    def method1():
        """ Required method """

# project
from library import ExpectedObjectBaseClass
from django.db import models

class SomeModel(models.Model):
    zope.interface.implements(ExpectedObjectBaseClass)

    def method1(self):
        pass

# project tests
from zope.interface.verify import verifyClass

class TestExpectedObject(TestCase):

    def test_should_have_compatible_interface(self):
        try:
            verifyClass(ExpectedObjectBaseClass, ExpectedObject)
        except Exception as e:
            assert False, "Incompatible interface: %s" % e

With above solution we don't affect runtime behaviour. We use duck-typing and the only use for interfaces is for documentation purposes and pre-deployment checking. We could write such testing functions by ourselves but Zope's community did it for us.

Confession

Now that you see the point of using some kind of informal interfaces in Python I have to tell you that... I have lied to you a little :)

You see I have intentionally use wrong first test as an example. I have written this test:

class TestExpectedObject(TestCase):

     def test_should_have_compatible_interface(self):
         try:
             expected_object = ExpectedObject()
             expected_object.method1
             expected_object.method2
             expected_object.method3
         except Exception as e:
             assert False, "Incompatible interface: %s" % e

Do you see a problem with above test? What object's behaviour it tests? None! Tests should check object's dynamic behaviour. That's why they are used even in statically typed languages. Above test checks for object's interface which has no value in running system. Important is not like it looks but how it behaves.

If we had a 100% tests coverage we won't have to check if object has correct methods or not. If something would change our tests would fail and point us error location with surgical precision.

Interfaces are great for documentation. It's very convenient way of telling "I need this" and as such ABCs are used in standard library.

Besides that in interfaces can be interested for:

  • programmers coming from statically typed languages - switching to think in dynamically typed categories can be difficult and... I'm not sure is it always desired as new fresh ideas can come to Python from that world.
  • lazy programmers that don't want write tests ;)
  • for those which have a lot of legacy code and interfaces are one step forward to get this legacy fully testes in some future.

With over 10 years in Python professional programming I can't explain myself as newcomer in dynamically typed world. Although I consider myself as lazy (in terms of creative laziness ;) ) I'd rather put myself in the last group: I had a lot of legacy code to maintain. But I believe that in some day all those code will be fully tested. I try all my new code do with Test Driven Development and strongly encourage my co-workers to do the same. That gives us code 100% covered by tests. Of course it doesn't mean it will be 100% correct, but problems with incorrect object's interface can be easily detected with tests.

[1]Duck-typing https://docs.python.org/2/glossary.html#term-duck-typing
[2]The Zen of Python https://www.python.org/dev/peps/pep-0020/
[3]Abstract Base Classes https://www.python.org/dev/peps/pep-3119/
[4](1, 2) Zope Interfaces http://docs.zope.org/zope.interface/
dryobates
dryobates
Jakub Stolarski. Software engineer. I work professionally as programmer since 2005. Speeding up software development with Test Driven Development, task automation and optimization for performance are things that focus my mind from my early career up to now. If you ask me for my religion: Python, Vim and FreeBSD are my trinity ;) Email: jakub@stolarscy.com

Archive

Tag cloud