Reusable ansible roles for Juju charms

I’ve been writing Juju charms to automate the deployment of a few different services at work, which all happen to be wsgi applications… some Django apps, others with other frameworks. I’ve been using the ansible support for writing charms which makes charm authoring simpler, but even then, essentially each wsgi service charm needs to do the same thing:

1. Setup specific users
2. Install the built code into a specific location
3. Install any package dependencies
4. Relate to a backend (could be postgresql, could be elasticsearch)
5. Render the settings
6. Setup a wsgi (gunicorn) service (via a subordinate charm)
7. Setup log rotation
8. Support updating to a new codebase without upgrading the charm
9. Support rolling updates a new codebase

Only three of the above really change slightly from service to service: which package dependencies are required, the rendering of the settings and the backend relation (which usually just causes the settings to be rerendered anyway).

After trying (and failing) to create a nice reusable wsgi-app charm, I’ve switched to utilise ansible’s built-in support for reusable roles and created a charm-bootstrap-wsgi repo on github, which demonstrates all of the above out of the box (the README has an example rolling upgrade). The charm’s playbook is very simple, just reusing the wsgi-app role:

 

roles:
    - role: wsgi-app
      listen_port: 8080
      wsgi_application: example_wsgi:application
      code_archive: "{{ build_label }}/example-wsgi-app.tar.bzip2"
      when: build_label != ''

 

and only needs to do two things itself:

tasks:
    - name: Install any required packages for your app.
      apt: pkg={{ item }} state=latest update_cache=yes
      with_items:
        - python-django
        - python-django-celery
      tags:
        - install
        - upgrade-charm

    - name: Write any custom configuration files
      debug: msg="You'd write any custom config files here, then notify the 'Restart wsgi' handler."
      tags:
        - config-changed
        # Also any backend relation-changed hooks for databases etc.
      notify:
        - Restart wsgi

 

Everything else is provided by the reusable wsgi-app role. For the moment I’ve got the source of the reusable roles in a separate github repo, but I’d like to get these into the charm-helpers project itself eventually. Of course there will be cases where the service charm may need to do quite a bit more custom functionality, but if we can encapsulate and reuse as much as possible, it’s a win for all of us.

If you’re interested in chatting about easier charms with ansible (or any issues you can see), we’ll be getting together for a hangout tomorrow (Jun 11, 2014 at 1400UTC).

Easier juju charms with Python helpers

Have you ever wished you could just declare the installed state of your juju charm like this?

deploy_user:
    group.present:
        - gid: 1800
    user.present:
        - uid: 1800
        - gid: 1800
        - createhome: False
        - require:
            - group: deploy_user

exampleapp:
    group.present:
        - gid: 1500
    user.present:
        - uid: 1500
        - gid: 1500
        - createhome: False
        - require:
            - group: exampleapp


/srv/{{ service_name }}:
    file.directory:
        - group: exampleapp
        - user: exampleapp
        - require:
            - user: exampleapp
        - recurse:
            - user
            - group


/srv/{{ service_name }}/{{ instance_type }}-logs:
    file.directory:
        - makedirs: True

While writing charms for Juju a long time ago, one of the things that I struggled with was testing the hook code – specifically the install hook code where the machine state is set up (ie. packages installed, directories created with correct permissions, config files setup etc.) Often the test code would be fragile – at best you can patch some attributes of your module (like “code_location = ‘/srv/example.com/code'”) to a tmp dir and test the state correctly, but at worst you end up testing the behaviour of your code (ie. os.mkdir was called with the correct user/group etc.). Either way, it wasn’t fun to write and iterate those tests.

But support has improved over the past year with the charmhelpers library. And recently I landed a branch adding support for declaring saltstack states in yaml, like the above example. That means that the install hook of your hooks.py can be reduced to something like:

import charmhelpers.core.hookenv
import charmhelpers.payload.execd
import charmhelpers.contrib.saltstack


hooks = charmhelpers.core.hookenv.Hooks()


@hooks.hook()
def install():
    """Setup the machine dependencies and installed state."""
    charmhelpers.contrib.saltstack.install_salt_support()
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/dependencies.yaml')
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/installed.yaml')


# Other hooks...

if __name__ == "__main__":
    hooks.execute(sys.argv)

…letting you focus on testing and writing the actual hook functionality (like relation-set’s etc. I’d like to add some test helpers that will automatically check the syntax of the state yaml files and template rendering output, but haven’t yet).

Hopefully we can add similar support for puppet and Ansible soon too, so that the charmer can choose the tools they want to use to declare the local machine state.

A few other things that I found valuable while writing my charm:

• Use a branch for charmhelpers – this way you can make improvements to the library as you develop and not be dependent on your changes landing straight away to deploy (thanks Sidnei – I think I just copied that idea from one of his charms). The easiest way that I found for that was to install the branch into mycharm/lib so that it’s included in both dev and when you deploy (with a small snippet in your hooks.py.
• Make it easy to deploy your local charm from the branch… the easiest way I found was a link-test-juju-repo make target – I’m not sure what other people do here?
• In terms of writing actual hook functionality (like relation-set events etc), I found the easiest way to develop the charm was to iterate within a debug-hook session. Something like:
  1. write new test+code then juju upgrade-charm or add-relation
  2. run the hook and if it fails…
  3. fix and test right there within the debug-hook
  4. put the code back into my actual charm branch and update the test
  5. restore the system state in debug hook
  6. then juju upgrade-charm again to ensure it works, if it fails, iterate from 3.
• Use the built-in support of template rendering from saltstack for rendering any config files that you need.

I don’t think I’d really appreciated the beauty of what juju is doing until, after testing my charm locally together with a gunicorn charm and a solr backend, I then setup a config using juju-deployer to create a full stack with an Apache front-end, a cache load balancer for multiple squid caches, as well as a load balancer in front of potentially multiple instances of my charms wsgi app, then a back-end loadbalancer in between my app and the (multiple) solr backends… and it just works.

Testing django migrations

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

Develop and deploy with virtualenv

What is the dream setup for developing and deploying Django apps? I’m looking for a solution that I can use consistently to deploy apps to servers where I may or may not have the ability to install system packages, or where I might need my app temporarily to use a newer version of a system-installed package while giving other apps running on the same server breathing space to update (think: updating a system-installed Django package on a server running four independent apps).

Specifically, the goals I have for this scenario are:

• It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment),
• Updating any virtualenv environment should be automatic, but transparent (ie. if the pip requirements.txt changes, the next time I run tests, devserver or deployed server, it’ll automatically ensure the virtualenv is correct),
• I shouldn’t have to wait unnecessarily for virtualenvs to be created (ie. if I make a change to the requirements to try a new version of a package, and then change it back, I don’t want to re-create the original virtualenv). Similarly, if I revert a deployment to a previous version, the previous virtualenv should still be available.
•  For deployment, the virtualenv shouldn’t unnecessarily replace system python packages, but allow this as an option (ie. not a –no-site-packages virtualenv).

There are a lot of virtualenv/fabric posts out there for both development and deployment, and using a SHA of the requirements.txt seems an obvious way to go. What I ended up with for my project was this develop and deploy with virtualenv snippet which so far is working quite well (although I’m yet to try a deploy where I override system packages). If the deployed version is using virtualenv exclusively, the requirements.txt file can be shared, but otherwise it would just be a matter of including the requirements.txt for the deploy with the other configuration data (settings.py etc.).

If you can see any reasons why this is not a good idea, or improvements, please let me know!

Sharing your development environment across branches

If you’re working on a project that bootstraps a development environment (using virtualenv or similar) it can be painful to bootstrap the environment for every bug/feature branch that you work on. There are lots of ways you can avoid this (local cache etc.), but if you’re using bzr to manage your branches, a single working directory for all your branches can solve the problem for you. Most bzr users will be familiar with this, but people new to bzr might benefit.

The one-time setup is pretty straight forward – create a repository with a local copy of trunk, and a branch of trunk for your new feature/bug:

$ cd dev/
$ bzr init-repo myproject
$ cd myproject
$ bzr branch lp:myproject trunk
$ bzr branch trunk myfeaturex

From now on, you can simply `cd trunk && bzr pull` when you want to update your local trunk [1]. Now we’ll create a light-weight checkout and bootstrap our environment there. I tend to call this directory ‘current_work’ but whatever works for you. Also, most of the projects I’m working on are using fabric for setting up the virtualenv and other tidbits, but replace that with your own bootstrap command:

$ bzr checkout --lightweight myfeaturex current_work
$ cd current_work && fab bootstrap

Assuming everything went well, you now have your development environment setup and are ready to work on myfeaturex. Make changes, commit etc., push to launchpad – it’ll all be for your myfeaturex branch. But what if half-way through, you need to switch and work on an urgent bug? Easy: commit (or shelve) any changes you’ve got for your current branch, then:

$ cd ..
$ bzr branch trunk criticalbugx
$ cd current_work
$ bzr switch ../criticalbugx  [2]

Edit 2011-05-23: See Ricardo’s comment – the above 4 lines can be replaced by his two.

That’s it – your current_work directory now reflects the criticalbugx branch – and is already bootstrapped [3], you can work away, commit push etc., and then switch back to your original branch with:

$ bzr switch ../myfeaturex

If you’re working on large branches, once you’ve got the above workflow, you may want to try next keeping large features reviewable with bzr pipelines.

[1] It’s a good idea to never do any work in your trunk tree… always work in a branch of trunk.
[2] The ../ is not actually required, but when you’ve lots of branches with long names, it means you can tab-complete the branch name.
[3] Of course, if one of your branches changes any virtualenv requirements or similar, you’ll need to re-bootstrap, but that isn’t so often in practise. Or sometimes a new version of a dependency is released and you won’t get it until you rebootstrap.

Keeping large features reviewable with bzr pipelines

One of the things I love is reviewing smaller targeted branches – 400-500 lines focused on one or two particular points. But you don’t want to be blocked on the review of one branch to start the next dependent branch. Here’s a quick screencast (with text version below) that I made for some colleagues showing how Bzr’s pipeline plugin can help your workflow in this situation:

Text version

Assuming you’ve already got a lightweight checkout of your project code, create a new branch for part-1 of your feature and switch to it:

$ bzr branch trunk part-1
$ cd current_work;bzr switch ../part-1

Make your changes, commit and push to Launchpad:

$ bzr commit -m "Finished part 1 of feature-x that does x, y and z";bzr push

While that branch is being reviewed, you can start on the next part of the feature:

$ bzr add-pipe part-2
Created and switched to pipe "part-2"
$ bzr pipes
    part-1
*   part-2

Now make your further changes for the feature and commit/push those as above (Launchpad will create a new branch for part-2). Now let’s say that the review for part-1 comes back, requiring a small change, so we switch back to part-1, make the change, commit and pump it (so that our part-2 branch will include the change also):

$ bzr switch part-1
$ bzr pipes
*   part-1
    part-2
(make changes)
$ bzr commit -m "Changes from review";bzr pump

Now we can see the changes have been merged into the part-2 branch:

$ bzr switch part-2
$ bzr log --limit 1 --line
102: Michael Nelson 2011-01-13 [merge] Merged part-1 into part-2.

So, let’s say we’re now ready to submit part-2 for review. The handy lp-submit bzr plugin will automatically know to set part-1 as a prerequisite branch, so that the diff shown on the part-2 merge proposal includes only the changes from part-2 (as well as making sure that your branches are pushed up to LP):

$ bzr lp-submit
Pushing to lp:~michael.nelson/rnr-server/part-1
Pushing to lp:~michael.nelson/rnr-server/part-2

Nice! Now, what if there have been some changes to trunk in the mean-time that you want to include in your branch? Easy, just:

$ bzr pump --from-submit

and pipelines will automatically merge trunk into part-1, commit, then merge part-1 into part-2.

More info with `bzr help pipeline` (once it’s installed), or more information (including how to get it) on the Bazaar Pipeline wiki page.

Mocking in Python just got easier

Voidspace just released Mock 0.4.0 – an update to the excellent Mock library which brings a few conveniences and easier patching of module and class-level attributes.

This makes tests using mocks easier to read, for example inside a test case:

self.assertEquals(my_mock.my_method.call_args,     (('goodbye',),{'hello': False}))

 

Can now be written as:

my_mock.my_method.assert_called_with('goodbye', hello=False)


Check it out!

Mocking with Django and Google AppEngine

Since working with RSpec over the past 6 months – a Behaviour-Driven Development framework for ruby – I’ve been wondering if there’s anything comparable in Python (my preferred tool for development!). One of the things I love about RSpec is the ease with which Mock objects can be used to keep tests focused.

While there are a number of mock libraries around for Python most don’t result in particularly readable test code. But I was pleasantly suprised to discover Michael Foord’s Mocking and Testing utilities.

A simple example: I’ve got a Django application hosted on Google AppEngine and say I want to write a simple test to verify that my view does require the user to be logged in with their Google account and if not, redirects appropriately – but I don’t want to have to manually log a user in, or even use the Google api as part of my test. Here’s a snippet showing how easy this is with Mock:

from mock import Mock
from google.appengine.api import users
   

class MySpecialView(TestCase):

    def setUp():
        """ Create the required mocks for the view tests """
        # Mock the get_current_user method of the google users api
        users.get_current_user = Mock()

        # Create a mock user that we'll pretend is logged in
        mock_user = Mock()

        # Just for readability, save the special app-engine login url as 
        # an instance variable
        self.url = reverse('my-special-view-name')
        self.login_url = "http://testserver/_ah/login?continue=http%%3A//testserver%s" % self.url

    def test_logged_in_user_can_access_page(self):
        """A logged in user should not be redirected to the login page"""
        # Set the return value for the mocked 
        # get_current_user method:
        users.get_current_user.return_value = mock_user

        response = do_request()

        # Make sure the mock method was called
        self.assertTrue(users.get_current_user.called)

        # And the redirect did not take place, but the 
        # normal template was rendered...
        self.assertTemplateUsed(response, 'myapp/overview.html')
       
    def test_anonymous_user_is_redirected_to_login(self):
        """ An anonymous user should be redirected to the login page"""
        # Set the google api's get_current_user method to return None
        users.get_current_user.return_value = None

        response = self.do_request()

        # Make sure the mock method was called
        self.assertTrue(users.get_current_user.called)

        # And that the redirect took place... note we can't use
        # the normal assertRedirects due to the app-engine specific
        # login url.
        self.assertEquals(response.status_code, 302)
        self.assertEquals(
            response['location'],
            "http://testserver/_ah/login?continue=http%%3A//testserver%s" % self.url
        )

Easy! Thanks Michael. The Mock object has lots of other goodies of course (such as auto-setting all the mock-methods from the real object, testing for call parameters etc.).