Something-driven development

Software development thoughts around Ubuntu, Python, Golang and other tools

Juju + Ansible = simpler charms

with 7 comments

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s hooks.py to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
    playbook_path='playbooks/site.yaml',
    default_hooks=['start', 'stop', 'config-changed',
                   'solr-relation-changed'])

@hooks.hook()
def install():
    charmhelpers.contrib.ansible.install_ansible_support(from_ppa=True)

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed 🙂

Advertisements

Written by Michael

November 8, 2013 at 11:42 am

Posted in Uncategorized

Tagged with ,

Easier juju charms with Python helpers

with one comment

logo-jujuHave you ever wished you could just declare the installed state of your juju charm like this?

deploy_user:
    group.present:
        - gid: 1800
    user.present:
        - uid: 1800
        - gid: 1800
        - createhome: False
        - require:
            - group: deploy_user

exampleapp:
    group.present:
        - gid: 1500
    user.present:
        - uid: 1500
        - gid: 1500
        - createhome: False
        - require:
            - group: exampleapp


/srv/{{ service_name }}:
    file.directory:
        - group: exampleapp
        - user: exampleapp
        - require:
            - user: exampleapp
        - recurse:
            - user
            - group


/srv/{{ service_name }}/{{ instance_type }}-logs:
    file.directory:
        - makedirs: True

While writing charms for Juju a long time ago, one of the things that I struggled with was testing the hook code – specifically the install hook code where the machine state is set up (ie. packages installed, directories created with correct permissions, config files setup etc.) Often the test code would be fragile – at best you can patch some attributes of your module (like “code_location = ‘/srv/example.com/code'”) to a tmp dir and test the state correctly, but at worst you end up testing the behaviour of your code (ie. os.mkdir was called with the correct user/group etc.). Either way, it wasn’t fun to write and iterate those tests.

But support has improved over the past year with the charmhelpers library. And recently I landed a branch adding support for declaring saltstack states in yaml, like the above example. That means that the install hook of your hooks.py can be reduced to something like:

import charmhelpers.core.hookenv
import charmhelpers.payload.execd
import charmhelpers.contrib.saltstack


hooks = charmhelpers.core.hookenv.Hooks()


@hooks.hook()
def install():
    """Setup the machine dependencies and installed state."""
    charmhelpers.contrib.saltstack.install_salt_support()
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/dependencies.yaml')
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/installed.yaml')


# Other hooks...

if __name__ == "__main__":
    hooks.execute(sys.argv)

…letting you focus on testing and writing the actual hook functionality (like relation-set’s etc. I’d like to add some test helpers that will automatically check the syntax of the state yaml files and template rendering output, but haven’t yet).

Hopefully we can add similar support for puppet and Ansible soon too, so that the charmer can choose the tools they want to use to declare the local machine state.

A few other things that I found valuable while writing my charm:

  • Use a branch for charmhelpers – this way you can make improvements to the library as you develop and not be dependent on your changes landing straight away to deploy (thanks Sidnei – I think I just copied that idea from one of his charms). The easiest way that I found for that was to install the branch into mycharm/lib so that it’s included in both dev and when you deploy (with a small snippet in your hooks.py.
  • Make it easy to deploy your local charm from the branch… the easiest way I found was a link-test-juju-repo make target – I’m not sure what other people do here?
  • In terms of writing actual hook functionality (like relation-set events etc), I found the easiest way to develop the charm was to iterate within a debug-hook session. Something like:
    1. write new test+code then juju upgrade-charm or add-relation
    2. run the hook and if it fails…
    3. fix and test right there within the debug-hook
    4. put the code back into my actual charm branch and update the test
    5. restore the system state in debug hook
    6. then juju upgrade-charm again to ensure it works, if it fails, iterate from 3.
  • Use the built-in support of template rendering from saltstack for rendering any config files that you need.

I don’t think I’d really appreciated the beauty of what juju is doing until, after testing my charm locally together with a gunicorn charm and a solr backend, I then setup a config using juju-deployer to create a full stack with an Apache front-end, a cache load balancer for multiple squid caches, as well as a load balancer in front of potentially multiple instances of my charms wsgi app, then a back-end loadbalancer in between my app and the (multiple) solr backends… and it just works.

Written by Michael

June 24, 2013 at 2:42 pm

Posted in juju, python, testing

Testing django migrations

with 8 comments

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

Written by Michael

March 1, 2013 at 6:18 pm

Posted in django, python, testing

Tagged with , ,

Connecting android fragment events

leave a comment »

Android UI Fragments look like a great way to build re-usable elements within an app, but they don’t work exactly as expected out of the box (well, exactly is I expected – but that could be a lack of experience with android):

After defining an onClick event on a button within my fragment:

  <Button android:id="@+id/add_goal_button"
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:text="@string/button_create"
      android:onClick="addGoal" />

I’d expected this event to be routed directly to my fragment class without the containing Activity class needing to know about it – but instead, the addGoal() method is expected on the containing Activity instead.

To connect the fragment event directly to a click handler on the fragment class (so the view doesn’t need to be handling the fragment events, you can do the following instead (thanks Brill Pappin):

public class NewGoalFragment extends Fragment {
	
	...
	
	@Override
	public View onCreateView(LayoutInflater inflater, ViewGroup container,
			                 Bundle savedInstanceState) {
		final View fragmentView = inflater.inflate(R.layout.fragment_new_goal,
							   container, false);
		...
		Button addGoalButton = (Button) fragmentView.findViewById(R.id.add_goal_button);
		addGoalButton.setOnClickListener(new OnClickListener() {
			@Override
			public void onClick(final View v) {
				// Pass the fragmentView through to the handler
				// so that findViewById can be used to get a handle on
				// the fragments own views.
				addGoal(fragmentView);
			}
		});
		return fragmentView;
	}
	
    public void addGoal(View view) {    	
    	EditText newGoal = (EditText) view.findViewById(R.id.new_goal);
	...
    }

Written by Michael

January 1, 2013 at 6:21 pm

Posted in android

A tour of Go – the web crawler exercise

with 5 comments

The last exercise in the Go Tour – parallelizing a web crawler – turned out to be quite a bit more interesting than I’d expected. If anyone has suggested improvements from which I can learn a bit more, or their own solutions posted, let me know – my exercise solution is on github. I’ve tried to stick to the tour content (ie. only using channels rather than the sync package for accessing shared data).

Spoiler Alert: If you are learning Golang and haven’t yet worked through the Go-Tour, go and do so now. If you get stuck, keep struggling, take a break, try again in a few days etc., before looking at other peoples’ solutions.

The solution I ended up with has a Crawl() function very similar to the original, just with two extra function parameters:

func Crawl(url string, depth int, fetcher Fetcher,
	startCrawl func(string) bool, crawlComplete chan string) {

	if depth <= 0 {
		crawlComplete <- url
 		return
	}

	body, urls, err := fetcher.Fetch(url)
	if err != nil {
		fmt.Println(err)
		crawlComplete <- url
 		return
	}

	fmt.Printf("found: %s %q\n", url, body)
	for _, u := range urls {
		if startCrawl(u) {
			go Crawl(u, depth-1, fetcher, startCrawl, crawlComplete)
		}
	}
	crawlComplete <- url
}

The two parameters are:

  • startCrawl func(url string) bool – used as a check before spawning a new ‘go Crawl(url)’ to ensure that we don’t crawl the same url twice.
  • crawlComplete chan string – used to signal that the Crawl function has fetched the page and finished spawning any child go-routines.

These two resources are created and passed in to the initial Crawl() call in the main() function:

func main() {
	startCrawl := make(chan StartCrawlData)
	crawlComplete := make(chan string)
	quitNow := make(chan bool)
	go processCrawls(startCrawl, crawlComplete, quitNow)

	// Returns whether a crawl should be started for a given
	// URL.
	startCrawlFn := func(url string) bool {
		resultChan := make(chan bool)
		startCrawl <- StartCrawlData{url, resultChan}
		return <-resultChan
	}

	Crawl("http://golang.org/", 4, fetcher, startCrawlFn,
		crawlComplete)

	<-quitNow
}

Access to the shared state of which urls have been crawled and when all Crawls() have finished etc., is managed via those channels in the processCrawls() go-routine, so that the main() can simply call the first Crawl() and then wait to quit. I want to check how cheap the temporary creation of a channel is (for the return value of the startCrawlFn above) – I think I saw this method on an earlier GoLang tutorial example, but otherwise I’m happy with the solution :-).

Other solutions to learn from:

Written by Michael

November 14, 2012 at 12:32 pm

Posted in golang

Kanban for kids

with 8 comments

Facing the arrival of another child, I set out a few months ago to experiment with ways to help each other get things done together as a family without becoming a task-master of a father. A very simplified Kanban board seemed like a good fit for our kitchen wall, as it visualises what needs doing or other fun things that are planned, helps us to help each other when needed, and enables the kids to take part in the planning and organising of the day.

The photo of our board above is what we’ve ended up with after a few months. It is really a habit tracker – with most of the items on the board being things that we do daily, every few days, weekly or fortnightly. Some points that we’ve found helpful:

  • Items specific to the kids are colour coded (ie. we have a “Put clothes away” item for each child in their colour).
  • We have some incentives – each item has a number of associated “service points” [1]. For eg. 5 for putting away clothes, 15 for taking the dog out for a walk etc. (agreed with the kids and adjusted on feedback).
  • We have incentives to have fun doing items together – if you complete an item together with someone else, you get 50% extra service points, simply to encourage having fun doing things together (we put a magnet on the item to signify this).
  • We very rarely tell the kids to do something – in extreme circumstances I’ve told my daughter I need her to take the dog out now as I can’t do it myself, but it’s something we generally avoid, in fact…
  • The kids generally choose what item they want to do next and when they want to do it (ie. “You’d like to help daddy set the table for lunch? Shall we do it now or after playing Lego for a while?”).
  • Each item has a drawing and is laminated with a small magnet, so that we can stack certain items together (for eg, we have a stack of cards related to the dog which are done in order each day – dog breakfast, walk 7am-ish, Walk 11am-ish, etc).
  • We start each day by pulling things that we want to do into the backlog (in addition to the daily stuff). Some items are weekly, which we store in a stack for each day on the top-left. Other items are ad-hoc which we pull in from the mess on the bottom left.
  • We limit the number of cards in the TODO and Doing lanes to 4, to help us help each flow (or review whether we really want to do something).
  • We don’t put normal stuff like playing lego or dressups there – the kids can play whenever they want without feeling they need to have an item on the board 😛

So far it’s been a great help, but as always, a continual learning process. Each child is motivated by different things, and yet the end goal is that we’d all enjoy helping each other without extrinsic motivation. Interestingly, one of the most freeing things has been for the kids (and my wife) to get used to the fact that it’s OK to say, no, I don’t feel like doing that today, I’m just going to move the card over to Thursday for the moment. That is, the board is something we control – a helpful way to organise and communicate – it does not tell us what we have to do.

[1] Service points – The kids can use the service points at the end of the day in ways such as staying up a bit later (1minute per point, within reason), or going out on their own with mum or dad for an ice-cream desert (50 points). We haven’t yet found a good solution for left over points, and are currently converting to Euro-cents as pocket money at the end of the week which works well in terms of value, but I don’t like the direct connection of being paid money to do stuff around the house. Another option I’m trying at the moment is using saved service points as bargaining power for larger items that the kids need (like a new school bag). I’d be glad to hear of other ideas.

Written by Michael

October 3, 2012 at 11:08 am

Posted in Uncategorized

A DerbyJS version of the TodoMVC project

with 2 comments

I’ve spent a few evenings this week implementing a derbyjs version of the Todo spec for the TodoMVC project [1] – and it was a great way to learn more about the end-to-end framework, and appreciate how neat the model-view bindings really are. Here’s a 2 minute demo showing the normal TodoMVC functionality as well as the collaborative editing which Derby brings out of the box:

 

It’s amazing how simple Derby’s model-view bindings enable the code to be. It’s really just two files containing the functionality:

The other files are just setup (define which queries are allowed, define the express server, and some custom style on top of the base.css from TodoMVC). Well done Nate and Brian, and the DerbyJS community (which seems to be growing quite a bit over the last few weeks)!

[1] I’ve still got a few things todo before I can submit a pull-request to get this added to TodoMVC.

Written by Michael

August 9, 2012 at 9:28 am

Posted in javascript

Tagged with ,