Behind the scenes I’m still loving Yahoo User Interfaces application framework together with the unit-test api, which allow me to create tests like “creating the view should render stats for existing goals”.

Next week I hope to start implementing the server-sync and investigating integration with Ubuntu’s notification system. Once I’ve sorted out the application-cache and build/packaging etc. for both Ubuntu and Android, I want to do a initial dummy release and then start iterating the features that will make Open Goal Tracker actually useful for facilitating social-yet-individual learning, both within and without of classrooms.

You can play with the little open-goal-tracker prototype (I’ve only tried webkit so far) or run the tests yourself.

1. Enables the user to specify a branch of a Python package containing the Django app/project for deploy. This python package will be `python setup.py install`’d on the instance, but it also
2. Enables you to configure extra debian packages to be installed first so that your requirements can be installed in a more reliable/trusted manner, along with the standard required packages (apache2, libapache2-mod-wsgi etc.). Here’s the example charm config used for apps.ubuntu.com,
3. Creates a django.wsgi and httpd.conf ready to serve your app, automatically collecting all the static content of your installed Django apps to be served separately from the same Apache virtual host,
4. When it receives a database relation change, it creates some local settings, overriding the database settings of your branch, sync’s and migrates the database (a noop if it’s the second unit) and restarts apache (See the database_settings.pp manifest for more details).

Here’s a quick demo which puts up a postgresql unit and two app servers with these commands:

$ juju deploy --repository ~/charms local:postgresql
$ juju deploy --config ubuntu-app-dir.yaml --repository ~/apache-django-wsgi/ local:apache-django-wsgi
$ juju add-relation postgresql:db apache-django-wsgi
$ juju add-unit apache-django-wsgi

Things that I think need to be improved or I’m uncertain about:

1. `gem install puppet-module` is included in the install hook (a 3rd way of installing something on the system :/). I wanted to use the vcsrepo puppet module to define bzr resource types and puppet-module-tool seems to be the way to install 3rd-party puppet modules. Using this resource-type enables a simple initial_state.pp manifest. Of course, it’d be great to have ‘necessary’ tools like that in the archive instead.
2. The initial_state.pp manifest pulls the django app package to /home/ubuntu/django-app-branch and then pip installs it on the system. Requiring the app to be a valid python package seemed sensible (in terms of ensuring it is correctly installed with its requirements satisfied) while still allowing the user to go one step further if they like and provide a debian package instead of a python package in a branch (which I assume we would do ultimately for production deploys?)
3. Currently it’s just a very simple apache setup. I think ideally the static file serving should be done by a separate unit in the charm (ie. an instance running a stripped down apache2 or lighttpd). Also, I would have liked to have used an ‘official’ or ‘blessed’ puppet apache module to benefit from someone else’s experience, but I couldn’t see one that stood out as such.
4. Currently the charm assumes that your project contains the configuration info (ie. a settings.py, urls.py etc.), of which the database settings can be simply overridden for deploy. There should be an additional option to specify a configuration branch (and it shouldn’t assume that you’re using django-configglue), as well as other options like django_debug, static_url etc.
5. The charm should also export an interface (?) that can be used by a load balancer charm.

I’m loving the MVC separation in the client code, separation of templates, ability to use real links for navigating within the client (which will also work when copy-n-pasted to hit the server) etc.

If anyone is interested, you can play with the code yourself:

$ bzr branch lp:~michael.nelson/open-goal-tracker/html5-client-play
$ cd html5-client-play
$ fab test

which will setup the dev environment and run the django tests… followed by

$ fab runserver

To play with the demo html5 client: http://localhost:8000/static/index.html
To run the javascript unit tests: http://localhost:8000/static/app/tests/test_models.html

or find out more about the purpose of the app with the first 2 mins of this video from a year ago outlining where i hope to go with the application: http://www.youtube.com/watch?v=OT7P-u-86sM (when time permits ).

More recently I’ve started thinking about best-practices for deploying services using juju, while still using puppet to setup individual units. As a simple experiment, I wrote a juju charm to deploy an irssi service [1] to dig around. Here’s what I’ve found so far [2]. The first is kind of obvious, but worth mentioning:

Install hooks can be trivial:

#!/bin/bash
sudo apt-get -y install puppet

juju-log "Initialising machine state."
puppet apply $PWD/hooks/initial_state.pp

Normally the corresponding manifest (see initial_state.pp) would be a little more complicated, but in this example it’s hardly worth mentioning.

Juju config changes can utilise Puppet’s Facter infrastructure:

This enables juju config options to be passed through to puppet, so that config-changed hooks can be equally simple:

#!/bin/bash
juju-log "Getting config options"
username=`config-get username`
public_key=`config-get public_key`

juju-log "Configuring irssi for user"
# We specify custom facts so that they're accessible in the manifest.
FACTER_username=$username FACTER_public_key=$public_key puppet apply $PWD/hooks/configured_state.pp

In this example, it is the configured state manifest that is more interesting (see configured_state.pp). It adds the user to the system, sets up byobu with an irssi window ready to go, and adds the given public ssh key enabling the user to login.

The same would go for other juju hooks (db-relation-changed etc.), which is quite neat – getting the best of both worlds: the charm user can still think in terms of deploying services, while the charm author can use puppets declarative syntax to define the machine states.

Next up: I hope to experiment with an optional puppet master for a real project (something simple like the Ubuntu App directory), so that

1. a project can be deployed without the (probably private) puppet-master to create a close-to-production environment, while
2. configuring a puppet-master in the juju config would enable production deploys (or deploys of exact replicas of production to a separate environment for testing).

If you’re interested in seeing the simple irssi charm, the following 2min video demos:

# Deploy an irssi service
$ juju deploy --repository=/home/ubuntu/mycharms  local:oneiric/irssi
# Configure it so a user can login
$ juju set irssi username=michael public_key=AAAA...
# Login to find irssi already up and running in a byobu window
$ ssh michael@new.ip.address

and the code is on Launchpad.

[1] Yes, irssi is not particularly useful as a juju service (as I don’t want multiple units, or relating it to other services etc.), but it suited my purposes for a simple experiment that also automates something I can use for working in the cloud.

[2] I’m not a puppet or juju expert, so if you’ve got any comments or improvements, don’t hesitate.

The one pain-point was that I SSH’d into my own box at home – and if my home connection went (anything from a thunderstorm to a curious kid pulling a chord), I had to then work on my much slower netbook.

Now, after reading on an email thread that Aaron was doing something similar, I’m running a tiny cloud instance for my irc and development work, and documenting everything that I’m doing to get my dev environment just right – so that I can automate the setup (not really something for an ensemble formula – just a simple script).

That, and having all my music in the cloud for the past 6 months make switching computers and/or physical locations a non-obtrusive part of my work-day.

If anyone has small tips that make their own dev environment enjoyable, let me know!

Specifically, the goals I have for this scenario are:

• It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment),
• Updating any virtualenv environment should be automatic, but transparent (ie. if the pip requirements.txt changes, the next time I run tests, devserver or deployed server, it’ll automatically ensure the virtualenv is correct),
• I shouldn’t have to wait unnecessarily for virtualenvs to be created (ie. if I make a change to the requirements to try a new version of a package, and then change it back, I don’t want to re-create the original virtualenv). Similarly, if I revert a deployment to a previous version, the previous virtualenv should still be available.
•  For deployment, the virtualenv shouldn’t unnecessarily replace system python packages, but allow this as an option (ie. not a –no-site-packages virtualenv).

There are a lot of virtualenv/fabric posts out there for both development and deployment, and using a SHA of the requirements.txt seems an obvious way to go. What I ended up with for my project was this develop and deploy with virtualenv snippet which so far is working quite well (although I’m yet to try a deploy where I override system packages). If the deployed version is using virtualenv exclusively, the requirements.txt file can be shared, but otherwise it would just be a matter of including the requirements.txt for the deploy with the other configuration data (settings.py etc.).

If you can see any reasons why this is not a good idea, or improvements, please let me know!

The one-time setup is pretty straight forward – create a repository with a local copy of trunk, and a branch of trunk for your new feature/bug:

$ cd dev/
$ bzr init-repo myproject
$ cd myproject
$ bzr branch lp:myproject trunk
$ bzr branch trunk myfeaturex

From now on, you can simply `cd trunk && bzr pull` when you want to update your local trunk [1]. Now we’ll create a light-weight checkout and bootstrap our environment there. I tend to call this directory ‘current_work’ but whatever works for you. Also, most of the projects I’m working on are using fabric for setting up the virtualenv and other tidbits, but replace that with your own bootstrap command:

$ bzr checkout --lightweight myfeaturex current_work
$ cd current_work && fab bootstrap

Assuming everything went well, you now have your development environment setup and are ready to work on myfeaturex. Make changes, commit etc., push to launchpad – it’ll all be for your myfeaturex branch. But what if half-way through, you need to switch and work on an urgent bug? Easy: commit (or shelve) any changes you’ve got for your current branch, then:

$ cd ..
$ bzr branch trunk criticalbugx
$ cd current_work
$ bzr switch ../criticalbugx  [2]

Edit 2011-05-23: See Ricardo’s comment – the above 4 lines can be replaced by his two.

That’s it – your current_work directory now reflects the criticalbugx branch – and is already bootstrapped [3], you can work away, commit push etc., and then switch back to your original branch with:

$ bzr switch ../myfeaturex

If you’re working on large branches, once you’ve got the above workflow, you may want to try next keeping large features reviewable with bzr pipelines.

[1] It’s a good idea to never do any work in your trunk tree… always work in a branch of trunk.
[2] The ../ is not actually required, but when you’ve lots of branches with long names, it means you can tab-complete the branch name.
[3] Of course, if one of your branches changes any virtualenv requirements or similar, you’ll need to re-bootstrap, but that isn’t so often in practise. Or sometimes a new version of a dependency is released and you won’t get it until you rebootstrap.

As a small learning project for Golang, I set out the other day to experiment writing a simple form field validation library in my spare time – as it seems there is not yet anything along the lines of Django’s form API (email thread on go-nuts).

The purpose was to provide an API for creating forms that can validate http.Request.Form data, cleaning the data when it is valid, and collecting errors when it is not.

The initial version provides just CharField, IntegerField and RegexField, allowing form creation like:

    egForm := forms.NewForm(
        forms.NewCharField("description"),
        forms.NewIntegerField("purchase_count"))         

    egForm.SetFormData(request.Form)
    if egForm.IsValid() {
        // process the now populated egForm.CleanedData() and 
        // redirect.
    } else {
        // Use the now populated egForm.Errors map to report
        // the validation errors.
    }

The GoForms package is installable with `goinstall launchpad.net/goforms` or you can browse the goforms code on Launchpad (forms_test.go and fields_test.go have examples of the cleaned data and error). Let me know if you see flaws in the direction, or better ways of doing this in Go.

As a learning project it has been great – I’ve been able to use GoCheck for the tests, use embedded types (for sharing BaseField functionality – similar to composition+delegation without the bookkeeping) and start getting a feel for some of the subtleties of working with interfaces and other types in Go (this felt like all the benefits of z3c interfaces, without any of the overhead). Next I hope to include a small widget library for rendering basic forms/fields.

Getting setup on my Ubuntu Maverick VM was pretty straight forward:

• Add Gustavo’s software channel:

sudo add-apt-repository ppa:niemeyer/ppa
sudo apt-get update
sudo apt-get install golang
export GOROOT=/usr/lib/go

On a related note, I’ve wondered of late whether I could use “Agreed” rather than “Approved” when reviewing a branch. Agreed (IMO) implies one type of relationship – peer discussion resulting in agreement – rather than an approval from an authority (perhaps without interaction). It is a small and maybe insignificant distinction…