Something-driven development

Software development thoughts around Ubuntu, Python, Golang and other tools

Archive for the ‘Uncategorized’ Category

Using Apache2’s mod_proxy to transition traffic

with 3 comments

I was recently in the situation of wanting to transition traffic gradually from an old deployment to a new deployment. It’s a large production system, so rather than just switching the DNS entries to point at the new deployment, I wanted to be able to shift the traffic over in a couple of controlled steps.

It turns out, Apache’s mod_proxy makes this relatively straight forward. You can choose which resource for which you want to move traffic, and easily update the proportion of traffic for that resource which should go through to the new env. Might be old news to some, but not having needed this before, I was quite impressed by Apache2’s configurability:

# Pass any requests for specific-url through to the balancer (defined below)
# to transition traffic from the old to new system.
ProxyPass /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/
ProxyPassReverse /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/

# Send all other requests straight to the backend for the old system.
ProxyPass /myapp/ http://old.backend.ip:1234/myapp/
ProxyPassReverse /myapp/ http://backend.ip:1234/myapp/

# Send 50% of the traffic to the old backend, and divide the rest between the
# two new frontends.
<Proxy balancer://transition-traffic>
    BalancerMember http://old.backend.ip:1234 timeout=60 loadfactor=2
    BalancerMember http://new.frontend1.ip:80 timeout=60 loadfactor=1
    BalancerMember http://new.frontend2.ip:80 timeout=60 loadfactor=1
    ProxySet lbmethod=byrequests

Once the stats verify that the new env isn’t hitting any firewall or load issue, the loadfactor can be updated (only need to graceful apache) to ramp up traffic so that everything is hitting the new env. Of course, it adds one extra hop for serving requests, but it’s then much safer to switch the DNS entries when you *know* your new system is already handling the production traffic.


Written by Michael

July 17, 2015 at 6:37 am

Posted in Uncategorized

Creating a Mir Snap in the cloud

leave a comment »

I tend to do all my work in the cloud (and work physically from a useful but comparatively powerless chromebook running Ubuntu). So creating a Mir Snap on my local machine wasn’t an option I was keen to try.

Mainly for my own documentation, the first attempt at creating a Mir snap (a display server for experimenting with Kodi on Ubuntu Snappy), went like this:

First spin up an Ubuntu Wily development instance in a(ny) openstack cloud (I’m using a Canonical internal one):

$ nova boot --flavor cpu2-ram4-disk100-ephemeral20 --image 1461273c-af73-4839-9f64-3df00446322a --key-name ******** wily_dev

Less than a minute later, ssh in and use the existing snappy packaging branch for mir:

$ sudo add-apt-repository ppa:snappy-dev/tools
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install snappy-tools bzr cmake gcc-multilib ubuntu-snappy-cli mir-demos mir-graphics-drivers-desktop dpkg-dev

$ bzr branch lp:~mir-team/mir/snappy-packaging && cd snappy-packaging && make

The last step uses deb2snap to create the snap package from the installed deb packages. So, the (76M) snap is for the amd64 architecture – as the instance I created is amd64:

$ ls -lh mir_snap1_amd64.snap
-rw-rw-r-- 1 ubuntu ubuntu 76M Jun 21 11:20 mir_snap1_amd64.snap

Next up… using either QEMU or an ARM cloud instance (I’m not sure that the latter is available) to create an ARM Mir snap for my Raspberry Pi, and testing it out…

Written by Michael

June 21, 2015 at 1:25 pm

Posted in Uncategorized

A Kodi (XBMC) snap for the RaspberryPi?

leave a comment »

orange-matchboxI just received an Orange Matchbox today – a Raspberry Pi 2B with a PiGlow and Ubuntu-engraved casing – to try out Snappy Ubuntu.

As a first project, I want to see if it’s possible to get Kodi (formally XBMC) packaged as a snap. I’m not yet sure if this is possible, but the plan is to install Ubuntu on a separate microSD and setup deb2snap and see whether I can create a snap (minus certain dependencies). Obviously then install the snap on my snappy microSD and profit! (The devel’s in the details of course.)

I’m not sure that deb2snap will work with Ubuntu Trusty (the default image for RPi2), as it was recently announced that XMir will no longer be supported for anything less than Wily. We’ll see how it goes.

I already found and reported my first issue while testing snappy on a VM last week, which got fixed for todays stable release. Let the fun begin…

Written by Michael

June 11, 2015 at 9:14 am

Posted in Uncategorized

Deploying swift with juju for development

with 2 comments

Just documenting for later (and for a friend and colleague who needs it now) – my notes for setting up openstack swift using juju. I need to go back and check whether keystone is required – I initially had issue with the test auth so switched to keystone.

First, create the config file to use keystone, local block-devices on the swift storage units (ie. no need to mount storage), and using openstack havana:

cat >swift.cfg <<END
    zone-assignment: auto
    replicas: 3
    auth-type: keystone
    openstack-origin: cloud:precise-havana/updates
    zone: 1
    block-device: /etc/swift/storagedev1.img|2G
    openstack-origin: cloud:precise-havana/updates
    admin-token: somebigtoken
    openstack-origin: cloud:precise-havana/updates

Deploy it (this could probably be replaced with a charm bundle?):

juju deploy --config=swift.cfg swift-proxy
juju deploy --config=swift.cfg --num-units 3 swift-storage
juju add-relation swift-proxy swift-storage
juju deploy --config=swift.cfg keystone
juju add-relation swift-proxy keystone

Once everything is up and running, create a tenant and user with the user having admin rights for the tenant (using your keystone unit’s IP address for keystone-ip). Note, below I’m using the names of tenant, user and role – which works with keystone 0.3.2, but apparently earlier versions require you to use the uuids instead. Check with `keystone help user-role-add`).

$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken tenant-create --name mytenant
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-create --name myuser --tenant mytenant --pass userpassword
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-role-add --tenant mytenant --user myuser --role Admin

And finally, use our new admin user to create a container for use in our dev environment (specify auth version 2):

$ export OS_REGION_NAME=RegionOne
$ export OS_TENANT_NAME=mytenant
$ export OS_USERNAME=myuser
$ export OS_PASSWORD=userpassword
$ export OS_AUTH_URL=http://keystone-ip:5000/v2.0/
$ swift -V 2 post mycontainer

If you want the container to be readable without auth:

$ swift -V 2 post mycontainer -r '.r:*'

If you want another keystone user to have write access:

$ swift -V 2 post mycontainer -w mytenant:otheruser

Verify that the container is ready for use:
$ swift -V 2 stat mycontainer

Please let me know if you spot any issues (these notes are from a month or two ago, so I haven’t just tried this).

Written by Michael

January 9, 2014 at 10:54 am

Posted in Uncategorized

Tagged with , ,

Juju + Ansible = simpler charms

with 7 comments

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
    default_hooks=['start', 'stop', 'config-changed',

def install():

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed 🙂

Written by Michael

November 8, 2013 at 11:42 am

Posted in Uncategorized

Tagged with ,

Kanban for kids

with 8 comments

Facing the arrival of another child, I set out a few months ago to experiment with ways to help each other get things done together as a family without becoming a task-master of a father. A very simplified Kanban board seemed like a good fit for our kitchen wall, as it visualises what needs doing or other fun things that are planned, helps us to help each other when needed, and enables the kids to take part in the planning and organising of the day.

The photo of our board above is what we’ve ended up with after a few months. It is really a habit tracker – with most of the items on the board being things that we do daily, every few days, weekly or fortnightly. Some points that we’ve found helpful:

  • Items specific to the kids are colour coded (ie. we have a “Put clothes away” item for each child in their colour).
  • We have some incentives – each item has a number of associated “service points” [1]. For eg. 5 for putting away clothes, 15 for taking the dog out for a walk etc. (agreed with the kids and adjusted on feedback).
  • We have incentives to have fun doing items together – if you complete an item together with someone else, you get 50% extra service points, simply to encourage having fun doing things together (we put a magnet on the item to signify this).
  • We very rarely tell the kids to do something – in extreme circumstances I’ve told my daughter I need her to take the dog out now as I can’t do it myself, but it’s something we generally avoid, in fact…
  • The kids generally choose what item they want to do next and when they want to do it (ie. “You’d like to help daddy set the table for lunch? Shall we do it now or after playing Lego for a while?”).
  • Each item has a drawing and is laminated with a small magnet, so that we can stack certain items together (for eg, we have a stack of cards related to the dog which are done in order each day – dog breakfast, walk 7am-ish, Walk 11am-ish, etc).
  • We start each day by pulling things that we want to do into the backlog (in addition to the daily stuff). Some items are weekly, which we store in a stack for each day on the top-left. Other items are ad-hoc which we pull in from the mess on the bottom left.
  • We limit the number of cards in the TODO and Doing lanes to 4, to help us help each flow (or review whether we really want to do something).
  • We don’t put normal stuff like playing lego or dressups there – the kids can play whenever they want without feeling they need to have an item on the board 😛

So far it’s been a great help, but as always, a continual learning process. Each child is motivated by different things, and yet the end goal is that we’d all enjoy helping each other without extrinsic motivation. Interestingly, one of the most freeing things has been for the kids (and my wife) to get used to the fact that it’s OK to say, no, I don’t feel like doing that today, I’m just going to move the card over to Thursday for the moment. That is, the board is something we control – a helpful way to organise and communicate – it does not tell us what we have to do.

[1] Service points – The kids can use the service points at the end of the day in ways such as staying up a bit later (1minute per point, within reason), or going out on their own with mum or dad for an ice-cream desert (50 points). We haven’t yet found a good solution for left over points, and are currently converting to Euro-cents as pocket money at the end of the week which works well in terms of value, but I don’t like the direct connection of being paid money to do stuff around the house. Another option I’m trying at the moment is using saved service points as bargaining power for larger items that the kids need (like a new school bag). I’d be glad to hear of other ideas.

Written by Michael

October 3, 2012 at 11:08 am

Posted in Uncategorized

More YUI and html5 fun

with 2 comments

Continuing on my YUI MVC experiment, I got to spend a bit of time last week playing with my prototype for Open Goal Tracker, and was blown away by the simplicity of two new html attributes: x-webkit-speech (just ‘speech’ when standardised) and contenteditable. Here’s a 2min video showing how I might use both of them in practise (in dire need of styling as usual):



Behind the scenes I’m still loving Yahoo User Interfaces application framework together with the unit-test api, which allow me to create tests like “creating the view should render stats for existing goals”.

Next week I hope to start implementing the server-sync and investigating integration with Ubuntu’s notification system. Once I’ve sorted out the application-cache and build/packaging etc. for both Ubuntu and Android, I want to do a initial dummy release and then start iterating the features that will make Open Goal Tracker actually useful for facilitating social-yet-individual learning, both within and without of classrooms.

You can play with the little open-goal-tracker prototype (I’ve only tried webkit so far) or run the tests yourself.

Written by Michael

November 29, 2011 at 11:16 am

Posted in Uncategorized