Using Apache2’s mod_proxy to transition traffic

I was recently in the situation of wanting to transition traffic gradually from an old deployment to a new deployment. It’s a large production system, so rather than just switching the DNS entries to point at the new deployment, I wanted to be able to shift the traffic over in a couple of controlled steps.

It turns out, Apache’s mod_proxy makes this relatively straight forward. You can choose which resource for which you want to move traffic, and easily update the proportion of traffic for that resource which should go through to the new env. Might be old news to some, but not having needed this before, I was quite impressed by Apache2’s configurability:

# Pass any requests for specific-url through to the balancer (defined below)
# to transition traffic from the old to new system.
ProxyPass /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/
ProxyPassReverse /myapp/specific-url/ balancer://transition-traffic/myapp/specific-url/

# Send all other requests straight to the backend for the old system.
ProxyPass /myapp/ http://old.backend.ip:1234/myapp/
ProxyPassReverse /myapp/ http://backend.ip:1234/myapp/

# Send 50% of the traffic to the old backend, and divide the rest between the
# two new frontends.
<Proxy balancer://transition-traffic>
    BalancerMember http://old.backend.ip:1234 timeout=60 loadfactor=2
    BalancerMember http://new.frontend1.ip:80 timeout=60 loadfactor=1
    BalancerMember http://new.frontend2.ip:80 timeout=60 loadfactor=1
    ProxySet lbmethod=byrequests
</Proxy>

Once the stats verify that the new env isn’t hitting any firewall or load issue, the loadfactor can be updated (only need to graceful apache) to ramp up traffic so that everything is hitting the new env. Of course, it adds one extra hop for serving requests, but it’s then much safer to switch the DNS entries when you *know* your new system is already handling the production traffic.

Creating a Mir Snap in the cloud

I tend to do all my work in the cloud (and work physically from a useful but comparatively powerless chromebook running Ubuntu). So creating a Mir Snap on my local machine wasn’t an option I was keen to try.

Mainly for my own documentation, the first attempt at creating a Mir snap (a display server for experimenting with Kodi on Ubuntu Snappy), went like this:

First spin up an Ubuntu Wily development instance in a(ny) openstack cloud (I’m using a Canonical internal one):

$ nova boot --flavor cpu2-ram4-disk100-ephemeral20 --image 1461273c-af73-4839-9f64-3df00446322a --key-name ******** wily_dev


Less than a minute later, ssh in and use the existing snappy packaging branch for mir:


$ sudo add-apt-repository ppa:snappy-dev/tools
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install snappy-tools bzr cmake gcc-multilib ubuntu-snappy-cli mir-demos mir-graphics-drivers-desktop dpkg-dev

$ bzr branch lp:~mir-team/mir/snappy-packaging && cd snappy-packaging && make


The last step uses deb2snap to create the snap package from the installed deb packages. So, the (76M) snap is for the amd64 architecture – as the instance I created is amd64:


$ ls -lh mir_snap1_amd64.snap
-rw-rw-r-- 1 ubuntu ubuntu 76M Jun 21 11:20 mir_snap1_amd64.snap


Next up… using either QEMU or an ARM cloud instance (I’m not sure that the latter is available) to create an ARM Mir snap for my Raspberry Pi, and testing it out…

A Kodi (XBMC) snap for the RaspberryPi?

I just received an Orange Matchbox today – a Raspberry Pi 2B with a PiGlow and Ubuntu-engraved casing – to try out Snappy Ubuntu.

As a first project, I want to see if it’s possible to get Kodi (formally XBMC) packaged as a snap. I’m not yet sure if this is possible, but the plan is to install Ubuntu on a separate microSD and setup deb2snap and see whether I can create a snap (minus certain dependencies). Obviously then install the snap on my snappy microSD and profit! (The devel’s in the details of course.)

I’m not sure that deb2snap will work with Ubuntu Trusty (the default image for RPi2), as it was recently announced that XMir will no longer be supported for anything less than Wily. We’ll see how it goes.

I already found and reported my first issue while testing snappy on a VM last week, which got fixed for todays stable release. Let the fun begin…

Deploying swift with juju for development

Just documenting for later (and for a friend and colleague who needs it now) – my notes for setting up openstack swift using juju. I need to go back and check whether keystone is required – I initially had issue with the test auth so switched to keystone.

First, create the config file to use keystone, local block-devices on the swift storage units (ie. no need to mount storage), and using openstack havana:

cat >swift.cfg <<END
swift-proxy:
    zone-assignment: auto
    replicas: 3
    auth-type: keystone
    openstack-origin: cloud:precise-havana/updates
swift-storage:
    zone: 1
    block-device: /etc/swift/storagedev1.img|2G
    openstack-origin: cloud:precise-havana/updates
keystone:
    admin-token: somebigtoken
    openstack-origin: cloud:precise-havana/updates
END

Deploy it (this could probably be replaced with a charm bundle?):

juju deploy --config=swift.cfg swift-proxy
juju deploy --config=swift.cfg --num-units 3 swift-storage
juju add-relation swift-proxy swift-storage
juju deploy --config=swift.cfg keystone
juju add-relation swift-proxy keystone

Once everything is up and running, create a tenant and user with the user having admin rights for the tenant (using your keystone unit’s IP address for keystone-ip). Note, below I’m using the names of tenant, user and role – which works with keystone 0.3.2, but apparently earlier versions require you to use the uuids instead. Check with `keystone help user-role-add`).

$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken tenant-create --name mytenant
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-create --name myuser --tenant mytenant --pass userpassword
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-role-add --tenant mytenant --user myuser --role Admin


And finally, use our new admin user to create a container for use in our dev environment (specify auth version 2):

$ export OS_REGION_NAME=RegionOne
$ export OS_TENANT_NAME=mytenant
$ export OS_USERNAME=myuser
$ export OS_PASSWORD=userpassword
$ export OS_AUTH_URL=http://keystone-ip:5000/v2.0/
$ swift -V 2 post mycontainer


If you want the container to be readable without auth:

$ swift -V 2 post mycontainer -r '.r:*'

If you want another keystone user to have write access:

$ swift -V 2 post mycontainer -w mytenant:otheruser

Verify that the container is ready for use:
$ swift -V 2 stat mycontainer

Please let me know if you spot any issues (these notes are from a month or two ago, so I haven’t just tried this).

Juju + Ansible = simpler charms

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s hooks.py to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
    playbook_path='playbooks/site.yaml',
    default_hooks=['start', 'stop', 'config-changed',
                   'solr-relation-changed'])

@hooks.hook()
def install():
    charmhelpers.contrib.ansible.install_ansible_support(from_ppa=True)

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed 🙂

Kanban for kids

Facing the arrival of another child, I set out a few months ago to experiment with ways to help each other get things done together as a family without becoming a task-master of a father. A very simplified Kanban board seemed like a good fit for our kitchen wall, as it visualises what needs doing or other fun things that are planned, helps us to help each other when needed, and enables the kids to take part in the planning and organising of the day.

The photo of our board above is what we’ve ended up with after a few months. It is really a habit tracker – with most of the items on the board being things that we do daily, every few days, weekly or fortnightly. Some points that we’ve found helpful:

• Items specific to the kids are colour coded (ie. we have a “Put clothes away” item for each child in their colour).
• We have some incentives – each item has a number of associated “service points” [1]. For eg. 5 for putting away clothes, 15 for taking the dog out for a walk etc. (agreed with the kids and adjusted on feedback).
• We have incentives to have fun doing items together – if you complete an item together with someone else, you get 50% extra service points, simply to encourage having fun doing things together (we put a magnet on the item to signify this).
• We very rarely tell the kids to do something – in extreme circumstances I’ve told my daughter I need her to take the dog out now as I can’t do it myself, but it’s something we generally avoid, in fact…
• The kids generally choose what item they want to do next and when they want to do it (ie. “You’d like to help daddy set the table for lunch? Shall we do it now or after playing Lego for a while?”).
• Each item has a drawing and is laminated with a small magnet, so that we can stack certain items together (for eg, we have a stack of cards related to the dog which are done in order each day – dog breakfast, walk 7am-ish, Walk 11am-ish, etc).
• We start each day by pulling things that we want to do into the backlog (in addition to the daily stuff). Some items are weekly, which we store in a stack for each day on the top-left. Other items are ad-hoc which we pull in from the mess on the bottom left.
• We limit the number of cards in the TODO and Doing lanes to 4, to help us help each flow (or review whether we really want to do something).
• We don’t put normal stuff like playing lego or dressups there – the kids can play whenever they want without feeling they need to have an item on the board 😛

So far it’s been a great help, but as always, a continual learning process. Each child is motivated by different things, and yet the end goal is that we’d all enjoy helping each other without extrinsic motivation. Interestingly, one of the most freeing things has been for the kids (and my wife) to get used to the fact that it’s OK to say, no, I don’t feel like doing that today, I’m just going to move the card over to Thursday for the moment. That is, the board is something we control – a helpful way to organise and communicate – it does not tell us what we have to do.

[1] Service points – The kids can use the service points at the end of the day in ways such as staying up a bit later (1minute per point, within reason), or going out on their own with mum or dad for an ice-cream desert (50 points). We haven’t yet found a good solution for left over points, and are currently converting to Euro-cents as pocket money at the end of the week which works well in terms of value, but I don’t like the direct connection of being paid money to do stuff around the house. Another option I’m trying at the moment is using saved service points as bargaining power for larger items that the kids need (like a new school bag). I’d be glad to hear of other ideas.

More YUI and html5 fun

Continuing on my YUI MVC experiment, I got to spend a bit of time last week playing with my prototype for Open Goal Tracker, and was blown away by the simplicity of two new html attributes: x-webkit-speech (just ‘speech’ when standardised) and contenteditable. Here’s a 2min video showing how I might use both of them in practise (in dire need of styling as usual):

 

 

Behind the scenes I’m still loving Yahoo User Interfaces application framework together with the unit-test api, which allow me to create tests like “creating the view should render stats for existing goals”.

Next week I hope to start implementing the server-sync and investigating integration with Ubuntu’s notification system. Once I’ve sorted out the application-cache and build/packaging etc. for both Ubuntu and Android, I want to do a initial dummy release and then start iterating the features that will make Open Goal Tracker actually useful for facilitating social-yet-individual learning, both within and without of classrooms.

You can play with the little open-goal-tracker prototype (I’ve only tried webkit so far) or run the tests yourself.

Working in the cloud

For a while now I’ve done all my development work and irc communication via SSH and byobu – I love the freedom of just grabbing my netbook (ie. switching computers) and heading to a cafe and picking up right where I left off (or even letting tests run while I walk).

The one pain-point was that I SSH’d into my own box at home – and if my home connection went (anything from a thunderstorm to a curious kid pulling a chord), I had to then work on my much slower netbook.

Now, after reading on an email thread that Aaron was doing something similar, I’m running a tiny cloud instance for my irc and development work, and documenting everything that I’m doing to get my dev environment just right – so that I can automate the setup (not really something for an ensemble formula – just a simple script).

That, and having all my music in the cloud for the past 6 months make switching computers and/or physical locations a non-obtrusive part of my work-day.

If anyone has small tips that make their own dev environment enjoyable, let me know!

Code-reviews as relationship builders

I really enjoyed the emphasis of Ben’s Code-reviews as relationship builders post over the weekend. The focus on healthy learning relationships both for new and old is great – something I’ve tried to focus on myself over the past few years working in distributed teams.

On a related note, I’ve wondered of late whether I could use “Agreed” rather than “Approved” when reviewing a branch. Agreed (IMO) implies one type of relationship – peer discussion resulting in agreement – rather than an approval from an authority (perhaps without interaction). It is a small and maybe insignificant distinction…

Experimenting with daily stand-ups

My direct team-mates at Canonical live everywhere from the US west-coast through to Poland, so we don’t have the option of doing a ‘traditional’ stand-up each day.

In the past we’ve done IRC standups – each pasting our notes at one point in the day when we’re all online – which helps a few points (recording what we’re up to) but has quite a few drawbacks in my opinion (see Vincent’s thoughts on irc standups also). A lot of us on the team admitted that we didn’t always give the irc standups our full-attention.

Right now we’re trying an experiment which is kind-of neat. We’re doing voice standups in pairs/triplets with people in similar timezones, but adding our standup notes to a wiki page before the call. The advantages are that the calls are easy – I can read through Lukasz’s notes and we just discuss any issues or successes as well as the plan for today, and vice-versa. Also, as we all subscribe to the wiki page for email notifications, we actually read everyone else’s notes from different time-zones asynchronously (and all stakeholders can take part in calls or read the notes as desired).

Personally, having a brief voice call as part of my day is a huge benefit – just touching base with another voice from work, and having a bit of a chat after going through our stand-up gives a more human touch to working from home. It’ll be interesting to see whether others on the team find it more or less helpful.