Something-driven development

Software development thoughts around Ubuntu, Python, Golang and other tools

Develop and deploy with virtualenv

with 6 comments

What is the dream setup for developing and deploying Django apps? I’m looking for a solution that I can use consistently to deploy apps to servers where I may or may not have the ability to install system packages, or where I might need my app temporarily to use a newer version of a system-installed package while giving other apps running on the same server breathing space to update (think: updating a system-installed Django package on a server running four independent apps).

Specifically, the goals I have for this scenario are:

  • It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment),
  • Updating any virtualenv environment should be automatic, but transparent (ie. if the pip requirements.txt changes, the next time I run tests, devserver or deployed server, it’ll automatically ensure the virtualenv is correct),
  • I shouldn’t have to wait unnecessarily for virtualenvs to be created (ie. if I make a change to the requirements to try a new version of a package, and then change it back, I don’t want to re-create the original virtualenv). Similarly, if I revert a deployment to a previous version, the previous virtualenv should still be available.
  •  For deployment, the virtualenv shouldn’t unnecessarily replace system python packages, but allow this as an option (ie. not a –no-site-packages virtualenv).

There are a lot of virtualenv/fabric posts out there for both development and deployment, and using a SHA of the requirements.txt seems an obvious way to go. What I ended up with for my project was this develop and deploy with virtualenv snippet which so far is working quite well (although I’m yet to try a deploy where I override system packages). If the deployed version is using virtualenv exclusively, the requirements.txt file can be shared, but otherwise it would just be a matter of including the requirements.txt for the deploy with the other configuration data ( etc.).

If you can see any reasons why this is not a good idea, or improvements, please let me know!


Written by Michael

June 21, 2011 at 12:57 pm

Posted in django, python

6 Responses

Subscribe to comments with RSS.

  1. Looks really intereting. Internally at my company we are building something that kind of takes care of the overall problem. And we choose the same tools. I like your approach of hashing the requirements, will definitely take a look. Thanks for sharing.

    Jorge Vargas

    June 21, 2011 at 3:42 pm

    • Thanks Jorge. Out of interest, what do you see as the overall problem? Something different to the goals I outlined above?


      June 22, 2011 at 9:03 am

  2. Well, we’ve developed such thing for own projects and now preparing it for public use.

    Our goals were basically the same.

    Subscribe at if interested


    June 22, 2011 at 7:23 pm

  3. You might want to have a look at buildout + djangorecipe. Buildout is a build tool comparable to javas ant.
    Combined with virtualenv and fabric you get a quite powerful build and deploy tool set.


    June 23, 2011 at 10:34 am

    • Hi Christian! I was previously using buildout (both with and without djangorecipe) for django projects – and mostly it was great, but unfortunately when something went wrong, it was quite difficult to debug :/ (maybe I should have tried harder, but I usually depended on inside expertise from Gary Poster).

      I think it’s mainly the first goal above which IMO wasn’t satisfied by buildout:

      It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment)

      Debugging issues like broken development environments after a new buildout release were hard, and I think virtualenv sans buildout is much more transparent to developers (most of whom are pretty familiar with virtualenv, if not fabric). There were other small issues too that hindered the other goals above (that I didn’t end up resolving and possibly could have been resolved with more time), but I think basically for me it comes down to developer familiarity and transparency of the tool (if something goes wrong, an unfamiliar dev should be able to debug the environment problem).

      One thing I do miss about buildout was the way I could specify dependencies in *either* the (for run-time) or the buildout.cfg (for development environment only), and buildout would just do the right thing 🙂


      June 23, 2011 at 11:04 am

  4. […] to install system packages, or where I might need my app temporarily to use a newer version of a… Read more… Categories: Python     Share | Related […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: