Something-driven development

Software development thoughts around Ubuntu, Python, Golang and other tools

Archive for the ‘golang’ Category

A tour of Go – the web crawler exercise

with 5 comments

The last exercise in the Go Tour – parallelizing a web crawler – turned out to be quite a bit more interesting than I’d expected. If anyone has suggested improvements from which I can learn a bit more, or their own solutions posted, let me know – my exercise solution is on github. I’ve tried to stick to the tour content (ie. only using channels rather than the sync package for accessing shared data).

Spoiler Alert: If you are learning Golang and haven’t yet worked through the Go-Tour, go and do so now. If you get stuck, keep struggling, take a break, try again in a few days etc., before looking at other peoples’ solutions.

The solution I ended up with has a Crawl() function very similar to the original, just with two extra function parameters:

func Crawl(url string, depth int, fetcher Fetcher,
	startCrawl func(string) bool, crawlComplete chan string) {

	if depth <= 0 {
		crawlComplete <- url

	body, urls, err := fetcher.Fetch(url)
	if err != nil {
		crawlComplete <- url

	fmt.Printf("found: %s %q\n", url, body)
	for _, u := range urls {
		if startCrawl(u) {
			go Crawl(u, depth-1, fetcher, startCrawl, crawlComplete)
	crawlComplete <- url

The two parameters are:

  • startCrawl func(url string) bool – used as a check before spawning a new ‘go Crawl(url)’ to ensure that we don’t crawl the same url twice.
  • crawlComplete chan string – used to signal that the Crawl function has fetched the page and finished spawning any child go-routines.

These two resources are created and passed in to the initial Crawl() call in the main() function:

func main() {
	startCrawl := make(chan StartCrawlData)
	crawlComplete := make(chan string)
	quitNow := make(chan bool)
	go processCrawls(startCrawl, crawlComplete, quitNow)

	// Returns whether a crawl should be started for a given
	// URL.
	startCrawlFn := func(url string) bool {
		resultChan := make(chan bool)
		startCrawl <- StartCrawlData{url, resultChan}
		return <-resultChan

	Crawl("", 4, fetcher, startCrawlFn,


Access to the shared state of which urls have been crawled and when all Crawls() have finished etc., is managed via those channels in the processCrawls() go-routine, so that the main() can simply call the first Crawl() and then wait to quit. I want to check how cheap the temporary creation of a channel is (for the return value of the startCrawlFn above) – I think I saw this method on an earlier GoLang tutorial example, but otherwise I’m happy with the solution :-).

Other solutions to learn from:


Written by Michael

November 14, 2012 at 12:32 pm

Posted in golang

GoForms – validating form data in Golang

leave a comment »

Learning a new language is fun…finding new ways of thinking about old problems and simple ways of expressing new ideas.

As a small learning project for Golang, I set out the other day to experiment writing a simple form field validation library in my spare time – as it seems there is not yet anything along the lines of Django’s form API (email thread on go-nuts).

The purpose was to provide an API for creating forms that can validate http.Request.Form data, cleaning the data when it is valid, and collecting errors when it is not.

The initial version provides just CharField, IntegerField and RegexField, allowing form creation like:

    egForm := forms.NewForm(

    if egForm.IsValid() {
        // process the now populated egForm.CleanedData() and 
        // redirect.
    } else {
        // Use the now populated egForm.Errors map to report
        // the validation errors.

The GoForms package is installable with `goinstall` or you can browse the goforms code on Launchpad (forms_test.go and fields_test.go have examples of the cleaned data and error). Update: The GoForms package is installable with `go get`, or you can browse the code. Let me know if you see flaws in the direction, or better ways of doing this in Go.

As a learning project it has been great – I’ve been able to use GoCheck for the tests, use embedded types (for sharing BaseField functionality – similar to composition+delegation without the bookkeeping) and start getting a feel for some of the subtleties of working with interfaces and other types in Go (this felt like all the benefits of z3c interfaces, without any of the overhead). Next I hope to include a small widget library for rendering basic forms/fields.

Written by Michael

May 18, 2011 at 9:03 am

Posted in golang, launchpad, testing

Getting going with Go (golang)

leave a comment »

For quite a while I’ve been wanting to get stuck into learning the Go programming language, after a colleague at Canonical (Gustavo Niemeyer) wrote enthusiastically about it… and today I took a bit of time to get started.

Getting setup on my Ubuntu Maverick VM was pretty straight forward:

  • Add Gustavo’s software channel:
sudo add-apt-repository ppa:niemeyer/ppa
sudo apt-get update
sudo apt-get install golang
export GOROOT=/usr/lib/go
  • This also adds ‘gorun‘, which effectively saves you having to compile and link each time you edit your go source file – nice
  • I had to then grab the go source to get the vim syntax highlighting etc. (files are provided for most editors in the source).
Whenever learning a new language I always find it incredibly helpful to be able to see all the options available – code auto-completion. So finally, I installed gocode, but had to make the following changes to get it to `make install`:
  • Installed golang-weekly (rather than golang) to get a more recent version…
  • Reverted the latest commit to gocode which was an update for the latest golang weekly release, which hadn’t yet hit the PPA.
With those changes, gocode installed and worked beautifully – learning with auto-completion is like seeing with peripheral vision for the first time.
Next week I hope to checkout the gocheck testing library as well as the Effective Go tutorial. As much as I love Python, it’s great getting stuck into a new language!

Written by Michael

April 15, 2011 at 6:18 pm

Posted in golang