A tour of Go – the web crawler exercise

The last exercise in the Go Tour – parallelizing a web crawler – turned out to be quite a bit more interesting than I’d expected. If anyone has suggested improvements from which I can learn a bit more, or their own solutions posted, let me know – my exercise solution is on github. I’ve tried to stick to the tour content (ie. only using channels rather than the sync package for accessing shared data).

Spoiler Alert: If you are learning Golang and haven’t yet worked through the Go-Tour, go and do so now. If you get stuck, keep struggling, take a break, try again in a few days etc., before looking at other peoples’ solutions.

The solution I ended up with has a Crawl() function very similar to the original, just with two extra function parameters:

func Crawl(url string, depth int, fetcher Fetcher,
	startCrawl func(string) bool, crawlComplete chan string) {

	if depth <= 0 {
		crawlComplete <- url
 		return
	}

	body, urls, err := fetcher.Fetch(url)
	if err != nil {
		fmt.Println(err)
		crawlComplete <- url
 		return
	}

	fmt.Printf("found: %s %q\n", url, body)
	for _, u := range urls {
		if startCrawl(u) {
			go Crawl(u, depth-1, fetcher, startCrawl, crawlComplete)
		}
	}
	crawlComplete <- url
}

The two parameters are:

• startCrawl func(url string) bool – used as a check before spawning a new ‘go Crawl(url)’ to ensure that we don’t crawl the same url twice.
• crawlComplete chan string – used to signal that the Crawl function has fetched the page and finished spawning any child go-routines.

These two resources are created and passed in to the initial Crawl() call in the main() function:

func main() {
	startCrawl := make(chan StartCrawlData)
	crawlComplete := make(chan string)
	quitNow := make(chan bool)
	go processCrawls(startCrawl, crawlComplete, quitNow)

	// Returns whether a crawl should be started for a given
	// URL.
	startCrawlFn := func(url string) bool {
		resultChan := make(chan bool)
		startCrawl <- StartCrawlData{url, resultChan}
		return <-resultChan
	}

	Crawl("http://golang.org/", 4, fetcher, startCrawlFn,
		crawlComplete)

	<-quitNow
}

Access to the shared state of which urls have been crawled and when all Crawls() have finished etc., is managed via those channels in the processCrawls() go-routine, so that the main() can simply call the first Crawl() and then wait to quit. I want to check how cheap the temporary creation of a channel is (for the return value of the startCrawlFn above) – I think I saw this method on an earlier GoLang tutorial example, but otherwise I’m happy with the solution :-).

Other solutions to learn from:

• Sonia Codes’ solution – sharing memory by communicating
• Kim Mason’s solution using sync package
• [I’ll add more as I find them, now that I’m finished my own solution :-)]

GoForms – validating form data in Golang

Learning a new language is fun…finding new ways of thinking about old problems and simple ways of expressing new ideas.

As a small learning project for Golang, I set out the other day to experiment writing a simple form field validation library in my spare time – as it seems there is not yet anything along the lines of Django’s form API (email thread on go-nuts).

The purpose was to provide an API for creating forms that can validate http.Request.Form data, cleaning the data when it is valid, and collecting errors when it is not.

The initial version provides just CharField, IntegerField and RegexField, allowing form creation like:

    egForm := forms.NewForm(
        forms.NewCharField("description"),
        forms.NewIntegerField("purchase_count"))         

    egForm.SetFormData(request.Form)
    if egForm.IsValid() {
        // process the now populated egForm.CleanedData() and 
        // redirect.
    } else {
        // Use the now populated egForm.Errors map to report
        // the validation errors.
    }

The GoForms package is installable with `goinstall launchpad.net/goforms` or you can browse the goforms code on Launchpad (forms_test.go and fields_test.go have examples of the cleaned data and error). Update: The GoForms package is installable with `go get github.com/absoludity/goforms/forms`, or you can browse the code. Let me know if you see flaws in the direction, or better ways of doing this in Go.

As a learning project it has been great – I’ve been able to use GoCheck for the tests, use embedded types (for sharing BaseField functionality – similar to composition+delegation without the bookkeeping) and start getting a feel for some of the subtleties of working with interfaces and other types in Go (this felt like all the benefits of z3c interfaces, without any of the overhead). Next I hope to include a small widget library for rendering basic forms/fields.

Getting going with Go (golang)

For quite a while I’ve been wanting to get stuck into learning the Go programming language, after a colleague at Canonical (Gustavo Niemeyer) wrote enthusiastically about it… and today I took a bit of time to get started.

Getting setup on my Ubuntu Maverick VM was pretty straight forward:

• Add Gustavo’s software channel:

sudo add-apt-repository ppa:niemeyer/ppa
sudo apt-get update
sudo apt-get install golang
export GOROOT=/usr/lib/go

• This also adds ‘gorun‘, which effectively saves you having to compile and link each time you edit your go source file – nice
• I had to then grab the go source to get the vim syntax highlighting etc. (files are provided for most editors in the source).

Whenever learning a new language I always find it incredibly helpful to be able to see all the options available – code auto-completion. So finally, I installed gocode, but had to make the following changes to get it to `make install`:

• Installed golang-weekly (rather than golang) to get a more recent version…
• Reverted the latest commit to gocode which was an update for the latest golang weekly release, which hadn’t yet hit the PPA.

With those changes, gocode installed and worked beautifully – learning with auto-completion is like seeing with peripheral vision for the first time.

Next week I hope to checkout the gocheck testing library as well as the Effective Go tutorial. As much as I love Python, it’s great getting stuck into a new language!