Go, Docker and a highly concurrent HTTP requester

Introduction

In this post I aim to introduce you to the Go programming language. Specifically I'll be covering the basics of how to get your environment set-up; but I'm not teaching the language syntax (if you want that there are many online resources to help you), although I will be explaining some example code.

I'll also run you through the open-source project "Go Requester" which is a highly concurrent component aggregator service (yeah, it sounds more fancy than it actually is).

So let's get stuck in...

Why Go?

My team has only recently gravitated towards exploring the Go programming language and eco-system of tools. It feels like it has been a natural progression from the Ruby programming language we've primarily been working with over the past three years. There were issues with the Ruby language that we wished we could resolve through different means but in fact were native aspects of Go's own design, which helped to peak our interest in the language.

We decided the safest way to utilise Go was to pick the smallest possible upcoming service we needed to build and to make the conscious decision to use Go instead of Ruby (in this case it was a configuration API service; which has been designed as a small micro service).

The reason for this decision was because the entire team recognised that Go provides:

† including BBC Worldwide, Twitter, Facebook, Dropbox, Docker, Bitly, DataDog, Disque, GitHub, Gov.UK, Hailo, Mozilla, Rackspace, SoundCloud, Stack Exchange

The 'technical' considerations underlying the decision to use Go are completely sound and justifiable in our minds, and as a self-organising team it is fundamental to be able to evolve over time; this way of thinking helps us to facilitate working in an agile manner, which allows us to build the best services and tools possible.

Setting up Go

You're going to need Go installed (pretty obvious really). My laptop is running Mac OS X and so I'll cover two variations of how you can achieve this within the context of that operating system: the first being a manual install using the Homebrew package manager and the other being Docker; but your mileage may vary if you're on a different OS (e.g. Windows or Linux).

Note: I focus more on Docker, as I want to try and utilise containers more and have them act a clean gateway between my host environment and my development requirement. But as you'll see, it's not always smooth sailing

Homebrew

The first option is to install Go via the popular package manager "Homebrew", using the following command:

brew install go

Once installed you'll need to set-up your project environment. This requires a few things, the first being some environment variables and the second being a 'root' folder where all your Go projects will need to be built and run from.

The following is what you'll likely want to add to your own .bashrc or .zshrc files:

export GOPATH=$HOME/Projects/golang
export PATH=$HOME/Projects/golang/bin:$PATH

For the root folder, I decided to set it to $HOME/Projects/golang, which means once Go is installed you'll need to create that folder; and you'll also need to ensure it has the following sub folder structure:

├── bin
├── pkg
└── src
    ├── <domain>
    │   ├── <user>
    │   │   └── <project>

You'll notice the src directory has some additional structure to it (domain/user/project). You don't have to use that exact folder structure; but if you're deploying your code to GitHub (which I am), then you will do and I'll explain why in a moment. As an example, this is what my own folder structure looks like:

├── bin
├── pkg
└── src
    ├── github.com
    │   ├── integralist
    │   │   └── go-requester

So inside the src directory I've created my sub folder structure so that it maps to where the code for my project will reside online (e.g. https://github.com/integralist/go-requester). The reason this is important is because Go provides command-line tooling that allows you to pull packages and libraries from the internet. So if I want to build a Go program I need to ensure it has the same format as the online repository I'm using to store it.

Let's take a moment to describe what these folders will end up containing:

When you download and install Go packages you'll find that Go uses the $GOPATH value to determine where to install these packages. So you don't need to add anything yourself into the bin or pkg folders as that is handled automatically. But the folders do at least need to exist.

The specific Go command for downloading packages is:

go get <url>

So if you wanted to pull down my open-source project code, you would execute:

go get github.com/integralist/go-requester

In order to run a Go program you would need to mv into the relevant Go project directory and subsequently run:

go run <name_of_program.go>

Docker

Docker is somewhat simpler in that you don't need to install Go on your host machine, nor do you have to create any environment variables or even have to generate a messy nested set of folder directories. The basic premise is that you can download a Docker Golang image and mount your folder where your Go code resides into the running container and use that.

To start you'll need Docker Machine and VirtualBox installed (VirtualBox is only necessary if you're not working from a native Linux machine).

Note: I would actually recommend VMWare instead of VirtualBox; as VirtualBox's networking capabilities are a bit slow and (as we'll see later on) can interupt your workflow. The following assumes you're using VirtualBox as it is open-source/free

Once both are installed you can run the following commands in order to start Docker:

docker-machine create --driver virtualbox dev
docker-machine start dev

Note: you'll see from the command that I've named my VM (Virtual Machine) dev

Once Docker is started you can run the following command to download the latest Go Docker image:

docker pull golang

Now mv into wherever your Go project directory is and execute the following command to run a test Go program called test.go (source code is below, if you want to play along at home):

docker run --rm -v "$PWD":/app -w /app golang:latest go run test.go

Note: for those new to Docker, I'm 'mounting' my current folder as a 'volume' into the running Docker container using -v and I'm changing the working directory inside the container using -w

Here is the test.go example file I'm using with the above Docker run command:

package main

import "fmt"

func main() {
  fmt.Println("\nHello World")
}

What you should see is the message "Hello World" sent to stdout.

Now this is fine, but you might want/prefer to use the onbuild Dockerfile, which will allow you to modify how the image is run and also automates some of the process of installing dependencies.

For example, I don't have any dependencies defined in my simple test.go app and so using the docker run command above (or the following one using the onbuild Dockerfile) is OK and will work reasonably well.

But if you have dependencies (i.e. anything not part of the standard library) then you'll need to tell Go to download and install them before running your program. This can get complicated in itself and so later on we'll see a way of working around the dependency issue using an external tool called gb.

So below is our Dockerfile (this is a bit of a silly example because I'm not modifying the standard image in any way):

FROM golang:1.5-onbuild

What this image does is copy all the files/folders from your current directory into /go/src/app within the container.

You can then use this image like so:

docker build -t my-golang-app .
docker run --rm --name go-tester my-golang-app

Here we've built a new Docker image from the onbuild Go image and we then run that newly created image. Again, the output should be the message "Hello World" sent to stdout.

Godo

If you've ever written a program in Ruby then chances are good that you've at least heard of something called Guard. To quote the Guard website:

Guard is a command line tool to easily handle events on file system modifications

So you write a Guardfile (which is just Ruby) that says "I want to watch files x, y and z and when any of them change I want Guard to trigger this other thing to happen".

Godo is exactly the same but more powerful and written in Go.

To install Godo, you'll need to execute:

go get -u gopkg.in/godo.v1/cmd/godo

A very simple file I've been using looks like the following:

package main

import . "gopkg.in/godo.v1"

func tasks(p *Project) {
  p.Task("watch-server", func(c *Context) error {
    return Start(`main.go`)
  }).Watch("**/*.go")
}

func main() {
  Godo(tasks)
}

The above file creates a new 'task' called "watch-server", which watches for any changes to our application and restarts our application if a change is detected. In this example, I tell Godo to execute main.go which is the name of my applications main package entrypoint (you can name this anything you like though).

You'll also notice we pass an inline anonymous function (as Go supports first-class functions it can be made into a functional language, of sorts, if you really try); this anonymous function is the 'action' for whenever my event happens; and in this case my event is: whenever any of my go files change.

I define the event using the .Watch function and pass it a glob **/*.go, which tells it to recursively find all .go files. Now in order for this all to work, I need to open a new shell or terminal window and execute the following command:

godo watch-server --watch

From here it'll immediate execute the relevant task and then start watching my files.

A word about Docker...

Now, at this point it's important to realise something about the Go set-up we discussed earlier, which is: If you're using the Homebrew route of setting up Go then everything I explain here (and also about gb later on) will work fine. Things are a little different though if you're using Docker.

If you're using Docker you probably haven't set-up that same folder structure as we did when setting up Homebrew (that's because you didn't need to; everything was handled inside the Docker container). When setting stuff up with Docker, you'll find that you need to be pretty committed to it.

In other words, instead of just using Docker to run your Go program, you'll need to install and run Godo from within a running container as well. There are also issues around how tools such as gb work when run within a Docker container compared to on your host machine and potential conflicts with the container's own $GOPATH. I'll come back to all this later on, but it was important enough for me to stop and make mention of this now so you were at least aware of it.

To solve this issue, you could initially do everything from the command line without creating a Dockerfile, like the following code snippet demonstrates (you'll notice I've also broken the command out over multiple lines to make it easier to read):

Note: I tested this originally with an older version of my Go-Requester application (before I designed a quite complex Docker setup). So you'll notice that I'm mounting my volume into a specific directory structure within the container where - at the time - the $GOPATH was looking to find my project code

docker run \
  -it \
  -v "$PWD":/go/src/github.com/integralist/go-requester \
  -w /go/src/github.com/integralist/go-requester \
  -p 8080:8080 \
  golang:latest \
  bash -c "echo Installing Godo... && \
           go get -u gopkg.in/godo.v1/cmd/godo && \
           echo Install complete && \
           godo watch-server --watch"

But there are two problems doing this the "inline" route which is:

  1. It's verbose and you'd end up wanting to alias the command
  2. You miss the caching benefits of putting it into a Dockerfile

The latter issue being the bigger one, because every time you run your container you'd have to re-install Godo. Better to build that as an individual Docker image layer so it can be reused. Let's see how we can do this using a Dockerfile instead:

FROM golang:1.5-onbuild

RUN ["go", "get", "-u", "gopkg.in/godo.v1/cmd/godo"]

CMD ["godo", "watch-server", "--watch"]

Note: you can override the default CMD by specifying your own command during docker run

Once we've defined the Dockerfile, we'll need to build an image from this file, like so:

docker build -t my-golang-app .

The output of the docker build command will look something like the following:

Sending build context to Docker daemon 20.92 MB
Step 0 : FROM golang:1.5-onbuild
# Executing 3 build triggers
Trigger 0, COPY . /go/src/app
Step 0 : COPY . /go/src/app
Trigger 1, RUN go-wrapper download
Step 0 : RUN go-wrapper download
 ---> Running in 23c71aa1e221
+ exec go get -v -d
github.com/integralist/go-requester (download)
Trigger 2, RUN go-wrapper install
Step 0 : RUN go-wrapper install
 ---> Running in 9a0dbf12b11a
+ exec go install -v
github.com/integralist/go-requester/Godeps/_workspace/src/gopkg.in/yaml.v2
app
 ---> 827e083105c9
Removing intermediate container ba603d9887bc
Removing intermediate container 23c71aa1e221
Removing intermediate container 9a0dbf12b11a
Step 1 : RUN go get -u gopkg.in/godo.v1/cmd/godo
 ---> Running in b7eed1e393e8
 ---> Running in b7eed1e393e8
 ---> 4abf07921299
Removing intermediate container b7eed1e393e8
Step 2 : CMD godo watch-server --watch
 ---> Running in 006ce06f640b
 ---> 2321ca26f54d
Removing intermediate container 006ce06f640b
Successfully built 2321ca26f54d

Finally, we can run the built image as a container:

docker run \
  --rm \
  --name go-tester \
  -v "$PWD":/go/src/github.com/integralist/go-requester \
  -w /go/src/github.com/integralist/go-requester \
  -p 8080:8080 \
  my-golang-app

Note: although the onbuild automatically copies our files into the image, we go ahead and mount our same files as a volume so that Godo will recognise when a file has changed (otherwise we'd need to use something like docker exec to jump into the running container to edit our files; which isn't very efficient).

The downside to this approach is that I don't like having my files automatically copied into the container unless I really need them to be. In this case I only want the files to be mounted as a volume otherwise the overall image size will become a lot larger and in this context I'm using Docker more as a development environment. We'll see shortly a more custom Dockerfile which will allow us to work around this issue.

Now when we execute the above docker run command we'll end up kick starting Godo and once the application itself is running I will be able to access it using the following command:

curl $(docker-machine ip dev):8080

One last point to raise at this stage is that you might notice (especially if you're using VirtualBox instead of some other VM provider, like VMWare) that it can take some time for the initial watch command to get going (as well as taking a while for it to recognise a change to be picked up).

This is a really slow process for me and something I'm not overly happy with, but this isn't actually an issue with Go; rather the cause of the problem is an issue with using the VirtualBox VM provider (its network capabilities aren't as good as other paid for providers).

This is just something to be aware of and it might be a deciding factor as to whether you install Go on your host or continue to use Docker depending on whether you want a free (VirtualBox) or paid-for solution (VMWare).

Dependencies and Building Binaries

When it comes to managing dependencies, things get a little tricker due to the way that Go uses a single/global $GOPATH for all your projects rather than having a workspace specific $GOPATH.

There are a few different projects for managing dependencies:

Originally I was using godep. I've since switched to gb as I didn't like godep needing to rewrite the import paths within my code, but also it seemed quite easy to get things into a broken state.

Note: see http://getgb.io/ for full details

When using gb you'll notice that it aligns more closely with a feature that is soon to be supported officially by Go, and that is the concept of 'vendoring'. The idea being is that Go 1.5 currently has an experimental flag which you can enable. Once enabled if you have a vendor directory within your project then Go will look there to resolve your dependencies. If there isn't a folder or the dependency can't be located there, then it'll defer to your global $GOPATH.

So gb works in a similar way in that you tell it what dependencies your project needs and it places them inside a vendor/src directory inside your project workspace. It also creates a vendor/manifest file which contains the details of the dependencies you've fetched and what version they're set to.

Note: if you've ever used Ruby, you'll know this is the concept of Bundler and a Gemfile.lock

When you build your application (using the gb build command) it'll place your project specific dependencies inside the binary for you. So this is a nice approach because it means you get a truer 'project specific' GOPATH setup.

Now in order to avoid committing all the dependencies into my repo (yes there are arguments in the community - across all types of programming languages - about the pros/cons of committing dependencies, but I shan't go into that here) I will simply add the vendor/src directory to my project's .gitignore file. This means I do commit the vendor/manifest file.

So now when a user clones down my project, they might not have any of the dependencies installed, so all they need to do is run gb vendor restore and it'll make sure all the dependencies defined inside the manifest are downloaded into the project's vendor/src directory ready to be used.

Note: yes this does mean whoever is using your project also has to use gb

As far as using this set-up from within Docker, it means I need a bootstrap script to handle only fetching the dependencies if they're missing. This would look something like the following:

if [ ! -d "vendor/src" ]; then
  echo "Fetching dependencies described within manifest..."
  gb vendor restore
fi

if [ ! -d "/go/src/vendor" ]; then
  echo "Copying vendor packages into \$GOPATH..."
  cp -r /go/src/app/vendor/src/* /go/src/

  echo "Below is the container's \$GOPATH (with dependencies copied over)..."
  tree -L 3 /go/src
fi

Note: you can see a fully working version of this in my Go-Requester application

Pulling private dependencies

Earlier we looked at the go get command, which allows you to pull down dependencies from anywhere on the internet. At this point you should be aware of a small issue with using this command in its standard form, and that issue is related to how you can consume private repos from GitHub.

Effectively go get uses HTTPS and when trying to pull a dependency from a private GitHub repository you'll likely want to use SSH instead for the purposes of authentication; otherwise if you don't use your existing SSH keys, then you'll be forced to enter a username and password (not to mention 2FA) and that kills any chance of automation (think Jenkins).

The solution instead is to force go get to use SSH instead of HTTPS. You can do this by executing the following command:

git config --global url."git@github.com:".insteadOf "https://github.com/"

This way when go get makes a request to a HTTPS based GitHub url, our local Git client will force it to be changed into a SSH based request.

Note you can restrict it to a specific organisation as well:
git config --global url."git@github.com:foo/".insteadOf "https://github.com/foo/"

So when you want a private repository such as git@github.com:foo/private.git, you can run:

go get github.com/foo/private

Docker again?

So the above works fine when you have Go installed on the host machine, but when you're using Docker you'll find this wont work; so you'll need to do something different (such as mounting your SSH keys into the running container).

This is where things can get ...fiddly.

In order to achieve what we want we're going to create a bootstrap script (because trying to do 'everything' within a single Dockerfile just isn't practical). In summary we need the following:

Note: you can see a fully working version of this in my Go-Requester application, where I also allow users to utilise expect to automate the entering of their ssh key's passphrase (you do use a passphrase, don't you?); although I'm fully aware that using 'expect' to automate entering your passphrase is painfully insecure and just as bad as having a passphrase-less private key

Let's start at the end, by seeing the Docker run command we'll be using:

docker run \
  -it \
  -v "$HOME/.ssh/github_rsa":/go/src/app/github_rsa \
  -v "$PWD":/go/src/app \
  -p 8080:8080 \
  my-golang-app "ssh_setup"

As you can see, I'm mounting my own GitHub private key ($HOME/.ssh/github_rsa) into the running container, I also pass a string argument of "ssh_setup". In a typical Dockerfile you would need to specify an actual shell command to run, so passing a string wouldn't work. In my case it does. So let's take a look at the Dockerfile to see what changes I've made to facilitate this:

FROM golang:1.5

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "tree"]
RUN ["go", "get", "-u", "gopkg.in/godo.v1/cmd/godo"]
RUN ["go", "get", "-u", "github.com/constabulary/gb/..."]

COPY bootstrap.sh /
RUN chmod +x /bootstrap.sh

RUN mkdir -p /go/src/app
WORKDIR /go/src/app

ENTRYPOINT ["/bin/bash", "/bootstrap.sh"]

You can see that I've modified the ENTRYPOINT for the container to always run our bootstrap script. So the string "ssh_setup" is actually passed to that script as an argument (instead of being interpreted as a command itself).

From here we now should take a look at what our bootstrap script would look like (the following is a small snippet from an actual bootstrap script I'm using in my Go-Requester application):

# allow `go get` for private repositories
git config --global url."git@github.com:".insteadOf "https://github.com/"

if [ "$1" = "ssh_setup" ]; then
  # start our ssh agent
  eval "$(ssh-agent -s)"

  # at this point the user will be asked for their private key's passphrase
  ssh-add /go/src/app/github_rsa

  # automate trusting github as a remote hote
  ssh -o StrictHostKeyChecking=no git@github.com uptime
fi

As you can probably tell by the comments in the script shown above, I'm configuring git and then running an ssh-agent within the container only when the user indicates they require it to handle pulling dependencies from a private repository.

Our docker run command used interactive mode (-it) because we'll be asked to enter our passphrase to decode the GitHub private key we've mounted inside the container.

After that we're working around an issue where the container is unable to verify the host is safe to communicate with by disabling StrictHostKeyChecking whilst trying to ssh into the GitHub domain using the standard git user. It's a dirty hack, but it works.

Baking in Certificates

If you need to include a certificate within your Docker image, then there is one way to achieve this reasonably safely and that is to create an image from an already running container.

To create an image from a running container you must give the container a name (e.g. using docker run --name) and then use docker commit along with the name of the running container.

Finally, you pass the result of that operation to docker tag, which will push the resulting image up to your registry.

docker run --name foo -v /path/to/cert/on/host:/cert.pem centos:centos6
docker tag $(docker commit foo) my-registry.domain.com/foo:1.0.0
docker push my-registry.domain.com/foo:1.0.0

In the above example snippet you'll see we've run a container and mounted our certificate into the container. We then tag the current commit for foo and push that tag up to our registry

Configuring Vim

If you're not a Vim user then you can skip this section. Otherwise, read on...

There are many Go based plugins and configurations you can utilise within Vim, but in my humble opinion, the most essential plugin is Vim-Go (see :h vim-go for the full feature set). When using this plugin you can configure things however you like, but for me the only thing I do is set the following small .vimrc configuration change:

let g:go_fmt_command = "goimports"

By default when you write a buffer that contains a .go formatted file, then the plugin will execute :GoFmt which typically will re-format your entire buffer to the recognised standards the Go community enforces.

In my .vimrc I'm changing that command to use goimports instead, which still formats your entire buffer's content, but also makes sure that it imports any packages that your code references. This is good because it means you can start with an empty Go file and enter fmt.Println("hello") and when you write your buffer you'll find the fmt package has been automatically added to the top of your file as import "fmt".

The plugin includes many useful commands that delegate to the underlying Go installation (such as :GoLint), but what this means in practice is that if you're using Docker for your Go setup, then you'll either have to install Go on your host (e.g. using Homebrew) so when editing Go code from the host it will have access to the Go toolkit; or you have Vim installed in the Go container and edit your files from within the container (Docker all the way down).

Note: to jump to the full list of commands use :h go-commands

So if you're going the Docker route, for running Go, you'll likely want to go "all in" with the Docker ethos and have Docker containers for everything (including Vim). You don't have to obviously, but some people like a challenge.

Vim, Go and Docker

The main reason to do this is because if you're using Docker to run Go applications and you don't have Go installed on your host machine, then the vim-go plugin isn't going to be very useful to you because it relies on Go being available on the machine where Vim is running. Hence, you'll need to have Vim running in a container that has Go installed.

Let's consider a simple example in order to demonstrate the principles. Let's start with a simple Dockerfile that installs Vim inside a container built from a pre-existing Go based image. This will allow us to focus on a simplified docker run command as well. So here follows is our said Dockerfile:

FROM golang:1.5-onbuild

# Install Vim
RUN apt-get update && apt-get install vim -y

Let's now revisit our original super simple Go application which we'll use for the purposes of this demonstration:

package main

import "fmt"

func main() {
  fmt.Println("\nHello World")
}

From here we can build the new image:

docker build -t go-container-with-vim .

And finally, we can run the image as a container and mount inside of it our .vim directory of plugins along with our .vimrc configuration file as well (you can use a non onbuild version of a Go Dockerfile if you prefer, as it'll allow you to choose where you copy/add your application files into; it's totally up to you):

docker run \
  -it \
  -v "$HOME/.vimrc":/root/.vimrc \
  -v "$HOME/.vim":/root/.vim \
  go-container-with-vim /bin/bash

Note: I had to tweak/comment out one line in my .vimrc file to prevent the Linux version of Vim from throwing an error; your mileage may vary

As far as running Vim inside a container that has Go pre-installed as well, that's pretty much all there is to it.

Compilation

We've already seen how to build a simple binary using the gb command earlier (e.g. gb build all). This sticks the built binary into the directory bin inside your project workspace. But this binary, by default (for me any way, running things on a Mac) is a Mac OS X binary, so what about compiling binaries for another OS such as Linux?

Well, with gb you can compile a binary for any OS architecture like so:

GOOS=linux GOARCH=amd64 gb build

You just need to change the environment variables before running the standard gb build command.

Note: Gox is a nice alternative,
but effectively it does the same thing so might as well keep with gb build

Go-Requester

OK, we're nearing the finish line!

The last thing I want to discuss is the Go-Requester application I've been referencing throughout this article so far. In summary it's a HTTP service that accepts a collection of "components", fans-out requests for these remote components and returns the results in an aggregated format ready to be consumed.

It's nothing fancy, but it utilises some interesting parts of the Go programming language; and if you're relatively new to Go (like I am) then you might find it interesting to see these different concepts in use 'out in the wild' rather than just the basic examples you typically see in API documentation.

Note: I wrote this very early on in my Go career and so there are quite a few refactors I'd like to make to this code base already. But in the interest of openness I demonstrate the code 'as is'. I'd also recommend sifting through the code on GitHub as you'll see it all in context, rather than the bits and pieces I'll be extracting below

So let's start with the main function:

if len(os.Args) < 2 {
  fmt.Println("No config file path provided")
  os.Exit(1)
}

configPath = os.Args[1] // zero index is the binary command itself

http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)

Nothing too complicated here, we're relying on a config file path to be provided (this file will dictate what 'components' need to be loaded and from where they can be found); and if a config file isn't provided then we exit with a status code of 1 along with an error message to indicate what the cause of the problem was.

Otherwise we move on and store the config path into a global variable for later use, then we setup a web server and associated handler function. So let's take a look at that handler function to see what it does:

var cr []componentResponse
var y componentsList

ch := make(chan componentResponse)

b := getComponents()
yaml.Unmarshal(b, &y)

timeout := time.Duration(1 * time.Second)
client := http.Client{
  Timeout: timeout,
}

var wg sync.WaitGroup
for i, v := range y.Components {
  wg.Add(1)
  go getComponent(&wg, &client, i, v, ch)
  cr = append(cr, <-ch)
}
wg.Wait()

j, err := json.Marshal(result{finalSummary(cr), cr})
if err != nil {
  fmt.Printf("Problem converting to JSON: %s\n", err)
  return
}

w.Header().Set("Content-Type", "application/json")
w.Write(j)

So let's break this down into stages. I'm not going to do a line-by-line explanation, but I will just cover the 'highlights' if you will. The handler function itself consists of the following functionality:

Now the interesting parts of the code is the use of goroutines to ensure that when we're requesting components from a remote endpoint, that this is as highly concurrent an operation as possible.

The issue with using goroutines is that because they act like threads (well, it is effectively a thread pool in use) your main thread (the one running the main function) will finish before all the components have been requested.

To work around this issue we use a sync.WaitGroup to ensure the main thread stays running as long as necessary (until all the gorountines have finished). It does this by keeping track of how many individual group 'items' there are (e.g. wg.Add(1)).

We create a WaitGroup, and for every loop iteration (i.e. each component) we add another increment wg.Add(1). Then we tell the main thread it has to wait: wg.Wait() until all the items have finished.

The way we indicate a thread/goroutine/item has finished is by executing wg.Done() at the end of the getComponent function. That ends up being the return value for the goroutine running that function.

Note: this is a similar concept to Promises, which are used in JavaScript

The getComponent function should run wg.Done() at the end of the function, but you'll notice it's written in the very first line of the function body. The reason this works is because of the defer statement which effectively says "defer executing this line of code until the end of this function".

The defer statement is really useful for allowing you to group together related data and code and yet have it execute at another point in time.

The other interesting aspect here is that we're appending data to an Array that's actually coming out of a channel (i.e. append(cr, <-ch)). When we call getComponent you'll notice that we pass our pre-existing channel into the function and then pass messages through that channel (whether it be a 'successful' componentResponse or a 'failed' one).

The use of channels here is really nice because it allows us to synchronise our communication across shared memory space (i.e. the goroutines/threads), thus avoiding the typical problems of concurrent/parallel code execution. This is one of the major tenets of the Go programming language.

Note: you'll find the exact same use of channels in the Clojure programming language, which recognised that CSP was a strong concurrency model and a good choice for their users

Another interesting aspect of the code is the use of 'type assertion inference' used with our checkError function:

if e, ok := err.(net.Error); ok && e.Timeout() {
  return 408
}
return 500

Here we're taking err, which is an error type, and trying to assert whether it's also an net.Error. If it is then that particular type has a Timeout method we can use to verify if the error was caused because of a net timeout issue (if it was a timeout then we'll return a timeout specific status code; otherwise we'll return a generic 500 status code).

Note: Go also has a switch statement specifically for handling type assertions in this manner: https://golang.org/doc/effectivego.html#typeswitch and is well worth your time to investigate further

One last thing worth mentioning for those readers who are new to Go, is the use of complex nested JSON data structures. You'll see in my code that I'm creating multiple struct types to make handling the nested structures a little easier to deal with:

type componentResponse struct {
  ID        string `json:"id"`
  Status    int    `json:"status"`
  Body      string `json:"body"`
  Summary   string `json:"summary"`
  Mandatory bool   `json:"mandatory"`
}

type result struct {
  Summary    string              `json:"summary"`
  Components []componentResponse `json:"components"`
}

What we've done here is defined a struct called result and stated it should have two keys Summary and Components. The latter is of type Array and that Array will consist of elements of type componentResponse (itself a struct).

We then define componentResponse as a separate struct and drill down into the nested JSON keys it contains (e.g. json:"id" etc).

If you've not seen how Go handles JSON before, then effectively Go requires keys in its structs to be capitalised; but JSON conventions state keys should be lowercase, so we use struct 'tags' to determine how to convert the keys between JSON and Go. So when you're marshalling a Struct into JSON or vice-versa you'll have the keys translated in the way that you expect/require them to be. A very nice feature indeed.

Conclusion

I think that does it for now. We've covered a lot of ground regarding the practicalities of tooling (specifically Docker) surrounding Go and how I'm currently utilising it along with Vim and other Go based tools to help improve my Go programming workflow.

I hope you found this all very interesting, and if you have any comments then please come find me on twitter or open a discussion up on GitHub.


Links