Managing project tools with Go

Mon, 12 May 2025 00:00:00 +0000

There are multiple ways to deal with non-application dependencies (i.e. “tools” that your project needs).

go tool

As of Go 1.24 (Feb 2025)

To add a new tool:

go get -tool golang.org/x/lint/golint
go get -tool github.com/mgechev/revive@latest

To run the tool:

go tool golint -h
go tool golang.org/x/lint/golint -h # in case of naming overlap

To see a list of all tools:

go tool

To update all tools:

go get -u tool

If you check the go.mod you’ll see a new tool syntax:

module testing-tools

go 1.23.4

tool (
    github.com/mgechev/revive
    golang.org/x/lint/golint
)

Caveats and Issues

Now, there is a problem (sort of), which is that you’ll see a bunch of indirect dependencies showing up in the go.mod.

This is because these are the dependencies that your “tools” need.

I’m less concerned about that as a side-effect of using the new go tools feature, but I appreciate it’s not ideal.

My concern being: it’s more mental overhead.

You don’t know if these indirect dependencies are transient dependencies used by your application dependencies, or if they’re dependencies for the “tools” you’ve installed.

The reason I’m not usually that fussed by this is because I only really care about the “direct” dependencies, and those are always clear because they don’t have // indirect following them.

So the following instructions are only relevant if you really care about this.

Multiple Module Files

There is another option on the table that we can use, and it doesn’t appear to be too much additional maintenance or mental overhead, which is great. But it does have a downside (see the IMPORTANT note at the end of this section).

Essentially, the approach is to have a separate modfile for tools.

It means we’d have multiple files now, like this…

go.mod
go.sum
tools.mod
tools.sum

❗ IMPORTANT

If you give the tools.mod a unique module name, let’s say go.mod uses github.com/example/foo, and so you make tools.mod use github.com/example/foo/tools then be aware that the use of go mod isn’t going to make your tools.mod think it needs the module from go.mod and it’ll add it as a dependency (this makes things weird in special cases), so it might be worth making the module name the same between go.mod and tools.mod.

To install a new tool:

# instead of...
go get -tool github.com/mgechev/revive

# we do...
go get -modfile=tools.mod -tool github.com/mgechev/revive

💡 TIP

To remove a tool you can do the above but set the version to @none.

And if we want to use that tool we have to make sure to specify the modfile:

$ go tool revive --version
go: no such tool "revive"

$ go tool -modfile=tools.mod revive --version
version 1.7.0

Having to specify the -modfile flag isn’t a big issue as we already have go tool abstracted inside the various Makefile targets, so we should only ever be calling a Makefile target (or in the case of stringer have it codified in the go generate directive in the code itself).

As far as updating tools, you can either do it a dependency at a time or all of them at once:

# instead of...
go get -u -tool github.com/mgechev/revive@latest
go get -u tool

# we do...
go get -u -modfile=tools.mod -tool github.com/mgechev/revive@latest
go get -u -modfile=tools.mod tool

Same for listing the installed tools:

# instead of...
go tool

# we do...
go tool -modfile=tools.mod

💡 TIP

Can also try go list -modfile=tools.mod tool

To verify the integrity of the tool dependencies:

go mod verify -modfile=tools.mod

Here’s an associated Makefile:

.PHONY: deps-app-update
deps-app-update: ## Update all application dependencies
    go get -u -t ./...
    go mod tidy
    if [ -d "vendor" ]; then go mod vendor; fi
    
.PHONY: deps-outdated
deps-outdated:  ## Lists direct dependencies that have a newer version available
    @go list -u -m -json all | go tool -modfile=tools.mod go-mod-outdated -update -direct
    
TOOLS = \
    cuelang.org/go/cmd/cue \
    github.com/client9/misspell/cmd/misspell \
    github.com/go-delve/delve/cmd/dlv \
    github.com/mgechev/revive \
    github.com/psampaz/go-mod-outdated \
    github.com/stealthrocket/wasi-go/cmd/wasirun \
    github.com/stern/stern \
    github.com/tetratelabs/wazero/cmd/wazero \
    golang.org/x/lint/golint \
    golang.org/x/tools/cmd/stringer \
    golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness \
    golang.org/x/vuln/cmd/govulncheck \
    honnef.co/go/tools/cmd/staticcheck \
    mvdan.cc/gofumpt \

.PHONY: tools
tools:
    @$(foreach tool,$(TOOLS), \
        if ! go tool -modfile=tools.mod | grep "$(tool)" >/dev/null; then \
            go get -modfile=tools.mod -tool "$(tool)"@latest; \
        fi; \
    )

.PHONY: tools-update
tools-update:
    go get -u -modfile=tools.mod tool
    go mod tidy -modfile=tools.mod

❗ IMPORTANT

This approach keeps the main go.mod and go.sum clean of any tool dependencies, but not the other way around. So the tools.mod and tools.sum will ultimately contain all the dependencies from the main go.mod (that is a side-effect of running go mod tidy -modfile=tools.mod as go mod always consults the main go.mod, hence all of its dependencies end up in your tools.mod and tools.sum).

This is unavoidable. There is no way to get around it (trust me, I’ve tried 😅).

Now, this isn’t the end of the world as the tools directive is still at the top of the tools.mod and is very clear as to what “tools” are installed, but yeah, you’ll also see a bunch of require directives (related to your main Go project) as well, unfortunately.

One thing you could do, is only run the go get -u -modfile=tools.mod tool command, which would keep your tools.mod clean, and would only update tools.sum with the relevant updated dependencies. The problem with that is the old dependencies aren’t cleaned out. e.g. if you updated tool “foo” from version 1.0 to 2.0 then both versions appear in your tools.sum (this is why we have go mod tidy to ensure only 2.0 is present in the tools.sum). So one approach would simple be to manually clean up the go.sum everytime after running go get -u -modfile=tools.mod tool – it’s not that difficult as you just look for the new tool version added and remove the old one, but it’s a manual process and that sucks).

tools.go

ℹ️ NOTE

For more details on code generation in a general sense, refer to:
https://gist.github.com/Integralist/8f39eb897316e1cbeaf9eff8326cfa59

The following file internal/tools/tools.go uses a build tag to avoid the dependencies being compiled into your application binary…

//go:build tools

// Package tools manages go-based tools that are used to develop in this repo.
package tools

import (
    _ "github.com/nbio/cart"
    _ "github.com/nbio/slugger"
    _ "github.com/psampaz/go-mod-outdated"
    _ "github.com/stealthrocket/wasi-go/cmd/wasirun"
    _ "github.com/tetratelabs/wazero/cmd/wazero"
    _ "golang.org/x/lint/golint"
    _ "golang.org/x/tools/cmd/stringer"
    _ "golang.org/x/vuln/cmd/govulncheck"
)

//go:generate go install github.com/nbio/cart
//go:generate go install github.com/nbio/slugger
//go:generate go install github.com/psampaz/go-mod-outdated
//go:generate go install github.com/stealthrocket/wasi-go/cmd/wasirun
//go:generate go install github.com/tetratelabs/wazero/cmd/wazero
//go:generate go install golang.org/x/lint/golint
//go:generate go install golang.org/x/vuln/cmd/govulncheck
//go:generate go install golang.org/x/tools/cmd/stringer

Notice the go:generate comments? Yup, we invoke them like so (notice the -tags flag):

tools: internal/tools/tools.go
    go generate -v -x -tags tools ./internal/tools/...

go run

An alternative to this approach is to use go run directly, which downloads tools to a cache but doesn’t install them and yet still gives you explicit versioning consistency across developer’s machines…

//go:generate go run golang.org/x/tools/cmd/stringer@v0.25.0 -type=Scope -linecomment

I then invoke go generation with:

.PHONY: go-gen
go-gen: ## Invoke go generate
    @# The `-x` flag prints the shell commands that `go generate` runs.
    go generate -v -x ./mustang/status/...

💡 TIP

If you’re developing whilst offline, then one advantage the tools.go pattern has is that it works whilst offline because the tool is explicitly installed. But to work around that with go run you can set export GOPROXY=direct and as long as you have the module in your local cache you’ll be able to use it.

Manually install and auto-switch Golang versions

Sun, 02 Feb 2025 00:00:00 +0000

If you work on muliple Go projects, you’ll often find they require different Go versions. So how do you handle switching between Go versions?

You can’t rely on your package manager as it might only provide the latest version of Go (such as is the case when using Homebrew on macOS) or it might not provide all prior Go versions (maybe only a subset). Then you have to decide whether the switch is something you do manually or automatically when you cd into a Go project directory (but how do you determine and implement that?).

Typically you’ll use a third-party tool that does this for you. I’ve historically used a lot of tools. The last tool I used was stefanmaric/g and it was working fine …for a while, until one day it stopped working and for the life of me I couldn’t figure out why.

The problem with using a third-party tool is that if it does go wrong, it’s very hard to debug and fix. With this in mind, I decided I would have a go at solving the problem in a way that worked for me.

⚠️ WARNING

This is NOT a perfect solution, and in some cases it’s a poor solution.

In this article I’m going to show you the code I wrote to implement Go-version switching, as well as covering some of the caveats of the approach I took. But ultimately, this is a solution I’ve implemented from scratch and so I intrinsically understand it and will understand how it works better than anyone and will understand more easily what to do if for some reason there’s a bug or scenario that I didn’t account for when first creating it. This is why I provide the above warning note. Feel free to use my approach, or use the code as an example from which to build your own solution.

ℹ️ NOTE

As an alternative, some people prefer to have a single Go version install (i.e. GOROOT) and then when they use the go binary to install other versions they will simply create an alias (manually) or have a Makefile accept a GO_BIN override. It’s definitely a much simpler approach if you prefer that.

Let’s dig in and see what we have…

Shell Structure

OK, let’s start with how I like to structure my shell files.

I have a .zshrc from which I then load in other shell scripts.

To avoid muddying the water I’ll show a truncated version:

ℹ️ NOTE

If you want the full version of the code we’re discussing,
then refer to my dotfiles repo.

#!/usr/bin/zsh

function load_script {
    local path=$1
    if test -f $path; then
        source $path
    else
        echo "no $path found"
    fi
}

load_script ~/.config/zsh/tools.zsh

Cool, so we know we need a tools.zsh script. Let’s take a look at the relevant sections of that file. We’ll start with the exports…

Exports

export GOPATH="$HOME/go"
export GOROOT="$HOME/.go"
export PATH="$GOPATH/bin:$GOROOT/bin:$PATH";

The GOPATH is where we install Go CLI programs, and for our purposes it’s where we will install our different Go versions.

The GOROOT is where we install our primary Go version (this is the Go version we start with and is the version we keep up-to-date with the latest Go release).

The PATH is where our shell attempts to lookup executable binaries, such as the go binary.

You can see we make sure $GOPATH/bin is checked first, then failing that it’ll check $GOROOT/bin, before considering any other entries in the $PATH.

You can probably already guess the approach I’m taking, but if not, it’s this: by having $GOPATH/bin as the first entry in my $PATH, it means I can install my different Go version binaries there, and then create a symlink for go in that same directory to point to the specific Go version binary I want to be using.

Installing Go for GOROOT

The first thing we need to do is identify if we have Go installed at all. We do this by checking if there is a go binary file in our $GOROOT/bin directory. If there isn’t a file there, then we identify the latest Go version release and download it into $GOROOT/bin:

if [ ! -f $GOROOT/bin/go ]; then
    mkdir -p "$GOPATH"
    mkdir -p "$GOROOT"

    GO_VERSION=$(golatest)
    OS=$(uname | tr '[:upper:]' '[:lower:]')
    ARCH=$(uname -m)
    URL="https://go.dev/dl/go${GO_VERSION}.${OS}-${ARCH}.tar.gz"
    TMP_DL="/tmp/go.tar.gz"

    echo "Downloading latest Go archive from $URL"
    curl -Lo "$TMP_DL" "$URL"

    # Extract the tar.gz file to the installation directory
    # The --strip-components=1 skips the go/ directory within the archive.
    # This ensures the ~/.go directory contains bin/ rather than ~/.go/go/bin
    echo "Extracting Go archive to $GOROOT"
    tar -C "$GOROOT" --strip-components=1 -xzf "$TMP_DL"

    # Cleanup the downloaded archive
    echo "Cleaning up Go archive from $TMP_DL"
    rm "$TMP_DL"
fi

ℹ️ NOTE

I’ve used golatest in the above code. The implementation for that is as follows:

alias golatest="curl -L https://github.com/golang/go/tags 2>&1 | \
    rg '/golang/go/releases/tag/go[\w.]+' -o | \
    cut -d '/' -f 6 | \
    grep -v 'rc' | \
    awk NR==1 | \
    rg '\d.+' -o"

Triggering a switch on directory change

So the key part to all this is checking the current directory to see if we need to download a different Go version. In the Zsh shell you have access to a builtin function called chpwd which runs every time you change directory. Changing directory is typically done using cd but it also works if you use a tool like ajeetdsouza/zoxide (like I do) to quickly jump around common project directories.

ℹ️ NOTE

See the Zsh hook functions docs.

Here is the relevant parts of our chpwd function:

function chpwd() {
    ls

    # figure out go version
    #
    local v=""
    if [ -e go.mod ]; then
        v=$(awk '/^go [0-9]+\.[0-9]+/ { print $2 }' go.mod)
        # go.mod isn't always going to contain a complete version (e.g. 1.20 vs 1.20.1)
        # we need a complete version for installing and symlinking.
        #
        if [[ ! "$v" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
            latest_patch=$(gh api repos/golang/go/tags --jq '.[].name' --paginate \
                | grep -E "^go${v}\.[0-9]+$" \
                | sed 's/^go//' \
                | sort -V \
                | tail -n 1)
            if [ -n "$latest_patch" ]; then
                v="$latest_patch"
            else
                echo "Failed to fetch the latest patch version for $v"
                go_symlink_remove # remove symlink so the PATH lookup finds the GOROOT binary.
                v="" # Ensure v is empty to prevent executing the install steps
            fi
        fi
    elif [ -e .go-version ]; then
        v="$(cat .go-version)"
    fi
    if [ -n "$v" ]; then
        # create go dependencies cache directory if it doesn't exist.
        local cache_dir="$HOME/.cache/go-deps"
        if [[ ! -d "$cache_dir" ]]; then
            mkdir -p "$HOME/.cache/go-deps"
        fi
        local cache_file="$cache_dir/go$v"

        if [[ ! -f "$cache_file" ]]; then
            go_install "$v" # installs the specified Go version
            go_symlink "$v" # ensures `go` now references the specified Go version
            go_tools # ensures we have all the tools we need for this Go version
            touch "$cache_file" # update last_modified date
        else
            go_symlink "$v" # ensures `go` now references the specified Go version

            local current_day=$(date +%Y-%m-%d)
            local last_modified_day=$(date -r "$cache_file" +%Y-%m-%d)

            # if the cache file was last modified on a different day, run the command
            if [ "$current_day" != "$last_modified_day" ]; then
                echo "updating go$v dependencies (last updated: $last_modified_day)"
                go_tools # ensures we have all the tools we need for this Go version
                touch "$cache_file" # update last_modified date
            fi
        fi

        r # reload shell so starship can display the updated go version
    fi

    # clean out any .DS_Store files
    #
    if [[ $PWD != $HOME ]]; then
        # find . -type f -name '.DS_Store' -delete
        fd '.DS_Store' --type f --hidden --absolute-path | xargs -I {} rm {}
    fi
}

In the above script we do the following:

Check if the directory contains a go.mod or a .go-version.
If there’s a go.mod then we check if it’s a ‘complete’ version.
If it’s not a complete version, we identify the latest patch available.
If it’s a .go-version then we know that will contain a full version.
We store whatever version we find or calculate into v.
We then call some custom functions and pass them v.

For that last bullet, the functions we call are:

go_install
go_symlink
go_tools

The implementation for those functions are:

# go_install installs the specified version
function go_install() {
  if [ -z "$1" ]; then
        echo "Pass a Go version (e.g. 1.21.13)"
    return
  fi
    local v="$1"
    go install "golang.org/dl/go$v@latest"
    "$GOPATH/bin/go$v" download
    "$GOPATH/bin/go$v" version
}

# go_symlink is called by chpwd to allow a different go version binary to be used.
# if the specified version binary doesn't exist, we install it first.
function go_symlink() {
  if [ -z "$1" ]; then
        echo "Pass a Go version (e.g. 1.21.13)"
    return
  fi
    local v=$1
    if [ ! -f "$GOPATH/bin/go$v" ]; then
        go_install "$v"
    fi
    ln -sf "$GOPATH/bin/go$v" "$GOPATH/bin/go"
}

# go_tools installs/updates necessary Go tools.
function go_tools {
  local golangcilatest=$(curl -s "https://github.com/golangci/golangci-lint/releases" | \
    grep -o 'tag/v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -n 1 | cut -d '/' -f 2)
  curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | \
    sh -s -- -b $(go env GOPATH)/bin "$golangcilatest"
  go install github.com/rakyll/gotest@latest
  go install github.com/mgechev/revive@latest
  go install golang.org/x/tools/gopls@latest
  go install mvdan.cc/gofumpt@latest
  go install honnef.co/go/tools/cmd/staticcheck@latest # https://github.com/dominikh/go-tools
  go install golang.org/x/vuln/cmd/govulncheck@latest
  go install github.com/go-delve/delve/cmd/dlv@latest
  go install go.uber.org/nilaway/cmd/nilaway@latest
  go install golang.org/x/tools/cmd/goimports@latest
  go install github.com/incu6us/goimports-reviser/v3@latest
  go install github.com/google/gops@latest
  go install github.com/securego/gosec/v2/cmd/gosec@latest
}

Finally, we run r which is an alias that reloads the .zshrc file:

alias r="source ~/.zshrc"

Why do we reload the shell configuration? Well, I use the Starship shell prompt, and that has its own logic for determining the Go version, and now with the above workflow it often reports the wrong Go version. But once I reload my shell configuration it’ll pick up the go binary that is now being symlinked to a specific Go version.

Now, there’s a performance improvement I made to the code (which you can see in the earlier code snippet but I didn’t explain):

# create go dependencies cache directory if it doesn't exist.
local cache_dir="$HOME/.cache/go-deps"
if [[ ! -d "$cache_dir" ]]; then
    mkdir -p "$HOME/.cache/go-deps"
fi
local cache_file="$cache_dir/go$v"

if [[ ! -f "$cache_file" ]]; then
    go_install "$v" # installs the specified Go version
    go_symlink "$v" # ensures `go` now references the specified Go version
    go_tools # ensures we have all the tools we need for this Go version
    touch "$cache_file" # update last_modified date
else
    go_symlink "$v" # ensures `go` now references the specified Go version

    local current_day=$(date +%Y-%m-%d)
    local last_modified_day=$(date -r "$cache_file" +%Y-%m-%d)

    # if the cache file was last modified on a different day, run the command
    if [ "$current_day" != "$last_modified_day" ]; then
        echo "updating go$v dependencies (last updated: $last_modified_day)"
        go_tools # ensures we have all the tools we need for this Go version
        touch "$cache_file" # update last_modified date
    fi
fi

That improvement was to check if a cache file exists for the Go version, and if so, we’ll see if the file was updated at some point in the last day. If it has been updated already then we don’t bother updating the Go tools and dependencies for the specified Go version

We only really want to do that once a day, otherwise every time you change directory to another Go project it will unnecessarily downloads dependencies you already have and that takes time.

Go Concurrency Patterns

Fri, 15 Nov 2024 00:00:00 +0000

Goroutines

The go keyword is used to start a goroutine.

A goroutine is a lightweight, managed thread used by the Go runtime to run functions concurrently.

Unlike OS threads, which have a fixed stack size (often around 1MB), goroutines start with a very small stack, around 2KB, and can grow or shrink dynamically as needed. This makes it possible to run thousands or even millions of goroutines simultaneously, depending on the available memory.

Goroutines are sometimes compared to “green threads,” which are threads that are scheduled in user space rather than by the OS. The problem with green threads is that they may not leverage multiple CPU cores efficiently since they don’t interact directly with the OS’s scheduler.

Goroutines are similar to green threads in that they are scheduled by the Go runtime rather than the OS. However, they differ in a crucial way: Go uses a model called Mscheduling, where the Go runtime maps multiple goroutines (M) onto multiple OS threads (N). This allows the runtime to distribute goroutines across multiple CPU cores when possible, making Go’s concurrency model more efficient and scalable than traditional green threads.

❗ IMPORTANT

The following example uses a time.Sleep to wait for the goroutine to finish.
This is done for simplicity. Do NOT use this approach.
I’ll explain alternative options afterwards.

package main

import (
    "fmt"
    "time"
)

func sayHello() {
    fmt.Println("Hello from a goroutine!")
}

func main() {
    go sayHello() // Launches sayHello in a new goroutine

    fmt.Println("Hello from main!")
    time.Sleep(1 * time.Second) // Wait for the goroutine to complete (only for example)
}

https://play.golang.com/p/SdOdZ90-exI

ℹ️ NOTE

In Go, the main function is effectively the “initial” goroutine.
When a Go program starts, the Go runtime creates a goroutine to run main.
This main goroutine can then spawn additional goroutines as needed.

Channels

Channels in Go are a powerful way to communicate between goroutines and to synchronize them. They allow you to send and receive values across goroutines, and they help avoid race conditions by enabling safe data sharing.

❗ IMPORTANT

Sending and receiving channel messages can block.
In the following example the <-ch BLOCKS the main() function.
While the ch <- BLOCKS the sendMessage() function.
This can cause a deadlock, hence we use a goroutine to unblock things.

package main

import (
    "fmt"
)

func sendMessage(ch chan string) {
    ch <- "Hello from a goroutine!" // Send a message to the channel
                                    // ⚠️ This BLOCKS but as it's running in a goroutine 
                                    // the program is unaffected.
}

func main() {
    // Create a new channel of type string
    ch := make(chan string)

    // Start a goroutine to send a message
    go sendMessage(ch)

    // Receive the message from the channel
    msg := <-ch // ⚠️ This BLOCKS the program (hence sendMessage() runs in a goroutine)
    fmt.Println(msg) // Output: Hello from a goroutine!
}

https://play.golang.com/p/2Qn_NacVw-0

❗ IMPORTANT

You can range over a channel, but the loop will never stop unless the channel is closed.
So when ranging over a channel, think how the program can proceed and when is the channel going to be closed.

[!NOTE] Depending on what you application needs, you can create buffered channels.
Sends to a buffered channel block only when the buffer is full.

[!TIP] The most crucial best practice is to close the channel from the sender side, not the receiver.
The sender is the goroutine that writes data to the channel.
Hence the sender knows when there’s no more data to be sent, not the receiver.

Select Statement

The select statement is used to wait on multiple channel operations.
It blocks until one of its cases can proceed, which makes it essential for handling multiple asynchronous tasks.

Use select when you have multiple channels to listen to, and you want to respond to whichever channel receives data first.

❗ IMPORTANT

In the following example, the first goroutine uses a time.Sleep.
This is to simulate the operation taking a long time.
It results in the select pulling a value from the second goroutine.

package main

import (
    "fmt"
    "time"
)

func main() {
    ch1 := make(chan string)
    ch2 := make(chan string)

    go func() {
        time.Sleep(1 * time.Second)
        ch1 <- "result from ch1"
    }()

    go func() {
        ch2 <- "result from ch2"
    }()

    select {
    case msg1 := <-ch1:
        fmt.Println("Received:", msg1)
    case msg2 := <-ch2:
        fmt.Println("Received:", msg2)
    }
}

https://play.golang.com/p/HXe-bZ__EEy

A common use case for select is to timeout a potential deadlock:

❗ IMPORTANT

In the following example we use time.After to cause a timeout.

package main

import (
    "fmt"
    "time"
)

func main() {
    // Create an unbuffered channel
    ch := make(chan string)

    // Start a goroutine that simulates a delayed send
    go func() {
        time.Sleep(3 * time.Second)   // Simulate a delay
        ch <- "Hello from goroutine!" // Send a message after delay
    }()

    select {
    case msg := <-ch:
        fmt.Println("Received:", msg)
    case <-time.After(2 * time.Second): // Timeout after 2 seconds
        fmt.Println("Timeout! No message received.")
    }
}

https://play.golang.com/p/6BMPeUKdqkg

Wait Groups

A sync.WaitGroup waits for a collection of goroutines to finish. It helps coordinate a group of goroutines and ensures the program waits until all of them have completed before proceeding.

Use a WaitGroup when you need to wait for multiple goroutines to finish before moving on.

In this example, a sync.WaitGroup is used to wait for three goroutines to complete. Each goroutine represents a worker, and each one calls wg.Done() to signal that it’s finished. The main function calls wg.Wait() to block until all workers are done.

package main

import (
    "fmt"
    "sync"
)

func worker(id int, wg *sync.WaitGroup) {
    defer wg.Done() // Decrement counter when goroutine completes
    fmt.Printf("Worker %d starting\n", id)
    fmt.Printf("Worker %d done\n", id)
}

func main() {
    var wg sync.WaitGroup

    for i := range 3 {
        wg.Add(1) // Track each goroutine started
        go worker(i, &wg)
    }

    // Wait for all goroutines to finish
    wg.Wait()

    fmt.Println("All workers done.")
}

https://play.golang.com/p/LhEdSQIPp1R

💡 UPDATE:
As of Go 1.25 you no longer need to wg.Add/wg.Defer for simple cases.
Instead you can just use wg.Go().

package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup

    wg.Go(func() {
        fmt.Println("go is awesome")
    })

    wg.Go(func() {
        fmt.Println("cats are cute")
    })

    wg.Wait()
    fmt.Println("done")
}

https://go.dev/play/p/b_WdOO7IYVM?v=gotip

Error Groups

The errgroup package is like a sync.WaitGroup but with added capabilities for error handling and context cancellation.

While a WaitGroup simply waits for a collection of goroutines to finish, an errgroup also collects any errors returned by those goroutines and can cancel all of them if one fails.

In the following example, when one worker fails, it triggers a cancellation in the other workers (some might complete in time, but those that don’t will be forcefully cancelled):

package main

import (
    "context"
    "fmt"
    "net/http"
    "time"

    "golang.org/x/sync/errgroup"
)

func main() {
    ctx := context.Background()

    // errgroup.WithContext creates a new Group and a derived Context.
    // The errgroup context is cancelled when the first goroutine returns a non-nil
    // error or when the g.Wait function returns.
    g, ctx := errgroup.WithContext(ctx)

    // List of URLs to fetch
    urls := []string{
        "https://http-me.fastly.dev/status=200",
        "https://http-me.fastly.dev/?wait=1000&status=201", // This never completes (i.e. goroutine is cancelled)
        "https://http-me.fastly.dev/?wait=1000&status=202", // This never completes (i.e. goroutine is cancelled)
        "https://http-me.fastly.dev/status=500",            // This one will return an error
    }

    for _, url := range urls {
        u := url

        // Start a goroutine for each URL
        g.Go(func() error {
            fmt.Println("Goroutine for ", u)

            // Create a request with context
            req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
            if err != nil {
                return fmt.Errorf("creating request: %w", err)
            }

            resp, err := http.DefaultClient.Do(req)
            if err != nil {
                return fmt.Errorf("request failed: %w", err)
            }
            defer resp.Body.Close()

            // Treat non-200 status codes (1xx, 3xx, 4xx, 5xx) as errors
            if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
                return fmt.Errorf("non-200 status: %d from %s", resp.StatusCode, u)
            }

            select {
            case <-time.After(time.Millisecond):
                // don't block forever, just enough to check context.Done
            case <-ctx.Done():
                // This block is executed if the context is cancelled before the work is finished.
                fmt.Printf("Goroutine %s cancelled\n", u)
                return ctx.Err()
            }

            fmt.Printf("Successfully fetched: %s\n", u)
            return nil
        })
    }

    // Wait for all goroutines to complete or any to return an error
    if err := g.Wait(); err != nil {
        fmt.Printf("Error: %v\n", err)
    } else {
        fmt.Println("All fetches succeeded.")
    }
}

The output of the program is non-deterministic. So, for example, you might see:

Goroutine for  https://http-me.fastly.dev/status=500
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=202
Goroutine for  https://http-me.fastly.dev/status=200
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=201
Error: non-200 status: 500 from https://http-me.fastly.dev/status=500

Notice in the above output no goroutine is successful. The error is picked up from the status=500 URL and so all other goroutines are cancelled immediately.

Where as if you run the code again you might see this:

Goroutine for  https://http-me.fastly.dev/status=500
Goroutine for  https://http-me.fastly.dev/status=200
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=201
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=202
Goroutine https://http-me.fastly.dev/status=200 cancelled
Error: non-200 status: 500 from https://http-me.fastly.dev/status=500

Notice in the above output we now see one of the goroutines were explicitly cancelled, i.e. the select statement had time to run the check on the ctx.Done() call.

If we want to see any of these goroutines fail we need to play around with the delays to give one of them a chance of succeeding. So something like this would do it:

urls := []string{
    "https://http-me.fastly.dev/status=200",
    "https://http-me.fastly.dev/?wait=1000&status=201",
    "https://http-me.fastly.dev/?wait=1000&status=202",
    "https://http-me.fastly.dev/?wait=500&status=500",
}

Notice we’ve set a smaller delay (wait=500) on the status=500 URL.

When running the code again we’d see something like this:

Goroutine for  https://http-me.fastly.dev/?wait=500&status=500
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=201
Goroutine for  https://http-me.fastly.dev/status=200
Goroutine for  https://http-me.fastly.dev/?wait=1000&status=202
Successfully fetched: https://http-me.fastly.dev/status=200
Error: non-200 status: 500 from https://http-me.fastly.dev/?wait=500&status=500

Notice in the above output we now see at least one successful case.

💡 TIP

You can limit the number of active goroutines in the group with:
g.SetLimit(10)
Calls to g.Go will block until an active goroutine can be added.

Mutex

Go’s sync.Mutex provides mutual exclusion, allowing only one goroutine at a time to access a critical section of code.
While sync.RWMutex is a variant that allows multiple readers or a single writer but not both.

Use sync.Mutex or sync.RWMutex when you need fine-grained control over data access and want to protect shared data from race conditions.

In the below example sync.Mutex ensures that only one goroutine modifies counter.value at a time, preventing race conditions:

package main

import (
    "fmt"
    "sync"
)

type Counter struct {
    mu    sync.Mutex
    value int
}

func (c *Counter) Increment() {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.value++
}

func (c *Counter) Value() int {
    c.mu.Lock()
    defer c.mu.Unlock()
    return c.value
}

func main() {
    counter := &Counter{}
    var wg sync.WaitGroup

    for range 10 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            counter.Increment()
        }()
    }

    wg.Wait()
    
    fmt.Println("Final Counter:", counter.Value())
}

https://play.golang.com/p/VIbNkQaPfZI

ℹ️ NOTE

The use of defer in the Increment() method is redundant.
It’s more useful for long complex functions where errors can occur.
Here I should have just placed the Unlock() call after the c.value++.

Atomic Operations

The sync/atomic package provides low-level atomic operations on simple types like integers and pointers, ensuring operations are performed atomically.

Use atomic operations when you need lock-free synchronization for counters or flags, but only for basic integer or pointer manipulations.

In the following example, atomic.AddInt32 safely increments counter without a lock, making it ideal for high-performance counters or flags:

package main

import (
    "fmt"
    "sync"
    "sync/atomic"
)

func main() {
    var counter int32
    var wg sync.WaitGroup

    for range 10 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            atomic.AddInt32(&counter, 1)
        }()
    }

    wg.Wait()
    
    fmt.Println("Final Counter:", atomic.LoadInt32(&counter))
}

https://play.golang.com/p/qsRPoC4GPNv

Once

Go’s sync.Once ensures that a function only executes once, even if multiple goroutines attempt to run it.

Use sync.Once when you need to perform a one-time initialization, such as setting up a shared resource.

In the following example, even though multiple goroutines call once.Do(initialize), initialize only runs once. This is especially useful for lazy initialization of global resources:

package main

import (
    "fmt"
    "sync"
)

var once sync.Once

func initialize() {
    fmt.Println("Initializing...")
}

func main() {
    var wg sync.WaitGroup
    for range 3 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            once.Do(initialize)
        }()
    }
    wg.Wait()
}

https://play.golang.com/p/5J1ApCPc1iU

Context

Go’s context.Context is not a strict concurrency primitive but is widely used to manage timeouts, cancellations, and deadlines across goroutines.

💡 TIP

You’ve seen context used in the Error Groups example earlier.

Use context.Context to signal cancellation or control the lifespan of goroutines, particularly in networked or long-running tasks.

In the following example, context.WithTimeout creates a context that automatically cancels after 1 second, which is useful for controlling tasks that may hang or take too long:

package main

import (
    "context"
    "fmt"
    "time"
)

func process(ctx context.Context) {
    select {
    case <-time.After(2 * time.Second): // use time.After to simulate slow operation
        fmt.Println("Completed work")
    case <-ctx.Done():
        fmt.Println("Work cancelled")
    }
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
    defer cancel()

    go process(ctx)

    time.Sleep(2 * time.Second)
}

https://play.golang.com/p/diSmAp0SJkg

Map

The sync package has a Map type which you will likely not need to use.

The Go authors even document it as such…

The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.

The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

Conditions

Go’s sync.Cond is probably the most confusing and hard to use of all concurrency tools (hence I’ve left it till last). It’s used for signaling between goroutines. It lets goroutines wait until they are notified to continue, which is useful when one goroutine needs to wait for a certain condition to be met by another goroutine.

Use sync.Cond when you need goroutines to wait for certain conditions, such as producer-consumer scenarios.

In the following example, cond.Wait() blocks until cond.Signal() is called. It’s useful for waiting on complex conditions where other primitives like chan may not be ideal:

❗ IMPORTANT

The call to cond.L.Lock() in the main goroutine just before for !ready is required, otherwise you’ll get the error fatal error: sync: unlock of unlocked mutex. This is because cond.Wait() expects the caller to hold the lock before calling Wait() (see this video for details). Once Wait() returns, it reacquires the lock, ensuring the main goroutine can safely check ready and exit the loop.

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var mu sync.Mutex
    cond := sync.NewCond(&mu)
    ready := false

    go func() {
        time.Sleep(1 * time.Second)
        cond.L.Lock()
        ready = true
        cond.L.Unlock()
        cond.Signal() // Notify one waiting goroutine
    }()

    cond.L.Lock()
    for !ready {
        cond.Wait() // Wait until condition is met
    }
    fmt.Println("Ready is true, proceeding.")
    cond.L.Unlock()
}

https://play.golang.com/p/n_txZaH7lPA

In the following example we have multiple worker goroutines waiting on a shared condition to be “notified.” We’ll see how both .Signal() and .Broadcast() work when notifying waiting goroutines:

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(id int, cond *sync.Cond) {
    cond.L.Lock() // Lock the condition
    defer cond.L.Unlock()

    fmt.Printf("Worker %d is waiting\n", id)
    cond.Wait() // Wait for a signal or broadcast
    fmt.Printf("Worker %d is proceeding\n", id)
}

func main() {
    lock := &sync.Mutex{}
    cond := sync.NewCond(lock)

    // Start multiple worker goroutines that will wait on the condition
    for i := range 3 {
        go worker(i, cond)
    }

    // Allow time for all workers to start and wait
    time.Sleep(1 * time.Second)

    // Use Signal to wake up one goroutine
    fmt.Println("Notifying one worker")
    cond.Signal() // Notifies one waiting worker
    time.Sleep(1 * time.Second)

    // Use Broadcast to wake up all remaining goroutines
    fmt.Println("Broadcasting to all remaining workers")
    cond.Broadcast() // Notifies all remaining waiting workers

    // Allow time for all goroutines to complete
    time.Sleep(2 * time.Second)
    fmt.Println("Main function exiting.")
}

https://play.golang.com/p/41ibtmUmKaN

Each worker goroutine locks the condition, calls cond.Wait(), and then waits. This releases the lock (as we now understand from the earlier IMPORTANT note, see above if you missed it), allowing other goroutines to call Wait() as well.

The cond.Signal() call in the main function wakes up one of the waiting goroutines, allowing it to proceed.

After a short delay, cond.Broadcast() wakes up all remaining waiting goroutines, allowing them to proceed simultaneously.

This is useful for scenarios where multiple tasks need to wait for a common event or state change to proceed.

ℹ️ NOTE

Notifications are not ordered.
Any one of the waiting goroutines can be chosen to proceed first.
Broadcast ensures that all waiting goroutines eventually proceed.

Now you might be thinking “hmm, it looks like I could use channels instead and they’re more idiomatic”.

Well, here are some reasons for why you might need to choose sync.Cond over channels:

Fine-Grained Control: sync.Cond allows precise control over waiting and signaling, suitable for cases where specific conditions must be checked or managed.
Broadcast Capability: Broadcasting to multiple goroutines is straightforward with sync.Cond, whereas channels require individual signaling, which can be inefficient.
Reduced Complexity for State-Based Waiting: sync.Cond is ideal for situations where goroutines need to wait for specific conditions to be true, rather than for individual values or events passed through a channel.
Avoiding Channel Overhead: Channels introduce buffering and management overhead, especially with many goroutines, whereas sync.Cond relies on a shared mutex with a direct wait/notify mechanism, which is often faster.

In summary, sync.Cond is best suited for use cases that involve waiting for and signaling conditions, especially when you need more control over synchronization and when goroutines are reacting to shared state changes rather than discrete message passing.

Real Examples

Here are some real world application code I’ve had to write.

Bulk Deletion API

Below is a ‘real world’ example where we need to delete a bunch of keys from a data store.

The API that is provided, does not support bulk deleting of keys.

The API does provide an endpoint that lets us paginate the available keys, and we then need to stream that information as quickly as possible using a pool of goroutines coordinated with both channels and wait groups.

It’s a nice example because it brings together several different concurrency primitives (goroutines, channels, select, wait groups, atomic operations).

💡 TIP

Keep reading after the code snippet for a brief breakdown of what the code does.

const (
    // PoolSize is the goroutine/thread-pool size.
    // Each pool will take a 'key' from a channel and issue a DELETE request.
    const PoolSize int = 100

    // MaxErrors is the maximum number of errors we'll allow before
    // stopping the goroutines from executing.
    const MaxErrors int = 100
)

func DeleteAllKeys(storeID string, out io.Writer) error {
    // Create a 'spinner' which helps visually update the user on the progress.
    spinnerMessage := "Deleting keys"
    var spinner text.Spinner

    var err error
    spinner, err = text.NewSpinner(out)
    if err != nil {
        return err
    }
    err = spinner.Start()
    if err != nil {
        return err
    }
    spinner.Message(spinnerMessage + "...")

    // Create a keys paginator.
    p := fastly.NewListKVStoreKeysPaginator(&fastly.ListKVStoreKeysInput{
        StoreID: storeID,
    })

    // Channel for tracking errors when deleting keys.
    errorsCh := make(chan string, MaxErrors)
    
    // Channel for tracking keys to delete.
    keysCh := make(chan string, 1000) // this number correlates to the pagination 
                                      // page size defined by the API

    var (
        // Track the number of keys deleted.
        deleteCount atomic.Uint64
        
        // Track which keys failed to be deleted.
        failedKeys []string
        
        // This will help us wait for all goroutines to complete.
        wg sync.WaitGroup
    )

    // We have two separate execution flows happening at once:
    //
    // 1. Pushing keys from pagination data into a key channel.
    // 2. Pulling keys from key channel and issuing API DELETE call.
    //
    // We have a limit on the number of errors. Once that limit is reached we'll
    // stop the second set of goroutines processing the delete operation.

    wg.Add(1)
    go func() {
        defer wg.Done()
        defer close(keysCh)
        for p.Next() {
            for _, key := range p.Keys() {
                keysCh <- key
            }
        }
    }()

    // Limit the number of goroutines spun up to the specified pool size.
    for range PoolSize {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for key := range keysCh {
                err := fastly.DeleteKVStoreKey(
                    &fastly.DeleteKVStoreKeyInput{StoreID: c.StoreID, Key: key},
                )
                if err != nil {
                    select {
                    case errorsCh <- key:
                    default:
                        return // channel is full (i.e. MaxErrors limit reached)
                    }
                }
                // Update the TUI (Terminal UI) to reflect the current number of 
                // deleted keys.
                f := strconv.FormatUint(deleteCount.Add(1), 10)
                spinner.Message(spinnerMessage + "..." + f)
            }
        }()
    }

    wg.Wait()

    close(errorsCh)
    for err := range errorsCh {
        failedKeys = append(failedKeys, err)
    }

    spinnerMessage = "Deleted keys: " + strconv.FormatUint(deleteCount.Load(), 10)

    if len(failedKeys) > 0 {
        spinner.StopFailMessage(spinnerMessage)
        err := spinner.StopFail()
        if err != nil {
            return fmt.Errorf("failed to stop spinner: %w", err)
        }
        return fmt.Errorf("failed to delete %d keys", len(failedKeys))
    }

    spinner.StopMessage(spinnerMessage)
    if err := spinner.Stop(); err != nil {
        return fmt.Errorf("failed to stop spinner: %w", err)
    }

    text.Success(out, "\nDeleted all keys from KV Store '%s'", c.StoreID)
    return nil
}

So you can see we have multiple goroutines spun up (and we wait for them using sync.WaitGroup):

The first goroutine is iterating over the pagination data and pushing data into a channel.
The other goroutines (we have a limit of PoolSize) are pulling data from the channel and issuing key deletion API calls.

We also use the select statement to control whether we stop the goroutines processing the deletion operations. The way we do this is to define another channel (errorsCh) with a buffer size of MaxErrors, and then every time we get an error we push the error into that channel. If the channel becomes full (which it will do eventually because there’s nothing pulling messages from the errorsCh channel), then the select statement will fallthrough to its default block and we’ll return the goroutine (causing it to stop running)

The last interesting concurrency primitive we use is atomic.Uint64 for accurately tracking the number of deleted keys. We use its Add() method within the goroutine to safely increment the counter, and then at the end of the function we use its Load() method to safely extract the final value.

UPDATE 2024.11.01

Below is a modification to the ‘real world’ example code.

The difference is in how errors are handled. In the original code we would stop processing the deletion of keys when an error threshold was reached. But in the following version we don’t want that to happen. Instead we want to keep processing errors, but that introduces a new challenge related to channel buffer size (e.g. we can’t set an infinite amount of memory for a channel) and so at some point the errors channel is going to get filled up and we need to decide what to do.

In this case we can realistically only drop errors. But we can alleviate that a little bit by trying to pull messages out from the errors channel and appending them to the failedKeys slice concurrently. This will make space in the channel’s buffer and so we can push more errors into it if they occur.

The problem with pulling errors out of the errors channel concurrently is we need to range over the channel, but that will cause a deadlock if we don’t handle it correctly (because ranging over a channel only terminates the loop when the channel is closed). So we need to introduce not one sync.WaitGroup but two! See the code comments below for more details…

// deleteAllKVStoreKeys deletes all keys within the specified KV Store.
func deleteAllKVStoreKeys(
    conn *gofastly.Client, 
    storeID string, 
    maxErrors, poolSize int
) error {
    p := conn.NewListKVStoreKeysPaginator(&fastly.ListKVStoreKeysInput{
        StoreID: storeID,
    })

    errorsCh := make(chan string, maxErrors)
    keysCh := make(chan string, 1000) // number correlates to pagination page size

    var (
        failedKeys   []string
        mu           sync.Mutex
        wgProcessing sync.WaitGroup
        wgErrorCh    sync.WaitGroup
    )

    // We have three separate execution flows happening at once:
    //
    // 1. Pushing keys from pagination data into a key channel.
    // 2. Pulling keys from error channel and appending to failedKeys slice.
    // 3. Pulling keys from key channel and issuing API DELETE call.
    //
    // The second item is problematic, in that ranging over a channel only
    // terminates when the channel is closed. So we need to ensure we close the
    // errorsCh once we've finished processing the deletion of all the keys.
    //
    // To do that we need two sets of wait groups.
    //
    // The first is wgProcessing which keeps track of all goroutines related to
    // processing the pagination data (e.g. the goroutine ranging over the
    // paginator keys, and the goroutine ranging over the keysCh as part of the
    // poolSize loop).
    //
    // The second wait group is wgErrorCh which tracks when the first
    // (wgProcessing) has completed and then closes errorsCh.

    // The following goroutine finishes once all pagination keys have been
    // processed.
    wgProcessing.Add(1)
    go func() {
        defer wgProcessing.Done()
        defer close(keysCh)
        for p.Next() {
            for _, key := range p.Keys() {
                keysCh <- key
            }
        }
    }()

    // The following goroutine finishes once the errorsCh is closed.
    wgErrorCh.Add(1)
    go func() {
        defer wgErrorCh.Done()
        for err := range errorsCh {
            mu.Lock()
            failedKeys = append(failedKeys, err)
            mu.Unlock()
        }
    }()

    // The following goroutines close once they've pulled all data from keysCh.
    for i := 1; i <= poolSize; i++ {
        wgProcessing.Add(1)
        go func() {
            defer wgProcessing.Done()
            for key := range keysCh {
                err := conn.DeleteKVStoreKey(
                    &fastly.DeleteKVStoreKeyInput{StoreID: storeID, Key: key},
                )
                if err != nil {
                    select {
                    case errorsCh <- key:
                    default:
                        continue // the larger we make maxErrors 
                                 // the less likely we'll drop errors 
                                 // (obviously there's a memory trade-off to be made)
                    }
                }
            }
        }()
    }

    // The following goroutine is closed once the 'processing' goroutines are
    // finished.
    wgErrorCh.Add(1)
    go func() {
        defer wgErrorCh.Done()
        wgProcessing.Wait() // Wait for all deletion and pagination tasks.
        close(errorsCh)
    }()

    // Wait for the error-handling goroutines to finish processing.
    wgErrorCh.Wait()

    if len(failedKeys) > 0 {
        return fmt.Errorf("failed to delete %d keys", len(failedKeys))
    }

    return nil
}

TLS Certificate asynchronous Pipeline

I ended up needing to write some pretty complex asynchronous code with multiple layers of fan-out requests.

To explain, I was building an API as a thin wrapper around an ACME client for issuing TLS certificates and I have an asynchronous pipeline with multiple stages.

In one stage I have an ‘order’ which contains multiple ‘authorization’ objects. These objects are essentially each domain that needs to be validated (i.e. the server needs to confirm the customer owns the domains).

The server issues multiple challenges for each authorization. There are three challenge types: dns, http, tls. The customer just needs to validate one of them, but we don’t know which one they’ll use so we try them all. Hence we need to fan-out each challenge request concurrently.

We also process the parent authorization (i.e. the domains) concurrently because we don’t want to sequentially process them as that would take a lot longer.

So, as an example, if I (as the customer) had made an API request for a new TLS certificate that I wanted to have for an apex domain (example.com) and a CNAME (www.example.com), then we would:

Have two authorizations (each running concurrently)
Each authorization would have three challenges (each running concurrently)

The code for that looks like the following:

💡 The reason I’m sharing this code is because it uses lots of different concurrency mechanisms to ensure efficient use of memory resources (i.e. we need to ensure goroutines finishes more quickly, and actually get cleaned up completely and not leak memory).

package order

import (
    "context"
    "errors"
    "fmt"
    "io"
    "log/slog"
    "math"
    "net"
    "net/http"
    "slices"
    "strings"
    "sync"
    "time"

    "github.com/mholt/acmez/v3/acme"

    "github.com/fastly/ascerta/internal/httpx"
    "github.com/fastly/ascerta/internal/log"
)

const (
    // backoffMax is the maximum number to backoff by (i.e. 2 hours in minutes).
    backoffMax = 120
    // backoffMultiplier is the multiplier used for exponential backoff.
    backoffMultiplier = 2
    // checkTimeout short-circuits an authorization check if it stalls.
    // We don't expect an individual check to take longer than 5 seconds.
    checkTimeout = time.Duration(5) * time.Second
    // maxRetries is the maximum number of times a message can be re-queued.
    maxRetries = 10
)

var (
    // backoff is the duration to backoff by (in minutes).
    // This is a variable so it can be updated for exponential backoff purposes.
    backoff = 5.0

    // supportedChallengeTypes filters unsupported challenge types.
    supportedChallengeTypes = []string{
        acme.ChallengeTypeDNS01,
        acme.ChallengeTypeHTTP01,
    }
)

// consumerChecks takes new certificate orders from the queue and determines
// what pre-flight checks to perform (avoiding bothering a CA unnecessarily).
func consumerChecks() {
    l := log.New()
    ctx := context.Background()

    for o := range ordersCheck {
        sl := l.With(
            slog.Group("retries",
                slog.Int("current", o.retries),
                slog.Int("max", maxRetries),
            ),
        )
        sl.LogAttrs(ctx, slog.LevelInfo, "consumer_check", slog.String("state", "start"))

        if o.retries > maxRetries {
            sl.LogAttrs(ctx, slog.LevelWarn, "consumer_check", slog.String("state", "dead"))
            continue
        }

        // We have one 'authorization' per SAN list entry (i.e. per domain).
        // We then have multiple 'challenges' per authorization to check.
        //
        // We process each authorization asynchronously.
        // We also process each authorization's challenges asynchronously.

        var wg sync.WaitGroup

        for i, a := range o.authorizations {
            // If an order has been placed back on the queue, then we want to
            // avoid having to re-validate authorizations that were already
            // validated successfully.
            if !a.authorized {
                wg.Add(1)
                go func() {
                    defer wg.Done()
                    processAuthorization(ctx, l, i, &o)
                }()
            }
        }

        wg.Wait()

        // We now need to check if any authorizations failed.
        // If so, we need to re-queue the order.
        var (
            requeue bool
            failed  []string
        )
        for _, a := range o.authorizations {
            if !a.authorized {
                failed = append(failed, a.authorization.IdentifierValue())
                requeue = true
            }
        }
        if requeue {
            sl.LogAttrs(ctx, slog.LevelWarn, "consumer_check",
                slog.Any("err", errors.New("not all authorizations were successful")),
                slog.Bool("retry", true),
                slog.Float64("backoff", backoff),
                slog.String("failed", strings.Join(failed, ",")),
                slog.String("state", "re-queued"),
            )
            o.retries++

            time.Sleep(time.Duration(backoff) * time.Minute) // increases exponentially
            backoff = math.Min(backoff*backoffMultiplier, backoffMax)
            go func() {
                ordersCheck <- o // put the order back on the queue for retry.
            }()
            continue
        }

        sl.LogAttrs(ctx, slog.LevelInfo, "consumer_check",
            slog.Any("order", o),
            slog.String("state", "complete"),
        )
        ordersValidate <- o
    }
}

// processAuthorization processes each authorization concurrently. Each
// challenge within the authorization is also processed concurrently.
func processAuthorization(ctx context.Context, l *slog.Logger, i int, o *Order) {
    a := o.authorizations[i]
    sl := l.With(slog.String("authorization", a.authorization.Identifier.Value))

    // We wait for only one goroutine to complete successfully.
    authorized := make(chan acme.Challenge, 1)

    // The use of a cancellable context is to short-circuit running goroutines.
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    // The WaitGroup allows us to verify that all goroutines exit properly.
    var wg sync.WaitGroup

    for _, c := range a.authorization.Challenges {
        if slices.Contains(supportedChallengeTypes, c.Type) {
            wg.Add(1)
            go func(c acme.Challenge) {
                defer wg.Done()
                if err := challengeCheck(ctx, sl, c); err == nil {
                    // The use of select with a cancellable context allows to
                    // solve two issues:
                    //
                    // 1. Signal goroutines to stop if the channel gets blocked.
                    //    This can happen if both dns and http challenges pass.
                    //
                    // 2. Signal goroutines to stop when one challenge passes.
                    //    Allowing us to short-circuit more quickly.
                    //    And avoids waiting for the checkTimeout below.
                    select {
                    case <-ctx.Done():
                    case authorized <- c:
                    }
                }
            }(c)
        }
    }

    // Block until one of the challenges returns successfully.
    // Otherwise we'll timeout and cause a re-queue.
    select {
    case c := <-authorized:
        cancel()  // short-circuit any still running goroutines.
        wg.Wait() // validate we've cleaned up all goroutines.
        sl.LogAttrs(ctx, slog.LevelInfo, "authorization_check",
            slog.Bool("success", true),
            slog.Int("index", i),
            slog.String("validation_method", c.Type),
        )
        o.authorizations[i].authorized = true
        o.authorizations[i].challengeType = c.Type
    case <-time.After(checkTimeout): // in case no challenges are valid
        cancel()  // short-circuit any still running goroutines.
        wg.Wait() // validate we've cleaned up all goroutines.
        sl.LogAttrs(ctx, slog.LevelWarn, "authorization_check", slog.String("reason", "timeout"))
    }
}

// challengeCheck identifies the challenge type and triggers that check.
func challengeCheck(ctx context.Context, l *slog.Logger, c acme.Challenge) error {
    switch c.Type {
    case acme.ChallengeTypeDNS01:
        return checkDNS(ctx, l, c)
    case acme.ChallengeTypeHTTP01:
        return checkHTTP(ctx, l, c)
    }
    return nil
}

// checkDNS looks up the TXT record to see if it contains the correct value. It
// will return nil if the record exists with the correct value.
func checkDNS(ctx context.Context, l *slog.Logger, c acme.Challenge) error {
    // use a resolver to propagate context
    var r net.Resolver
    txtRecords, err := r.LookupTXT(ctx, c.DNS01TXTRecordName())
    if err != nil {
        err := fmt.Errorf("failed to lookup TXT record: %w", err)
        l.LogAttrs(ctx, slog.LevelWarn, "check_dns", slog.Any("err", err))
        return err
    }
    if slices.Contains(txtRecords, c.DNS01KeyAuthorization()) {
        return nil
    }
    err = fmt.Errorf("record exists but does not contain correct value: %s", strings.Join(txtRecords, ","))
    l.LogAttrs(ctx, slog.LevelWarn, "check_dns", slog.Any("err", err))
    return err
}

// checkHTTP looks up the .well-known/acme-challenge/... path to see if it
// exists at that path location and will return nil if it does.
func checkHTTP(ctx context.Context, l *slog.Logger, c acme.Challenge) error {
    var lastErr error

    for _, protocol := range []string{"http", "https"} {
        endpoint := fmt.Sprintf("%s://%s%s", protocol, c.Identifier.Value, c.HTTP01ResourcePath())
        sl := l.With(slog.String("endpoint", endpoint))

        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            lastErr = fmt.Errorf("failed to create a http.Request for '%s': %w", endpoint, err)
            sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", lastErr))
            continue
        }

        res, err := httpx.Client.Do(req)
        if err != nil {
            // Checking context cancelled allows us to return faster.
            // i.e. we're either already authorized or we timed out.
            if errors.Is(err, context.Canceled) {
                err := fmt.Errorf("failed to send http.Request: %w", ctx.Err())
                sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", err))
                return err
            }
            lastErr = fmt.Errorf("failed to send http.Request: %w", err)
            sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", lastErr))
            continue
        }

        // NOTE: It's unsafe to defer inside a loop so I manually close res.Body
        if res.StatusCode != http.StatusOK {
            res.Body.Close()
            lastErr = fmt.Errorf("invalid http status '%s'", res.Status)
            sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", lastErr))
            continue
        }

        body, err := io.ReadAll(res.Body)
        if err != nil {
            res.Body.Close()
            lastErr = fmt.Errorf("failed to read response body: %w", err)
            sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", lastErr))
            continue
        }
        res.Body.Close()

        if string(body) != c.KeyAuthorization {
            lastErr = fmt.Errorf("invalid response body '%s': expected '%s'", body, c.KeyAuthorization)
            sl.LogAttrs(ctx, slog.LevelWarn, "check_http", slog.Any("err", lastErr))
            continue
        }

        return nil // success
    }

    if lastErr != nil {
        return lastErr
    }

    // WARNING: We should never reach this part of the code.
    // We should have already returned a success or tracked an error.
    err := errors.New("unexpected code path reached")
    l.LogAttrs(ctx, slog.LevelError, "check_http", slog.Any("err", err))
    return err
}

The processAuthorization function specifically might look over engineered but it’s critical that we don’t have goroutines hanging around waiting to be unblocked. So all these concurrency mechanisms are in place to ensure the code is as robust as possible.

Reference Material

Bitwise Operations in Go

Sun, 10 Nov 2024 00:00:00 +0000

In the below Go file we use bitwise operators to manipulate individual flags (on/off switches) in a single integer, where each bit position represents a different status.

First we’ll look at the code, and then we’ll explain how it works:

package main

import (
    "fmt"
    "strings"
)

// Define bit flags as constants, where each status is represented by a unique bit position
const (
    StatusActive   = 1 << iota // 1 << 0 which is 0001 (binary)
    StatusAdmin                // 1 << 1 which is 0010
    StatusBanned               // 1 << 2 which is 0100
    StatusVerified             // 1 << 3 which is 1000
)

// Stringify statuses for easier output
func stringifyStatus(status int) string {
    statuses := []string{}

    if status&StatusActive != 0 {
        statuses = append(statuses, "Active")
    }
    if status&StatusAdmin != 0 {
        statuses = append(statuses, "Admin")
    }
    if status&StatusBanned != 0 {
        statuses = append(statuses, "Banned")
    }
    if status&StatusVerified != 0 {
        statuses = append(statuses, "Verified")
    }

    return strings.Join(statuses, ", ")
}

func main() {
    // Let's create a user status and use bitwise OR to combine different flags

    var userStatus int

    // Set the user as active and verified
    userStatus |= StatusActive | StatusVerified
    fmt.Println("Initial Status:", stringifyStatus(userStatus))

    // Add "Admin" status
    userStatus |= StatusAdmin
    fmt.Println("After adding Admin:", stringifyStatus(userStatus))

    // Remove "Verified" status using bitwise AND with NOT
    userStatus &^= StatusVerified
    fmt.Println("After removing Verified:", stringifyStatus(userStatus))

    // Add "Banned" status
    userStatus |= StatusBanned
    fmt.Println("After adding Banned:", stringifyStatus(userStatus))

    // Check if the user is an admin
    if userStatus&StatusAdmin != 0 {
        fmt.Println("User is an admin.")
    } else {
        fmt.Println("User is NOT an admin.")
    }

    // Remove "Admin" status using bitwise AND with NOT
    userStatus &^= StatusAdmin
    fmt.Println("After removing Admin:", stringifyStatus(userStatus))

    // Check if the user is banned
    if userStatus&StatusBanned != 0 {
        fmt.Println("User is banned.")
    } else {
        fmt.Println("User is NOT banned.")
    }

    // Check if the user is an admin
    if userStatus&StatusAdmin != 0 {
        fmt.Println("User is an admin.")
    } else {
        fmt.Println("User is NOT an admin.")
    }
}

Visualising Bits

In case you need a reminder of a what bit alignment and shifting look like:

Defining Bit Flags

Each status is assigned a unique power of 2 using bit shifting (1 << iota).

This ensures each flag only affects a single bit:

StatusActive has the binary value 0001 (1 << 0 == 1 in decimal).
StatusAdmin has the binary value 0010 (1 << 1 == 2 in decimal).
StatusBanned has the binary value 0100 (1 << 2 == 4 in decimal).
StatusVerified has the binary value 1000 (1 << 3 == 8 in decimal).

Setting Statuses

The following example combines two separate status flags:

userStatus |= StatusActive | StatusVerified

In binary, this results in 1001 (or 9 in decimal), which means both the “active” and “verified” flags are set.

Adding and Removing Flags

The following example sets the “admin” bit without affecting the other bits, resulting in 1011 (11 in decimal):

userStatus |= StatusAdmin

Comparing Statuses

Once userStatus has combined flags we can use the & operator to perform a bitwise AND operation, which means it compares each bit of two integers. For each bit position, if both bits are 1, the result at that position will be 1; otherwise, it will be 0.

So what happens when we compare userStatus&StatusAdmin != 0?

Well, StatusAdmin is a bit flag defined as 1 << 1, which results in 0010 in binary. This means that StatusAdmin occupies the second bit position in the binary representation of an integer. When we do userStatus&StatusAdmin, we’re effectively “masking” all bits except for the one represented by StatusAdmin (this is known as bit masking).

When we perform userStatus&StatusAdmin, we get a result where only the bit corresponding to StatusAdmin remains (and is set to 1 if that bit was already set in userStatus). If this result is non-zero (!= 0), it means the StatusAdmin bit is set in userStatus. If it’s zero, then StatusAdmin is not set in userStatus.

If we look at the code in bitwise.go we’ll see userStatus is initially set to include StatusActive and StatusVerified, so userStatus is 1001 in binary (which is 9 in decimal). Remember StatusActive occupied the first bit position (0001), while StatusVerified occupied the fourth bit position (1000) so if setting both flags we get the combined 1001.

Next, we add the StatusAdmin flag with userStatus |= StatusAdmin, making userStatus now 1011 in binary (which is 11 in decimal). When we check if StatusAdmin is set using userStatus&StatusAdmin != 0 we get back 2 from userStatus&StatusAdmin (which is 0010 in binary) because we’ve bit masked the other bits that might have been turned on (if you recall, using & turns each bit to zero except for those bits that were 1 in both numbers being compared), in order to reveal whether the StatusAdmin bit was set on or not (i.e. 0 != 2 so we know this person is an admin).

Programming at the edge with Fastly Compute

Wed, 12 Jun 2024 00:00:00 +0000

So you’ve heard about computing at the edge, and you’ve heard that Fastly let’s you run JavaScript, Go, Rust and any other language that compiles to Wasm at the edge… well, let’s take a look and while we’re at it let’s try and understand how caching works there too.

Getting started

OK, first thing you’re going to need is a fastly account so follow the link and sign up for FREE (for more details see here).

ℹ️ NOTE

Fastly has recently updated its UI to make creating a new Compute service even easier by providing a step through wizard-style experience to set up and configure a new service (go check it out).

Next you’re going to need the Fastly CLI to make interacting with Fastly’s services and products much more efficient, so again, follow the link and get it installed, or you can clone the public repo and run make install.

When running the CLI you’ll need an API token so you can either generate a token via the Fastly UI and then copy/paste it into the CLI using the following command (where it will prompt you to enter the token):

fastly profile create

Or you can use SSO (Single Sign-On) to automatically generate a short-lived token that is automatically assigned to the CLI profile (and the token will be refreshed automatically when it expires :chef-kiss:):

fastly profile create --sso

To make sure the CLI is set up correctly, run the following command:

fastly whoami

Create a new Compute project

A Compute project is a directory that contains your service/application code + some configuration files required by the CLI to build a Compute ‘package’.

A Compute package is a .tar.gz file containing a main.wasm binary and a fastly.toml manifest file. The CLI handles the creation of the package, and even the initial service/application code generation.

To create a new package you can run the following command, which will prompt you for all the information, such as what language you want to use and which Fastly starter kit should be used as the basis for your application code:

fastly compute init

Or if you know what language you want to use, then you can avoid all the prompts and let the CLI choose the ‘default’ starter kit using the following command:

fastly compute init --language go --non-interactive

The above command selects Go as the language I want to use for building my compute service.

The init subcommand is going to generate the following files:

.
├── README.md
├── fastly.toml
├── go.mod
├── go.sum
└── main.go

The only file here that will likely need explaining is the fastly.toml manifest file:

authors = [""]
cloned_from = "https://github.com/fastly/compute-starter-kit-go-default"
description = ""
language = "go"
manifest_version = 3
name = "example"
service_id = ""

[scripts]
  build = "go build -o bin/main.wasm ."
  env_vars = ["GOARCH=wasm", "GOOS=wasip1"]
  post_init = "go get github.com/fastly/compute-sdk-go@latest"

Compute packages are configured using this fastly.toml file in the root of the project directory tree. This file specifies configuration metadata related to a variety of tasks:

Attribution of the package (e.g., name, author)
Information required by the CLI to compile and upload it to a compatible Fastly service
Configuration of local server environments
Bootstrapping of service configuration

I’ll explain more, but for now let’s run our project locally so we can see it working…

Run your code locally

At this point we can actually run our project locally without even having to deploy it!

Running the following command, will compile your project, create a package, and pass the package to a local server environment called Viceroy which runs your package and exposes a local web server for you to interact with:

fastly compute serve

ℹ️ NOTE

If you want to iteratively develop your application then you can pass the --watch flag to have the CLI monitor your files for changes and to hot reload your application.

[!TIP] If you want to configure files to be ignored then create a .fastlyignore file. It works like .gitignore so should be familiar to you.

You should see CLI output that looks a bit like the following:

$ fastly compute serve

✓ Verifying fastly.toml
✓ Identifying package name
✓ Identifying toolchain
✓ Running [scripts.build]
✓ Creating package archive

SUCCESS: Built package (pkg/example.tar.gz)

✓ Running local server

INFO: Command output:
--------------------------------------------------------------------------------
2024-06-12T09:51:25.191538Z  WARN no backend definitions found in /private/tmp/example/fastly.toml
2024-06-12T09:51:25.191658Z  INFO Listening on http://127.0.0.1:7676

Opening http://127.0.0.1:7676 in your web browser should show the result of running the Go default starter kit code, which generates an <iframe> loading a Fastly hosted welcome page.

Sending requests to an origin server

OK, so let’s open up the main.go and delete all the code and replace it with the following:

package main

import (
    "context"
    "fmt"
    "io"
    "time"

    "github.com/fastly/compute-sdk-go/fsthttp"
)

func main() {
    fsthttp.ServeFunc(func(ctx context.Context, w fsthttp.ResponseWriter, r *fsthttp.Request) {
        r.Header.Add("TheTime", time.Now().String())
        r.CacheOptions.TTL = 30 

        resp, err := r.Send(ctx, "httpme")
        if err != nil {
            w.WriteHeader(fsthttp.StatusBadGateway)
            fmt.Fprintln(w, err.Error())
            return
        }

        resp.Header.Set("Cache-Control", "public, s-maxage=86400")
        w.Header().Reset(resp.Header)
        w.WriteHeader(resp.StatusCode)
        io.Copy(w, resp.Body)
    })
}

The fsthttp.ServeFunc essentially sets up a listener for incoming requests and the given function is the request handler. Inside the request handler we do the following things:

Add a HTTP header to the incoming request object:
```
r.Header.Add("TheTime", time.Now().String())
```
Override the HTTP caching behaviour of the response (I’ll come back to this):
```
r.CacheOptions.TTL = 30
```
Forward the request to a ‘backend’ called httpme (again, we’ll come back to this):
```
r.Send(ctx, "httpme")
```
Set (and possibly override) the Cache-Control HTTP header on the response:
```
resp.Header.Set("Cache-Control", "public, max-age=60")
```
Copy all of the response headers from the backend to the user’s response:
```
w.Header().Reset(resp.Header)
```
Send an appropriate response status code:
```
w.WriteHeader(resp.StatusCode)
```
Start streaming the backend response body to the user:
```
io.Copy(w, resp.Body)
```

OK, so there were two things we need to get a better grip on:

Fastly backends
Fastly caching behaviour

Let’s get into it…

Fastly backends

A backend is something that needs to be defined explicitly, it’s an origin server owned and managed by you. For example, you might have an API service that you want your Compute service/application to communicate with to get data.

To define a backend we need to first create one within our Compute service. We can do this in many ways (e.g. the Fastly UI, the Fastly API or the Fastly CLI).

For simplicity we’re going to use the fastly.toml manifest file to configure a backend, that will be created when we come to deploy our Compute service/application for the first time. We’ll also use the manifest file to configure a backend that will be used when running our Compute service/application locally, because we might want to use a mock backend locally (although for our case we’ll use the same production backend).

Add the following to your fastly.toml manifest file:

[setup.backends.httpme]
address = "http-me.glitch.me"
port = 443

[local_server.backends.httpme]
override_host = "http-me.glitch.me"
url = "https://http-me.glitch.me"

The first block [setup.backends.httpme] is used only once by the CLI when you come to deploy your service (I’ll show you that later). It tells the CLI, when it’s creating your Compute service, to also create a backend called httpme and to make sure it has the specified address and port.

The second block [local_server.backends.httpme] is used by Viceroy when running your Compute service/application locally. Whenever your code uses the Send method on a fsthttp.Request, it will ensure the request is forwarded to the specified backend, which in this case is the real service https://http-me.glitch.me.

So in our application code where we have r.Send("httpme") you can now see that this is sending the request to a ‘backend’ object called httpme which we’ve now defined/created indirectly via the fastly.toml manifest file.

Now, defining backends is a bit tedious, and the primary reason for this design is security. That said, if you’re willing to forego security, then you can ask use something called a ‘dynamic backend’ which allows you to define backends at runtime in your code (see here for an example).

Fastly caching behaviour

Now let’s look at Fastly’s caching behaviour as it’s a bit confusing. I’m going to reference the official Fastly doc Caching content with Fastly:

The Fastly edge cache is an enormous pool of storage across our network which allows you to satisfy end user requests with exceptional performance and reduce the need for requests to your origin servers. Most use cases make use of the readthrough cache interface, which works automatically with the HTTP requests that transit your Fastly service to save responses in cache so they can be reused. The first time a cacheable resource is requested at a particular POP, the resource will be requested from your backend server and stored in cache automatically. Subsequent requests for that resource can then be satisfied from cache without having to be forwarded to your servers.

So what this means is that Fastly uses something called a ‘readthrough’ cache by default. The readthrough cache respects HTTP caching semantics. So if your backend returns a response with a HTTP header such as:

Cache-Control: s-maxage=300, max-age=0

…then the readthrough cache will cache the response for 300s (i.e. it’ll use the value assigned to s-maxage). So when the next request comes into your Compute service/application, and it reaches the r.Send() line, then that will respond immediately with the cached content and not actually make a call to the backend.

But what about the max-age? Well, that’s for the browser. The user’s web browser will only see Cache-Control: max-age=0 as s-maxage is for proxies and CDNs and so Fastly will strip it from the response header.

ℹ️ NOTE

If you want more information on HTTP caching then take a look at my blog post HTTP Caching Guide which explains all the details.

Now, this is where the following lines in our Compute service/application code are going to affect things. Let’s take a look…

So before we forwarded the incoming request to the backend (using Send) we actually configured the readthrough to ignore what is being set in the backend’s response (i.e. ignoring the Cache-Control header) and to blanket cache the response for 30 seconds. We do this using:

// https://pkg.go.dev/github.com/fastly/compute-sdk-go@v1.3.1/fsthttp#CacheOptions
r.CacheOptions.TTL = 30

The reason we have to do this before sending the request is because of how the readthrough cache works (i.e. it automatically handles caching the responses).

ℹ️ NOTE

Fastly also provides a ‘simple’ cache interface and a more advanced ‘core’ cache interface exposed via the Compute SDKs. You can see examples here.

Before we send our response to the user we actually modify the cache behaviour again, but this time for the user’s web browser by overriding the backend’s max-age=0 with max-age=60:

resp.Header.Set("Cache-Control", "public, max-age=60")

⚠️ WARNING

There’s a caveat to the readthrough cache that you’ll want to be careful with. If your backend sends Cache-Control: private, then understandably the readthrough cache will not cache the response because your backend has defined that behaviour. In this case, setting r.CacheOptions.TTL will have NO EFFECT. It’s only usable for responses that Fastly considers cacheable and private is not cacheable. You would need your backend to change the Cache-Control value to be something cacheable if you wanted r.CacheOptions.TTL to have any kind of effect.

Deploying your Compute service

To wrap up this post, let’s get our Compute service/application deployed:

fastly compute publish

The above command will prompt you whenever information is required, or you can simplify the process and add the --non-interactive flag to let Fastly choose default values for everything:

fastly compute publish --non-interactive

For the first time deploying you may find it takes a bit of time because Fastly is uploading your package across its global fleet of servers. For me I’ve noticed the first deploy takes around ~30s but after that, any further changes I make to my application is almost immediately uploaded/replicated 🎉

You should see output similar to the following:

$ fastly compute publish --non-interactive

✓ Verifying fastly.toml
✓ Identifying package name
✓ Identifying toolchain
✓ Running [scripts.build]
✓ Creating package archive

SUCCESS: Built package (pkg/example.tar.gz)

✓ Verifying fastly.toml
✓ Creating service
✓ Creating domain 'regularly-living-cheetah.edgecompute.app'
✓ Uploading package
✓ Activating service (version 1)
✓ Checking service availability (status: 200)

Manage this service at:
    https://manage.fastly.com/configure/services/IuZqijThLa4VawBcGy7ba6

View this service at:
    https://regularly-living-cheetah.edgecompute.app

SUCCESS: Deployed package (service IuZqijThLa4VawBcGy7ba6, version 1)

Good luck, and I hope you enjoy programming at the edge with Fastly 🙂

Go Typed Nil

Wed, 12 Jun 2024 00:00:00 +0000

The following code doesn’t do what you might expect:

package main

import "fmt"

func main() {
    var i *impl

    fmt.Println("i == nil:", i == nil)
    what(i)
}

type impl struct{}

func (i *impl) do() {}

func what(i interface{ do() }) {
    fmt.Println("i == nil:", i == nil)
}

If you expected the what function to print i == nil: true, then keep reading…

Typed Nils

The behavior observed is due to the way interfaces and nil values interact in Go. To understand why the what function is able to see the i argument as non-nil, we need to dig into the details of how Go handles interface values.

Interface Values: In Go, an interface value is a tuple of a type and a value. An interface value is nil only if both the type and the value are nil.
Concrete vs Interface nil: When you assign a concrete type value (which happens to be nil) to an interface, the interface itself is not nil. This is because the interface value now contains a type (the concrete type) and a value (nil).

Explanation

Declaring var i *impl initializes it to nil.
- i is a pointer to impl and is initialized to nil.
Printing i == nil in main is true
- because i is a nil pointer to impl
Calling what(i) function:
- the function what takes an argument of type interface{ do() }
Inside what function:
- i (which is nil) is passed to what
- it is assigned to the parameter i of type interface{ do() }
Interface value construction:
The value of i inside what is now an interface that holds:
- Type: *impl (the concrete type of the value passed in)
- Value: nil (the value of the concrete type)
Checking i == nil prints false because the interface i is not nil:
- The type part of the interface is *impl.
- The value part of the interface is nil.

Summary

The what function sees the i argument as non-nil because, in Go, an interface holding a nil pointer is not itself nil. The interface contains type information (*impl) and a nil value. Therefore, when the code checks if i is nil, it returns false since the type information (*impl) is present.

Reference material

Continuous Integration and Deployment to multiple environments with Terraform Cloud and GitHub Actions

Sun, 10 Mar 2024 00:00:00 +0000

Summary

In this post I cover what CI/CD means and how you can achieve it across multiple different environments (e.g. dev, stage, prod) using the GitHub Actions platform. The infrastructure is managed using Terraform and the state is coordinated using Terraform Cloud.

I detail specific issues I experienced while implementing CI/CD for a real high-scale/high-profile public API, and the workarounds I needed to implement to resolve them.

What is CI/CD?

Let’s start off by answering the question:
What is CI/CD?

CI is short for Continuous Integration.
CD refers to two separate concepts:

Continuous Delivery
Continuous Deployment

People often make the mistake of thinking they are interchangeable.
They’re not. But what these terms mean is actually not that complicated:

Continuous Integration: you’re testing every change (i.e. commit)
Continuous Delivery: you’re able to ship every change on demand
Continuous Deployment: you’re shipping every change automatically

Now we have a bit of an understanding of what CI/CD is,
let’s see how I’ve been able to use each of these individual concepts.

API Gateway

I needed to implement a complete CI/CD pipeline for my employer.

They had an API Gateway service which required a mixture of continuous integration, delivery and deployment across three separate ‘environments’ (i.e. a dev environment, a stage environment, and a production environment).

Here is the summary of their usage per environment:

Dev: Continuous Integration/Deployment
Stage: Continuous Integration/Deployment
Production: Continuous Delivery

Let’s dig into each environment to understand how they use CI/CD.

Dev Environment

The development environment is spun-up automatically when pushing to a PR.
It is automatically spun-down when the associated PR is merged.

ℹ️ NOTE

We control which files will trigger the dev environment creation.

The service name is api-gateway-dev-<BRANCH>.
The service domain is api-gateway-dev-<BRANCH>.edgecompute.app.
The <BRANCH> is calculated from github.head_ref.

The Terraform state that manages this environment is stored in a Terraform Cloud (TFC) project under a dynamically created, and managed, workspace called api-gateway-dev-<BRANCH>.

The dev CI workflow has some important steps it must take in order to spin-up a development environment that is isolated between different PR authors:

Call the TFC API to create a new workspace in the TFC project.
Cache the result of the workspace creation to avoid API rate limits.
Also create a workspace environment variable for FASTLY_API_KEY.

The last step is required as we’re deploying the service to Fastly.

ℹ️ NOTE

You’ll notice shortly that our stage and production flows use a single service and workspace, where as our dev environment is a bit more involved and requires its own dynamically created workspace and services. This is because we have multiple developers working on multiple features at any one time. So we use the ‘branch’ name as the unique identifier, and by having a different workspace per branch it means a developer can modify the Terraform infrastructure without affecting anyone else.

Stage Environment

There are two scenarios where the staging environment is deployed to:

Whenever a PR is merged
Whenever a commit is pushed directly to the main branch

ℹ️ NOTE

Again, we control which files will trigger the environment creation.

The service name is api-gateway-stg.
The service domain is api-gateway-stg.edgecompute.app.

Once the deployment is complete, the Compute package (i.e. the application code)
is uploaded as an artifact to GitHub and is deleted after 14 days.

The name of the artifact is the hash of the Compute package.
A git tag is pushed for each stage deployment, and the hash is appended to it.

During that 14 day time frame, the artifact can be promoted to production.

The Terraform state that manages this environment is stored in the Terraform Cloud project under the “api-gateway-stage” workspace.

Production Environment

The production environment is deployed to manually (i.e. Continuous Delivery) by triggering the relevant GitHub Actions workflow. The person triggering the deployment will first need to identify the Compute package ‘hash’ (which is appended to each staging tag).

The reason for ‘promoting’ the staging Compute package to production,
rather than recompiling the application code, is to ensure consistency
between the staging and production environments.

We require the person triggering the deployment to manually locate the relevant Compute package hash. We can’t rely on automation (e.g git tag --sort=-creatordate) because at the moment of checking the git tags, another person might have merged another PR and thus we would pick up that new staging tag and deploy the wrong artifact to the production environment.

The reason that is a concern, is because a newly merged PR should have some time
on the staging environment for any unexpected bugs to be identified. At some
point in the future we might have complete confidence in our test suite to move
from Continuous Delivery to Continuous Deployment, but that time isn’t now.

The service name is api-gateway-prd.
The service domain is api-gateway-prd.edgecompute.app.

The Terraform state that manages this environment is stored in the Terraform Cloud project under the “api-gateway-production” workspace.

Implementation

So here is where I’m going to display the complete set of GitHub Actions workflow configuration. We’ll review the code and I’ll make comments about specific sections of interest.

Let’s start with dev, then we’ll see stage, and then production.

Dev Config

# We deploy to dev environment when PR is pushed to.
# We name the environment after the branch to avoid collisions.
# When PR is merged we automatically tear-down the dev environment.
name: "API Gateway TF Dev"

# Stop any in-flight CI jobs when a new commit is pushed.
concurrency:
  group: api-gateway-dev-${{ github.ref_name }}
  cancel-in-progress: true

on:
  workflow_dispatch:
  pull_request:
    types: [opened, edited, closed, reopened, synchronize]
    paths:
      - .github/workflows/api-gateway-*
      - cmd/gateway/**
      - internal/gateway/**
      - infrastructure/fastly/compute/api-gateway/**

# IMPORTANT: for TF_VAR_* wrap double-quotes in single-quotes.
env:
  APP_DIRECTORY: "./cmd/gateway"
  CONFIG_DIRECTORY: "./infrastructure/fastly/compute/api-gateway"
  PACKAGE_PATH: "/pkg/gateway.tar.gz"
  TF_API_TOKEN: "${{ secrets.TF_API_TOKEN }}"
  TF_CLOUD_ORGANIZATION: "example"
  TF_VAR_backend: '"https://api-dev.example.com"'
  TF_VAR_env: '"dev"'

jobs:
  create-workspace:
    name: "Create Terraform Cloud Workspace"
    # The following if statement says: don't run this when the PR is closed
    if: ${{ github.event.action != 'closed' && github.actor != 'dependabot[bot]' }} 
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      - name: Cache/Restore Workspace ID
        id: cache-workspace-id
        uses: actions/cache@v4
        with:
          path: tfc-workspace-id
          key: ${{ runner.os }}-cache-workspace-id-${{github.head_ref}}

      # IMPORTANT: We don't want to run the following steps if we get a cache-hit.
      # That is because a cache-hit indiciates we've already created the workspace.
      # This means we're forced to add a `if:` check on each following step.

      - name: Set TF_VAR_branch
        if: steps.cache-workspace-id.outputs.cache-hit != 'true'
        uses: ./.github/actions/tf-var-branch

      - name: Set WORKSPACE_NAME
        if: steps.cache-workspace-id.outputs.cache-hit != 'true'
        uses: ./.github/actions/tfc-dev-workspace-name

      - name: Modify Workspace JSON Payload
        if: steps.cache-workspace-id.outputs.cache-hit != 'true'
        run: |
          payload=./infrastructure/fastly/compute/api-gateway/tfc-api/payloads/workspace.json
          jq '.data.attributes.name = "${{ env.WORKSPACE_NAME }}"' \
            "$payload" > temp.json && mv temp.json "$payload"

      - name: Modify Workspace Variable JSON Payload
        if: steps.cache-workspace-id.outputs.cache-hit != 'true'
        run: |
          payload=./infrastructure/fastly/compute/api-gateway/tfc-api/payloads/workspace-variables.json
          jq '.data.attributes.value = "${{ secrets.FASTLY_API_KEY }}"' \
            "$payload" > temp.json && mv temp.json "$payload"

      # NOTE: A cache is volatile/ephemeral.
      # If the cache is invalidated/cleared, then the following step could still
      # be run. This would mean the curl request to create the workspace would fail
      # (because it could already exist). This is OK, because we check the API
      # response for errors and the last step reacts to the failure status. The
      # intention is to avoid trying to read from a response.json that doesn't contain
      # the required structure (i.e. doesn't contain a .data.id field). As well as
      # avoid trying to call the TFC API to create a workspace variable when there was
      # an error initially creating the workspace.

      - name: Create Dev Workspace
        if: steps.cache-workspace-id.outputs.cache-hit != 'true'
        run: |
          curl -s \
            --header "Authorization: Bearer ${{ secrets.TF_API_TOKEN }}" \
            --header "Content-Type: application/vnd.api+json" \
            --request POST \
            --data @infra/fastly/compute/api-gateway/tfc-api/payloads/workspace.json \
            https://app.terraform.io/api/v2/organizations/example/workspaces > response.json
          if [[ $(jq '.errors | length == 0' response.json) == false ]]; then
            jq '.errors' response.json
            echo "STEP_STATUS=failed" >> "${GITHUB_ENV}"
          fi

      - name: Create FASTLY_API_KEY as Workspace Variable
        if: ${{ steps.cache-workspace-id.outputs.cache-hit != 'true' && env.STEP_STATUS != 'failed' }}
        run: |
          workspace_id=$(jq -r '.data.id' response.json)
          curl -s \
            --header "Authorization: Bearer ${{ secrets.TF_API_TOKEN }}" \
            --header "Content-Type: application/vnd.api+json" \
            --request POST \
            --data @infra/fastly/compute/api-gateway/tfc-api/payloads/variables.json \
            "https://app.terraform.io/api/v2/workspaces/$workspace_id/vars"
          echo "$workspace_id" > tfc-workspace-id

  terraform-plan:
    name: "Plan"
    runs-on: ubuntu-latest
    permissions: # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
      contents: read
      pull-requests: write
    needs: create-workspace
    # don't run this when the PR is closed (and also only if the create-workspace was successful)
    if: ${{ github.event.action != 'closed' && github.actor != 'dependabot[bot]' && success() }} 
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      - name: Install Go and Fastly CLI
        uses: ./.github/actions/fastly-cli

      - name: Build Compute Package
        run: fastly compute build --dir ${{ env.APP_DIRECTORY }} --metadata-show

      - name: Calculate Compute Package Hash
        run: fastly compute hash-files --skip-build --dir ${{ env.APP_DIRECTORY }}

      - name: Copy Compute Package to Terraform workspace
        run: cp ${{ env.APP_DIRECTORY }}${{ env.PACKAGE_PATH }} ${{ env.CONFIG_DIRECTORY }}

      - name: Set TF_VAR_branch
        uses: ./.github/actions/tf-var-branch

      - name: Set WORKSPACE_NAME
        uses: ./.github/actions/tfc-dev-workspace-name

        # NOTE: This job (and all jobs that follow) requires a TFC Workspace.
        # The name of the workspace is expected to match WORKSPACE_NAME.
        # The creation of the workspace happens in the `create-workspace` job.
        #
        # But there are scenarios where the workspace might not exist!
        # This can happen when there is an error in the `create-workspace` job.
        # For example, when calling the TFC API.
        #
        # Although the API might error, we don't want to mark the job as failed.
        # This is because the API could be called and fail for multiple reasons.
        # e.g. the cached API result wasn't found and we got a 422 from the API.
        #
        # A 422 code response from the TFC API thus indicates we tried to create
        # a workspace that already exists. So in this scenario we still want the
        # remaining jobs to run (as the workspace does exist).
        #
        # Ultimately, if the error was something else and the workspace thus
        # wasn't created, then at worst we'll get an error back from TFC when
        # trying to upload the TF configuration.

      - name: Upload Configuration
        uses: hashicorp/tfc-workflows-github/actions/upload-configuration@v1.2.0
        id: plan-upload
        with:
          workspace: ${{ env.WORKSPACE_NAME }}
          directory: ${{ env.CONFIG_DIRECTORY }}
          speculative: true

      - name: Create Plan Run
        uses: hashicorp/tfc-workflows-github/actions/create-run@v1.2.0
        id: plan-run
        with:
          workspace: ${{ env.WORKSPACE_NAME }}
          configuration_version: ${{ steps.plan-upload.outputs.configuration_version_id }}
          plan_only: true

      - name: Get Plan Output
        uses: hashicorp/tfc-workflows-github/actions/plan-output@v1.2.0
        id: plan-output
        with:
          plan: ${{ fromJSON(steps.plan-run.outputs.payload).data.relationships.plan.data.id }}

      - name: Update PR
        uses: actions/github-script@v7
        id: plan-comment
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            // Retrieve existing bot comments for the PR
            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
            });
            const botComment = comments.find(comment => {
              return comment.user.type === 'Bot' && comment.body.includes('TF Plan Output')
            });
            const output = `#### Terraform Cloud Plan Output
               \`\`\`
               Plan: ${{ steps.plan-output.outputs.add }} to add, 
               ${{ steps.plan-output.outputs.change }} to change, \
               ${{ steps.plan-output.outputs.destroy }} to destroy.
               \`\`\`
               [Terraform Cloud Plan](${{ steps.plan-run.outputs.run_link }})
               `;
            if (botComment) {
              github.rest.issues.deleteComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                comment_id: botComment.id,
              });
            }
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            });

  terraform-apply:
    name: "Apply"
    needs: terraform-plan
    runs-on: ubuntu-latest
    permissions:
      contents: read  # for Terraform Actions
      statuses: write # for Status Action (myrotvorets/set-commit-status-action)
    # don't run this when the PR is closed (and also only if the terraform-plan was successful)
    if: ${{ github.event.action != 'closed' && github.actor != 'dependabot[bot]' && success() }} 
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      - name: Install Go and Fastly CLI
        uses: ./.github/actions/fastly-cli

      - name: Build Compute Package
        run: fastly compute build --dir ${{ env.APP_DIRECTORY }} --metadata-show

      - name: Calculate Compute Package Hash
        run: fastly compute hash-files --skip-build --dir ${{ env.APP_DIRECTORY }}

      - name: Copy Compute Package to Terraform workspace
        run: cp ${{ env.APP_DIRECTORY }}${{ env.PACKAGE_PATH }} ${{ env.CONFIG_DIRECTORY }}

      - name: Set TF_VAR_branch
        uses: ./.github/actions/tf-var-branch

      - name: Set WORKSPACE_NAME
        uses: ./.github/actions/tfc-dev-workspace-name

      - name: Terraform Apply
        id: tf-apply
        uses: ./.github/actions/tf-apply
        with:
          workspace: ${{ env.WORKSPACE_NAME }}

      - name: "Terraform Apply Run URL"
        uses: myrotvorets/set-commit-status-action@master
        if: success()
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          status: "success"
          description: "click 'Details' to view"
          context: Terraform Apply Results
          targetUrl: "https://app.terraform.io/app/example/workspaces/\
            api-gateway/runs/${{ steps.tf-apply.outputs.run_id }}"

  # Remove any dev services once the corresponding PR has been merged.
  # https://docs.github.com/en/actions/using-workflows/\
  # events-that-trigger-workflows#running-your-pull_request-workflow-when-a-pull-request-merges
  terraform-destroy:
    name: "Cleanup Dev Environment"
    runs-on: ubuntu-latest
    if: ${{ github.event.pull_request.merged == true && github.actor != 'dependabot[bot]' }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      # NOTE: We still need to install the CLI and compile the Compute package.
      # This is because we need to do a two-step apply (see IMPORTANT below).

      - name: Install Go and Fastly CLI
        uses: ./.github/actions/fastly-cli

      - name: Build Compute Package
        run: fastly compute build --dir ${{ env.APP_DIRECTORY }} --metadata-show

      - name: Calculate Compute Package Hash
        run: fastly compute hash-files --skip-build --dir ${{ env.APP_DIRECTORY }}

      - name: Copy Compute Package to Terraform workspace
        run: cp ${{ env.APP_DIRECTORY }}${{ env.PACKAGE_PATH }} ${{ env.CONFIG_DIRECTORY }}

      - name: Set TF_VAR_branch
        uses: ./.github/actions/tf-var-branch

      - name: Set WORKSPACE_NAME
        uses: ./.github/actions/tfc-dev-workspace-name

        # IMPORTANT: Deleting a service with a resource-link is a two-step apply.
        # We can't pass a -target to the Terraform GH Action as a prerequisite step.
        # So this means we'll need to clear any `resource_link` blocks from the config.
        # Then, after we've applied the change, we'll need to run a TF destroy.

      - name: Remove resource_link blocks from TF config
        run: |
          while IFS= read -r line; do
              if [[ "$line" =~ "resource_link {" || "$clear_line" == true ]]; then
                if [[ "$line" =~ "}" ]]; then
                  clear_line=false
                else
                  clear_line=true
                fi
                echo ""
              else
                echo "$line"
              fi
          done < ${{ env.CONFIG_DIRECTORY }}/main.tf > ./main.tf && \
            mv ./main.tf ${{ env.CONFIG_DIRECTORY }}/main.tf

      - name: Terraform Apply
        uses: ./.github/actions/tf-apply
        with:
          workspace: ${{ env.WORKSPACE_NAME }}

        # NOTE: Destroy config once Resource Links have been removed.

      - name: Terraform Apply
        if: success()
        uses: ./.github/actions/tf-apply
        with:
          workspace: ${{ env.WORKSPACE_NAME }}
          destroy: true

        # NOTE: Once all Fastly resources are deleted, we delete the TFC workspace.

      - name: Delete Dev Workspace
        run: |
          curl -s \
            --header "Authorization: Bearer ${{ secrets.TF_API_TOKEN }}" \
            --header "Content-Type: application/vnd.api+json" \
            --request POST \
            https://app.terraform.io/api/v2/organizations/example/workspaces/\
            ${{ env.WORKSPACE_NAME }}/actions/safe-delete > response.json
          if [[ $(jq '.errors | length == 0' response.json) == false ]]; then
            jq '.errors' response.json
            exit 1
          fi

OK, the first thing I tripped up on was…

# Stop any in-flight CI jobs when a new commit is pushed.
concurrency:
  group: api-gateway-dev-${{ github.ref_name }}
  cancel-in-progress: true

I’m used to naming the group with ${{ github.ref_name }}. This has historically always worked fine for me, until I started developing a more complex CI/CD pipeline where there are lots of different events causing a workflow file to be triggered …and more importantly, causing a workflow file to be STOPPED!

Specifically, I ran into an issue where merging a PR should cause the dev environment to be spun-down, but I had a stage workflow also running and that caused by dev workflow to be unexpectedly cancelled, meaning I had infrastructure resources orphaned, and it was a pain to have to go back and manually clean them up.

So to resolve that I had to make sure to prefix each group with a unique name.

The next thing of interest is…

on:
  workflow_dispatch:
  pull_request:
    types: [opened, edited, closed, reopened, synchronize]
    paths:
      - .github/workflows/api-gateway-*
      - cmd/gateway/**
      - internal/gateway/**
      - infrastructure/fastly/compute/api-gateway/**

It took me a while to figure out the types to use. I have a very specific set of events I want to react to and I discovered that edited on a pull_request doesn’t mean “code pushed to the PR”. Nope, for that you need synchronize.

Also, I wanted to avoid having workflow files triggered unnecessarily, hence I restrict this particular workflow file to changes made to relevant API gateway files. Now, this isn’t perfect because the application code could change in the future to depend on other packages in the codebase (as it’s a mono-repo) and that would mean having to update this list (which as you can imagine is likely to get forgotten about).

The next thing of interest is…

# IMPORTANT: for TF_VAR_* wrap double-quotes in single-quotes.
env:
  APP_DIRECTORY: "./cmd/gateway"
  CONFIG_DIRECTORY: "./infrastructure/fastly/compute/api-gateway"
  PACKAGE_PATH: "/pkg/gateway.tar.gz"
  TF_API_TOKEN: "${{ secrets.TF_API_TOKEN }}"
  TF_CLOUD_ORGANIZATION: "example"
  TF_VAR_backend: '"https://api-dev.example.com"'
  TF_VAR_env: '"dev"'

When passing a value to Terraform via an environment variable, you need the value itself to be a string, but doing export FOO=bar doesn’t assign "bar" to the variable, it only assigns bar without the quotes and that causes Terraform to treat the value like an undefined variable. So it’s very important to wrap the string in single quotes.

The next thing of interest is…

# don't run this when the PR is closed
if: ${{ github.event.action != 'closed' && github.actor != 'dependabot[bot]' }}

Specifically, the bit I want to call out is the check for dependabot[bot]. This is because when this automated-bot opens a PR on your repo, it won’t have access to your secrets and so it won’t be able to make TFC API calls (and do other things) so you might as well skip the CI/CD pipeline workflow.

The next difficulty I ran into was figuring out how best to avoid API rate limits from the Terraform Cloud API. This particular set of workflow files would be run many multiple times a day, and using the TFC API each time would quickly cause us to hit a rate-limit threshold.

To avoid that situation I needed to figure out clever caching patterns. But then, I needed to also account for the fact that a cache is volatile/ephemeral and can cause unexpected error scenarios.

If you take a look at the create-workspace job, you’ll see that once I successfully create a new workspace, I’ll then call the API to create an environment variable that contains a required secret value (for interacting with the Fastly API) and when that is successful I create a file called tfc-workspace-id and it’s that file that I cache and restore on each job run.

When the workflow calls the terraform-plan job you’ll see I’ve added a comment about handling error scenarios. Originally, I would depend on the create-workspace job and only run terraform-plan if the other job had completed successfully. But that didn’t always work and ties back to my comment about a cache being volatile/ephemeral.

To explain: the terraform-plan job (and all jobs that follow it) requires a TFC Workspace. The name of the workspace is expected to match WORKSPACE_NAME. The creation of the workspace happens in the create-workspace job.

Part of the create-workspace is making API requests, and to make that easier I pass a JSON file in to be used as the data input rather than manually configuring it in a single curl call. But first I have to modify the JSON data to include the values required (as I don’t want them hardcoded into the JSON):

  - name: Modify Workspace JSON Payload
    if: steps.cache-workspace-id.outputs.cache-hit != 'true'
    run: |
      payload=./infrastructure/fastly/compute/api-gateway/tfc-api/payloads/workspace.json
      jq '.data.attributes.name = "${{ env.WORKSPACE_NAME }}"' "$payload" \
        > temp.json && mv temp.json "$payload"

  - name: Modify Workspace Variable JSON Payload
    if: steps.cache-workspace-id.outputs.cache-hit != 'true'
    run: |
      payload=./infrastructure/fastly/compute/api-gateway/tfc-api/payloads/workspace-variables.json
      jq '.data.attributes.value = "${{ secrets.FASTLY_API_KEY }}"' \
        "$payload" > temp.json && mv temp.json "$payload"

Here’s the payload JSON for both API calls:

{
  "data": {
    "type": "workspaces",
    "attributes": {
      "name": "api-gateway-dev-"
    },
    "relationships": {
      "project": {
        "data": {
          "type": "projects",
          "id": "prj-<REDACTED>"
        }
      }
    }
  }
}

{
  "data": {
    "type": "vars",
    "attributes": {
      "key": "FASTLY_API_KEY",
      "value": "",
      "description": "API key used by the Fastly Terraform Provider",
      "category": "env",
      "sensitive": true
    }
  }
}

But there are scenarios where the workspace might not exist! This can happen when there is an error in the create-workspace job. For example, when calling the TFC API. Although the API might error, we don’t want to mark the job as not being successful. This is because the API could be called and fail for multiple reasons. e.g. the cached API result wasn’t found and we got a 422 from the API.

A 422 code response from the TFC API thus indicates we tried to create a workspace that already exists. So in this scenario we still want the remaining jobs to run (as the workspace does exist).

Ultimately, if the error was something else and the workspace thus wasn’t created, then at worst we’ll get an error back from TFC when trying to upload the TF configuration to it.

The last issue worth noting is related to tearing-down the infrastructure when the PR is merged. There was an issue with the Fastly Terraform provider (or more specifically a Fastly API issue) where the removal of a Fastly ‘Config Store’ (which was needed by the application code) has to be done as a two-part terraform apply.

i.e. you have to remove the resource_link block from the fastly_service_compute resource and run terraform apply, and then you can trigger a terraform apply -destroy via the HashiCorp GitHub Action using the is_destroy field.

Stage Config

# We deploy to staging environment when PR is merged or we push to `main`.
name: "API Gateway TF Stage"

# Stop any in-flight CI jobs when a new commit is pushed.
concurrency:
  group: api-gateway-stg-${{ github.ref_name }}
  cancel-in-progress: true

on:
  workflow_dispatch:
  push:
    branches:
      - main
    paths:
      - .github/workflows/api-gateway-*
      - cmd/gateway/**
      - internal/gateway/**
      - infrastructure/fastly/compute/api-gateway/**

# IMPORTANT: for TF_VAR_* wrap double-quotes in single-quotes.
env:
  APP_DIRECTORY: "./cmd/gateway"
  CONFIG_DIRECTORY: "./infrastructure/fastly/compute/api-gateway"
  PACKAGE_PATH: "/pkg/gateway.tar.gz"
  TF_API_TOKEN: "${{ secrets.TF_API_TOKEN }}"
  TF_CLOUD_ORGANIZATION: "example"
  TF_VAR_backend: '"https://api-stage.example.com"'
  TF_VAR_env: '"stg"'
  TF_WORKSPACE: "api-gateway-stage"

jobs:
  terraform-apply:
    name: "Apply"
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      - name: Install Go and Fastly CLI
        uses: ./.github/actions/fastly-cli

      - name: Build Compute Package
        run: fastly compute build --dir ${{ env.APP_DIRECTORY }}

      - name: Calculate Compute Package Hash
        id: cph
        run: |
          echo "compute_package_hash=\
          $(fastly compute hash-files --skip-build --dir ${{ env.APP_DIRECTORY }} --quiet)"\ 
          >> $GITHUB_OUTPUT

      - name: Copy Compute Package to Terraform workspace
        run: cp ${{ env.APP_DIRECTORY }}${{ env.PACKAGE_PATH }} ${{ env.CONFIG_DIRECTORY }}

      - name: Terraform Apply
        uses: ./.github/actions/tf-apply
        with:
          workspace: ${{ env.TF_WORKSPACE }}

      - name: Upload file as artifact
        uses: actions/upload-artifact@v4
        with:
          name: ${{ steps.compute-package-hash.outputs.compute_package_hash }}
          path: ${{ env.APP_DIRECTORY }}${{ env.PACKAGE_PATH }}
          retention-days: 14
          overwrite: true

      - name: Create Git Tag
        run: |
          echo "GIT_STAGE_TAG=stg/$(date +'%Y-%m-%d/%H-%M-%SZ')/\
          $(echo ${{ github.triggering_actor }} | tr '[:upper:]' '[:lower:]' | \
          tr -d ' ')/${{ steps.cph.outputs.compute_package_hash }}" >> "${GITHUB_ENV}"

      - name: Check Git Tag
        run: echo "$GIT_STAGE_TAG"

      - name: Apply Git Tag
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: 'refs/tags/${{ env.GIT_STAGE_TAG }}',
              sha: context.sha
            })

The stage config isn’t as complex as the dev workflow, but there are some things of interest to point out, such as uploading our Compute package (which is our application code) to GitHub, which is for the purposes of later promoting that same package to production.

We then use the hash of the Compute package as part of a git tag for staging deploys. Again, this hash is required for the promoting of the staging artifact to the production environment.

Production Config

# We deploy to production environment when workflow is triggered manually.
# Person triggering workflow must provide hash of Compute package to deploy.
name: "API Gateway TF Prod"

# Stop any in-flight CI jobs when a new commit is pushed.
concurrency:
  group: api-gateway-prd-${{ github.ref_name }}
  cancel-in-progress: true

on:
  workflow_dispatch:
    inputs:
      compute_package_hash:
        type: string
        required: true
        description: hash of compute package (see `git tag`)

# IMPORTANT: for TF_VAR_* wrap double-quotes in single-quotes.
env:
  APP_DIRECTORY: "./cmd/gateway"
  CONFIG_DIRECTORY: "./infrastructure/fastly/compute/api-gateway"
  GH_TOKEN: ${{ github.token }} # Required to use the `gh` CLI
  PACKAGE_PATH: "/pkg/gateway.tar.gz"
  TF_API_TOKEN: "${{ secrets.TF_API_TOKEN }}"
  TF_CLOUD_ORGANIZATION: "example"
  TF_VAR_backend: '"https://api.example.com"'
  TF_VAR_env: '"prd"'
  TF_WORKSPACE: "api-gateway-production"

jobs:
  terraform-apply:
    name: "Apply"
    runs-on: ubuntu-latest
    permissions:
      actions: read
      contents: write
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Debug Information
        uses: ./.github/actions/debug

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod

      - name: Download Compute Package to Terraform workspace
        run: |
          gh run download -n ${{ inputs.compute_package_hash }}
          mv gateway.tar.gz ${{ env.CONFIG_DIRECTORY }}

      - name: Terraform Apply
        uses: ./.github/actions/tf-apply
        with:
          workspace: ${{ env.TF_WORKSPACE }}

      - name: Create Git Tag
        run: |
          echo "GIT_PROD_TAG=prd/$(date +'%Y-%m-%d/%H-%M-%SZ')/\
          $(echo ${{ github.triggering_actor }} | tr '[:upper:]' '[:lower:]' |\
          tr -d ' ')/${{ inputs.compute_package_hash }}" >> "${GITHUB_ENV}"

      - name: Check Git Tag
        run: echo "$GIT_PROD_TAG"

      - name: Apply Git Tag
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: 'refs/tags/${{ env.GIT_PROD_TAG }}',
              sha: context.sha
            })

Again, this workflow isn’t as complex as the dev one, so the only interesting part here is that we’re downloading the artifact from GitHub and then running a Terraform apply to upload the Compute package to our production environment.

Custom Actions

The only other files probably worth showing you are the .github/actions/tf-var-branch, .github/actions/tfc-dev-workspace-name and .github/actions/tf-apply steps which I moved into separate action files to make them easier to reuse.

Here is tf-var-branch:

name: "Terraform Variable Branch Calculation"
description: "Generates a value for TF_VAR_branch"
runs:
  using: composite
  steps:
    - name: Set TF_VAR_branch
      shell: bash
      run: |
        echo "TF_VAR_branch=\"-$(echo ${{ github.head_ref }} |\ 
        tr '[:upper:]' '[:lower:]' | tr '/' '-' | tr '.' '-' |\
        tr '_' '-')\"" >> "${GITHUB_ENV}"
        # NOTE: We only set the branch variable for dev (stg/prd will use an empty string).

ℹ️ NOTE

I had to update this action twice. The first was to replace dot characters with a hyphen as TFC complained. The second was to replace underscores with a hyphen as Fastly complained (i.e. an underscore is not a valid character in a domain).

Here is tfc-dev-workspace-name:

name: "Terraform Cloud Dev Workspace Name"
description: "Generates a workspace name for the PR dev environment"
runs:
  using: composite
  steps:
    - name: Set WORKSPACE_NAME
      shell: bash
      run: |
        # e.g. `"-example"` (set in ../tf-var-branch/action.yml)
        branch_quoted='${{ env.TF_VAR_branch }}' 
        # strip the leading quote (e.g. turn `"-example"` to `-example"`)
        branch_strip_leading_quote="${branch_quoted#\"}" 
        # strip the trailing quote (e.g. turn `-example"` to `-example`)
        branch="${branch_strip_leading_quote%\"}" 
        echo "WORKSPACE_NAME=api-gateway-dev${branch}" >> $GITHUB_ENV

Here is tf-apply:

# yamllint disable rule:line-length

name: "Terraform Apply"
description: "Uploads TF config, creates a run and applies it"
inputs:
  workspace:
    description: "The TFC Workspace"
    required: true
  destroy:
    description: "When true, this uses terraform to DESTROY the managed infrastructure"
    default: "false"
outputs:
  run_id:
    description: "The ID of the created run"
    value: ${{ steps.apply.outputs.run_id }}
runs:
  using: composite
  steps:
    - name: Upload Configuration
      uses: hashicorp/tfc-workflows-github/actions/upload-configuration@v1.2.0
      id: apply-upload
      with:
        workspace: ${{ inputs.workspace }}
        directory: ${{ env.CONFIG_DIRECTORY }}

    - name: Create Apply Run
      uses: hashicorp/tfc-workflows-github/actions/create-run@v1.2.0
      id: apply-run
      with:
        workspace: ${{ inputs.workspace }}
        configuration_version: ${{ steps.apply-upload.outputs.configuration_version_id }}
        # IMPORTANT: This will DESTROY the dev infrastructure.
        is_destroy: ${{ inputs.destroy == true || inputs.destroy == 'true' }}  

    - name: Apply
      uses: hashicorp/tfc-workflows-github/actions/apply-run@v1.2.0
      if: fromJSON(steps.apply-run.outputs.payload).data.attributes.actions.IsConfirmable
      id: apply
      with:
        run: ${{ steps.apply-run.outputs.run_id }}
        comment: "Apply Run from GitHub Actions CI ${{ github.sha }}"

Conclusion

I’ve not covered every single line of code in the GitHub configuration. I’ve also not included the Terraform config files because they are specific to my use case and so would be considered ‘noise’ as far as understanding the fundamental CI/CD flow we’re trying to learn about here.

But hopefully the explanation of each environment workflow, and then being able to compare that to the actual implementation files for GitHub Actions is enough to help you see how you might be able to implement your own CI/CD workflow across multiple environments.

Good luck out there!

The Power of OpenAPI: Simplifying API Design and Documentation

Fri, 18 Aug 2023 00:00:00 +0000

Introduction

In the rapidly evolving landscape of software development, creating robust and user-friendly APIs has become essential. One tool that has gained immense popularity for designing, documenting, and testing APIs is OpenAPI.

In this comprehensive guide, we’ll explore why OpenAPI is so important, how to write an OpenAPI document, and the key sections you need to know. Whether you’re a seasoned developer or just getting started, mastering OpenAPI can greatly enhance your API development process.

Why OpenAPI Matters

OpenAPI, formerly known as Swagger, is an open-standard format for describing APIs. It serves as both a machine-readable and human-friendly documentation that enables developers to understand, visualize, and interact with APIs effortlessly. By defining your API using OpenAPI, you unlock a host of benefits:

Clear Documentation: OpenAPI provides a clear and structured way to document your API, making it easy for both developers and non-developers to understand how to use it.
Consistency: With a standardized format, your API documentation and implementation remain consistent, reducing confusion and enhancing collaboration among teams.
Code Generation: OpenAPI enables automatic code generation for client libraries and server stubs in various programming languages, saving time and effort during development.
Testing and Validation: The OpenAPI specification can be used to validate requests and responses, ensuring that your API adheres to the defined contract.
Interactive Documentation: Tools like Swagger UI and ReDoc allow you to create interactive documentation that lets users explore and test your API in real-time.

Summary for those short on time

OpenAPI is straightforward to write once you understand the high-level structure.

Below is an OpenAPI document to get you going as a basic example
but read the full post to understand the details.

Also check out the related blog post on Fastly’s dev.to blog:
Better Fastly API clients with OpenAPI Generator

openapi: 3.0.3

info:
  title: Your API
  version: 1.0.0

servers:
  - url: https://api.example.com

paths:
  /teams/{team_id}/members:
    parameters:
      - $ref: "#/components/parameters/team_id"
    get:
      summary: List team members
      description: List all members for the specified team.
      operationId: list-members
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/list_members_response"
              examples:
                body:
                  $ref: "#/components/examples/list_members_response"

components:
  parameters:
    team_id:
      name: team_id
      in: path
      required: true
      style: simple
      schema:
        $ref: "#/components/schemas/team_id"

  schemas:
    team_id:
      type: string
      description: Alphanumeric string identifying a team.
      example: AB1C2defGhijKLMNop3qR

    list_members_response:
      type: array
      description: List of members within the specified team.
      items:
        type: string

  examples:
    list_members_response:
      value:
        - "Andrew"
        - "Bob"
        - "Christine"

The above example OpenAPI document describes an API. Specifically it describes:

The OpenAPI version supported (i.e. openapi:)
Some API Metadata (i.e. info:)
The API address (i.e. server:)
Supported API endpoints (i.e. paths:)
Different ‘components’ referenced by the paths: configuration (i.e. components:)

In practice the last section components: is where most of the ‘meat’ of the API configuration happens. You’ll see it contains different sections like parameters, schemas and examples.

The paths: section typically doesn’t define behaviours inline but instead will reference objects defined inside of components: whenever they need to describe some behaviour of the endpoint.

Any time you see $ref that means we’re about to reference an object defined elsewhere (might be in the same file, under components: or it could be from a separate file).

So in the above example we can see the paths: config references a few different components:

#/components/parameters/team_id: this describes the API path’s team_id input parameter.
- This parameter object also references a component (#/components/schemas/team_id) for describing the team_id.
#/components/schemas/list_members_response: this describes the schema for how the response body should look for this API endpoint.
#/components/examples/list_members_response: this demonstrates an example of what the schema looks like in practice.

This is the basic structure of an OpenAPI document. Yes, they can become more complex as the API grows, but at its foundation you will always find this familiar structure.

OK, we’ve got the quick “summary” out of the way, let’s dig a little deeper…

Getting Started with OpenAPI

Before diving into the intricacies of writing an OpenAPI document, let’s set up the basics.

If at any point throughout this post you are in doubt or you require some additional clarity, then please refer to the specification document.

ℹ️ NOTE

I’ve used version 3.0.3 for my examples.

Basic Structure

An OpenAPI document is written in YAML or JSON format (I’ve used YAML for my examples).

It consists of various sections that collectively describe your API.
At a high level, here’s what the structure looks like:

openapi: 3.0.3
info:
  title: Your API
  version: 1.0.0
paths: {}
components: {}

The above example is not exhaustive as it only describes three ‘objects’:

info: Info Object
paths: Paths Object
components: Components Object

Refer to the OpenAPI Object for a complete list of top-level (i.e. root object) fields.

Defining Endpoints

Now, let’s break down the process of defining endpoints in your OpenAPI document.

Paths and Methods

Endpoints are defined using the paths section. Each endpoint is associated with an HTTP method (e.g., GET, POST) and a URL path. Here’s an example:

paths:
  /users:
    get:
      summary: Get a list of users
      responses:
        '200':
          description: Successful response

REF: Path Item Object.

Parameters

You can add parameters to your endpoints using the parameters section. Parameters can be path parameters, query parameters, headers, and more. Here’s an example of a path parameter:

paths:
  /users/{userId}:
    parameters:
      - name: userId
        in: path
        required: true
        schema:
          type: integer

Possible values for the in field are: “query”, “header”, “path” or “cookie”.

REF: Parameter Object.

Structuring Data

Defining request and response bodies, along with data types, is crucial for a well-documented API.

Request and Response Bodies

You can specify request and response bodies using the requestBody and responses sections. Here’s how to define a request body:

paths:
  /users:
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'

The use of $ref allows us to avoid having an inline schema for one that is defined separately from the path object. This is useful in scenarios where the referenced schema might need to be reused across different paths. We’ll take a look at the components/schemas section next.

REF: Operation Object and Request Body Object.

Data Types

Data types are defined under the components/schemas section.
Here’s an example of defining a User schema:

components:
  schemas:
    User:
      type: object
      properties:
        id:
          type: integer
        username:
          type: string

The components section not only supports defining schemas separate from where they should be referenced but also responses, parameters, examples and more. We’ll take a look at some of these fields in more detail later.

REF: Components Object.

Adding Metadata

Enhance your API documentation by adding metadata and grouping related endpoints.

API Information

The info section provides high-level information about your API, such as title, version, and description:

info:
  title: Your API
  version: 1.0.0
  description: This is a sample API documentation.

REF: Info Object.

Tags and Grouping

You can group related endpoints using tags (typically added within the Operation Object):

tags:
  - name: Users
    description: Operations related to users

But tags can also be defined at the top-level (i.e. root object).

For example, Fastly uses the following conventions for its tags which determine how endpoints are documented on Fastly’s Developer Hub (DevHub):

tags:
  - name: unlisted # Publish on DevHub at an unlisted URL and exclude from search results
  - name: excludeFromSearch # Publish on DevHub but exclude from search results
  - name: internal  # Do not publish on DevHub or build into API clients
  - name: beta # Display "beta" notice on DevHub
  - name: limited-availability # Display "LA" notice on DevHub

REF: Tag Object.

Handling Errors

Communicating errors is crucial in API design, and OpenAPI helps you to define your error responses.

Status Codes

Specify status codes and their meanings in your responses section:

paths:
  /users/{userId}:
    get:
      responses:
        '200':
          description: Successful response
        '404':
          description: User not found

The responses field is a container for the expected responses of an operation. The container maps a HTTP response code (e.g. 200 or 404 etc) to the expected response.

REF: Responses Object.

Error Responses

You can also define error responses with detailed information, and for specific response types:

responses:
  '400':
    description: Invalid Credit
    content:
      application/json:
        schema:
          $ref: '#/components/schemas/Error'

In the above example we describe a 400 Bad Request error response that will have the Content-Type of application/json (i.e. the response will use JSON) and we reference an external schema.

Below is an example schema definition that uses the popular “Problem Details” format from RFC 7807:

components:
  schemas:
    Error:
      type: object
      properties:
        type:
          type: string
          description: A URI reference that identifies the problem type.
          example: "https://example.com/probs/out-of-credit"
        title:
          type: string
          description: A short, human-readable summary of the problem.
          example: "You do not have enough credit."
        status:
          type: integer
          description: The HTTP status code generated by the origin server for this occurrence of the problem.
          example: 400
        detail:
          type: string
          description: A human-readable explanation specific to this occurrence of the problem.
          example: "Your current balance is 30, but that costs 50."
        instance:
          type: string
          description: A URI reference that identifies the specific occurrence of the problem.
          example: "/account/12345/msgs/abc"
      required:
        - type
        - title
        - status

What this schema describes is the following example error JSON that a user might see:

 {
     "type": "https://example.com/probs/out-of-credit",
     "title": "You do not have enough credit.",
     "detail": "Your current balance is 30, but that costs 50.",
     "instance": "/account/12345/msgs/abc",
     "balance": 30,
     "accounts": ["/account/12345", "/account/67890"]
 }

REF: Response Object.

Testing and Validation

Ensure the reliability of your API by testing and validating it using OpenAPI tools.

Tools for Validation

OpenAPI supports validating your API against the defined schemas using a variety of tools, such as Swagger Inspector and Spectral.

Generating Client SDKs

Using OpenAPI also helps with generating (and maintaining) client SDKs by using tools like Swagger Codegen and OpenAPI-Generator to accelerate development for various programming languages.

I’ve written about this process on the Fastly dev.to blog:
Better Fastly API clients with OpenAPI Generator

Conclusion

Embracing OpenAPI as a core component of your API development process can lead to more efficient, well-documented, and collaborative projects.

By understanding the key sections of an OpenAPI document and leveraging its capabilities, you can create APIs that are not only robust but also user-friendly.

Whether you’re a solo developer or part of a large team, OpenAPI is a powerful tool that simplifies API design and documentation, ultimately contributing to a better developer experience.

In this brief guide, I’ve explored the importance of OpenAPI in API development, discussed its benefits, and walked you through the process of writing an OpenAPI document.

By breaking down the essential sections and providing practical examples, I hope you feel empowered to leverage OpenAPI for your next API project. Happy API designing!

Beginner's guide to creating a Terraform Provider

Sat, 12 Nov 2022 00:00:00 +0000

You have an API and you want your customers to be able to use that API via Terraform. In this post I’m going to explain how you can write your own Terraform provider to enable your customers to integrate your API into their “infrastructure as code” pipeline.

You don’t need an API to be able to follow along, nor am I going to make you write one as part of the learning process (that’s just a waste of time). Instead I’m going to demonstrate how a Terraform provider works by talking through the code for an example provider I’ve written.

ℹ️ NOTE

You can find my ‘mock’ provider here on GitHub if you’d prefer to read the code yourself. It’s heavily commented to make it as easy as possible.

The idea is that once you finish reading this post, you should (in theory) be able to just copy my mock provider code base and start adding in your own real API calls rather than using the stubbed data I’ve provided for the sake of keeping the example provider simple.

In case you’re interested, I am the maintainer for the popular Fastly Terraform provider (which demonstrates a much more complex provider). If you want to know how to use the Fastly Terraform provider, then you’re in luck as I’ve written about that subject on the Fastly blog here as well as written a Terraform ‘best practices’ guide on the Fastly Developer Hub here (which is definitely worth checking out).

I should also be clear and say that this isn’t a ‘complete’ guide. There are many aspects to writing a Terraform provider I will gloss over or omit entirely.

That all said, let’s get started…

Prerequisite

I’ve presumed the reader:

Is familiar with the Go programming language and how to write programs with it.
Is familiar with what Terraform is (and ideally has used it before).

go.mod

Let’s start with the direct dependencies needed by my ‘mock’ provider:

go 1.19

require (
    github.com/google/uuid v1.3.0
    github.com/hashicorp/terraform-plugin-docs v0.13.0
    github.com/hashicorp/terraform-plugin-sdk/v2 v2.24.0
)

The first dependency google/uuid is only used because I needed a quick and convenient way to generate a unique ID (as we’ll see later on), while the hashicorp/terraform-plugin-docs dependency is only used for the sake of generating documentation from your provider code (a really useful tool actually so you should definitely check it out).

The last dependency, hashicorp/terraform-plugin-sdk, is what fundamentally enables a Terraform provider to work like it does.

Hashicorp is working on a replacement to the terraform-plugin-sdk called the Terraform Plugin Framework. I’ve not yet played with the new SDK, so I may well write a follow-up post where I migrate my mock provider to this new framework and/or do so for the Fastly Terraform provider, from which I’ll share any useful lessons learned.

ℹ️ NOTE

It’s worth clarifying that the terraform-plugin-sdk isn’t going anywhere any time soon and in the meantime Hashicorp provides guidelines for which SDK you should use, so I recommend reading that if you’re unsure of which route to take.

UPDATE: I have since discovered there is a ‘scaffold’ project that Hashicorp provides here https://github.com/hashicorp/terraform-provider-scaffolding that is similar to my mock provider! So I would recomend, once you finish reading this post, giving that a read to see how their hierarchy compares to my mock provider (there’s also one for the new plugin framework, which I’ll be checking out: https://github.com/hashicorp/terraform-provider-scaffolding-framework).

main.go

Now let’s take a look at our main package entry point, which is ‘short and sweet’…

package main

import (
    "github.com/hashicorp/terraform-plugin-sdk/v2/plugin"
    "github.com/integralist/terraform-provider-mock/mock"
)


func main() {
    plugin.Serve(&plugin.ServeOpts{
        ProviderFunc: mock.Provider,
    })
}

We import the SDK’s plugin package and call the Serve function, telling it where to find our provider (i.e. mock.Provider) which is where we define the structure of our Terraform provider.

The following implementation of my mock provider is small because, well, my mock provider is small (by design), where as you’ll find real-world providers to be a lot larger than this:

func Provider() *schema.Provider {
    return &schema.Provider{
        Schema: map[string]*schema.Schema{
            "foo": {
                Type:        schema.TypeString,
                Optional:    true,
                DefaultFunc: schema.EnvDefaultFunc("MOCK_FOO", nil),
            },
        },
        ResourcesMap: map[string]*schema.Resource{
            "mock_example": resourceExample(),
        },
        DataSourcesMap: map[string]*schema.Resource{
            "mock_example": dataSourceExample(),
        },
    }
}

So there two parts to a Terraform provider, which will help to explain the above chunk of code:

Resources: A resource is a ‘thing’ you create, and then manage its life cycle via Terraform.
Data Sources: A data source is data you can query and reference within your resources.

The following diagram is one I designed for the Fastly blog post I mentioned earlier, and it does a good job visualising the overall Terraform workflow coordination between your terminal shell, the Terraform CLI and the Terraform provider itself.

ℹ️ NOTE

A data source is actually a subset of a resource. This will become apparent shortly when we discuss CRUD (Create, Read, Update, Delete) operations.

Looking at the above code, the first thing we do is define a provider (schema.Provider) and its ‘schema’ (schema.Schema). In the case of our mock provider we actually have a super simple provider schema, it’s just a single ‘attribute’ called foo.

Here is an example of what that would look like in a consumer’s Terraform configuration:

provider "mock" {
  foo = "example_value"
}

Because of the DefaultFunc field in our schema, if foo wasn’t set in the above example configuration, then the value would default to the value assigned to the environment variable MOCK_FOO (or nil if the environment variable wasn’t set).

Typically the provider’s schema is for defining top-level attributes such as an API key or API endpoint. Take a look at the Fastly Terraform provider for an example:

func Provider() *schema.Provider {
    provider := &schema.Provider{
        Schema: map[string]*schema.Schema{
            "api_key": {
                Type:        schema.TypeString,
                Optional:    true,
                DefaultFunc: schema.EnvDefaultFunc("FASTLY_API_KEY", nil),
                Description: "Fastly API Key from https://app.fastly.com/#account",
            },
            "base_url": {
                Type:        schema.TypeString,
                Optional:    true,
                DefaultFunc: schema.EnvDefaultFunc("FASTLY_API_URL", gofastly.DefaultEndpoint),
                Description: "Fastly API URL",
            },
            "force_http2": {
                Type:        schema.TypeBool,
                Optional:    true,
                Default:     false,
                Description: "Set this to `true` to disable HTTP/1.x fallback mechanism"
            },
            "no_auth": {
                Type:        schema.TypeBool,
                Optional:    true,
                Default:     false,
                Description: "Set to `true` if your config consumes data without auth",
            },
        },

    ...
  }

  ...
}

Now there’s one aspect to our provider that I’ve omitted and that’s the ConfigureFunc field. Because my mock provider doesn’t have a real API to use I’ve not defined that field in my mock provider but here is an example from the Fastly Terraform provider to give you an idea of what it could look like:

func Provider() *schema.Provider {
    provider := &schema.Provider{
        Schema: map[string]*schema.Schema{
      // ...
        },
        DataSourcesMap: map[string]*schema.Resource{
            // ...
        },
        ResourcesMap: map[string]*schema.Resource{
            // ...
        },
    }

    provider.ConfigureContextFunc = func(_ context.Context, d *schema.ResourceData) (any, diag.Diagnostics) {
        config := Config{
            APIKey:     d.Get("api_key").(string),
            BaseURL:    d.Get("base_url").(string),
            NoAuth:     d.Get("no_auth").(bool),
            ForceHTTP2: d.Get("force_http2").(bool),
            UserAgent:  provider.UserAgent(TerraformProviderProductUserAgent, version.ProviderVersion),
        }
        return config.Client()
    }

    return provider
}

In the Fastly Terraform provider you can see the ConfigureContextFunc (there are ‘context’ aware versions of most functions) we return a config.Client() which is effectively the Fastly API client. I’ll come back to this later when we look at the CREATE function of a resource to show you how you can get access to this API client.

For now, let’s jump back to our mock provider to remind ourselves of what was configured after the Schema field…

ResourcesMap: map[string]*schema.Resource{
    "mock_example": resourceExample(),
},
DataSourcesMap: map[string]*schema.Resource{
    "mock_example": dataSourceExample(),
},

So now we have a basic understanding of what a ‘resource’ is and what a ‘data source’ is, we can clearly see that a Terraform provider can be setup with multiple of each type.

The interesting part here is that there is a ‘convention over configuration’ style pattern at play here. Specifically, the map key has to be of the format <provider_name>_<resource|data_source>, and in my mock provider I only have one resource and one data source and I name them the same mock_example (as ‘mock’ is the name of my provider, and ‘example’ is the resource/data-source name).

In the case of the resource, the provider will attempt to resolve the function assigned to the map key by looking up the file resource_<provider_name>_<resource_name>.go, i.e. resource_mock_example.go while it will attempt to resolve the data source using data_source_<provider_name>_<data_source_name>, i.e. data_source_mock_example.go.

If we look at the tree structure of my mock provider we’ll see that’s exactly what it contains:

.
├── main.go
├── mock
│   ├── data_source_mock_example.go
│   ├── provider.go
│   └── resource_mock_example.go

OK, all good so far. Now let’s get into the meat of these two files, what exactly do they do?

Well, firstly the ‘example’ resource defines its own schema (e.g. the attributes it contains) and it also specifies CRUD operation functions (i.e. Create, Read, Update, Delete) for managing the life cycle of the resource. In our mock provider it looks like this:

func resourceExample() *schema.Resource {
    return &schema.Resource{
        Create: resourceCreate,
        Read:   resourceRead,
        Update: resourceUpdate,
        Delete: resourceDelete,

        Schema: map[string]*schema.Schema{
            "last_updated": {
                Type:     schema.TypeString,
                Computed: true,
            },
            "not_computed_optional": {
                Type:     schema.TypeString,
                Optional: true,
            },
            "not_computed_required": {
                Type:     schema.TypeString,
                Required: true,
            },
            "foo": {
                Type:     schema.TypeList,
                Optional: true,
                Elem: &schema.Resource{
                    Schema: map[string]*schema.Schema{
                        "bar": {
                            Type:     schema.TypeList,
                            MaxItems: 1,
                            Required: true,
                            Elem: &schema.Resource{
                                Schema: map[string]*schema.Schema{
                                    "number": {
                                        Type:     schema.TypeInt,
                                        Optional: true,
                                    },
                                    "version": {
                                        Type:     schema.TypeString,
                                        Computed: true,
                                    },
                                },
                            },
                        },
                    },
                },
            },
            "baz": {
                Type:     schema.TypeList,
                Required: true,
                Elem: &schema.Resource{
                    Schema: map[string]*schema.Schema{
                        "qux": {
                            Type:     schema.TypeString,
                            Required: true,
                        },
                    },
                },
            },
            "some_list": {
                Type:     schema.TypeList,
                Optional: true,
                Elem: &schema.Schema{
                    Type: schema.TypeString,
                },
            },
        },
    }
}

There are a few things to pay attention to in this schema:

last_updated is a ‘computed’ attribute, meaning the user shouldn’t attempt to set that in their configuration but instead leave it to the Terraform provider to set it dynamically. A user would then use output variables to access the value set by the provider.
not_computed_required: is marked as ‘required’ and so a user of our provider must set this attribute, while other attributes are marked as ‘optional’.
foo has a type of TypeList, meaning we need to define what schema its ‘elements’ should have. We use Elem to specify there should be a nested bar block, which itself contains number and version attributes (the latter being a ‘computed’ attribute).

In terms of a consumer of our Terraform provider, someone might write their configuration like so:

resource "mock_example" "testing" {
  not_computed_required = "some value"

  foo {
    bar {
      number = 1
    }
  }
  foo {
    bar {
      number = 2
    }
  }
  foo {
    bar {
      number = 3
    }
  }

  baz {
    qux = "x"
  }
  baz {
    qux = "y"
  }
  baz {
    qux = "z"
  }

  some_list = ["a", "b", "c"]
}

In the above example the user has set the attribute not_computed_required because it was marked as ‘required’ in our provider schema, while providing multiple instances of the optional foo block, and multiple instances of the required bar blocks. The some_list attribute is also provided, although it’s marked as ‘optional’.

It’s worth noting the above configuration can, through clever use of the Terraform Configuration Language (TCL), be made more concise/dynamic…

resource "mock_example" "testing" {
  not_computed_required = "some value"

  dynamic "foo" {
    for_each = [{ number = 1 }, { number = 2 }, { number = 3 }]
    content {
      bar {
        number = foo.value.number
      }
    }
  }

  dynamic "baz" {
    for_each = [{ something = "x" }, { something = "y" }, { something = "z" }]
    content {
      qux = baz.value.something
    }
  }

  some_list = ["a", "b", "c"]
}

…but this blog post isn’t about TCL so I’d recommend reading the official Terraform documentation.

Let’s now look back at the CRUD functions for our resource:

Create: resourceCreate,
Read:   resourceRead,
Update: resourceUpdate,
Delete: resourceDelete,

These functions are called at specific times depending on what terraform CLI command was executed by a user, and also depending on what state their project is in at the time the command is executed.

For example, running terraform apply for the first time will cause the ‘Create’ function to be called (so all the relevant resources defined in the user’s configuration will be created), followed (typically) by the ‘Read’ function (so that the user’s local project state will be updated with the new resources created via the ‘Create’ function).

The next time the user runs terraform apply they’ll find Terraform will trigger the ‘Read’ function to execute first (so that Terraform can update the project state and know if anything has changed), before then determining if the ‘Update’ function needs to be called, because let’s say the user has modified their configuration in such a way as to require one of the resources defined to be updated (e.g. to make API calls on our behalf).

Let’s now look at each CRUD function individually…

Create

The CREATE operation typically will make API calls to create resources. It won’t set anything in the terraform state, with the exception of setting a unique ID that will be used by all the other functions to access the resource data from state.

Let’s look at the code in our ‘mock’ provider to see what we’re doing…

func resourceCreate(d *schema.ResourceData, m any) error {
    foo := d.Get("foo").([]any)
    d.SetId("123")
    return resourceRead(d, m)
}

So we can see the function is passed a schema.ResourceData type. This is essentially the user’s local configuration data, and we can see that the first thing we do is try to get the value of the foo attribute. It also has a second argument which is effectively additional ‘metadata’ and is what we would reference in order to get access to our API client.

Now if this was a real provider, then we’d have an API client that would be creating an actual resource for the user. Remember earlier we talked about the ConfigureFunc function of the provider, which was responsible for defining an instance of an API client? Well, the way we get access to the API client would look something like this (where APIClient is some type that represents your API client):

client := m.(*APIClient).conn

Because foo is a TypeList you can see we type assert it to []any. Now as I said at the start, we don’t have an API to make calls to and so we don’t actually do anything here with the foo value defined in the user’s configuration.

Let’s pretend we made an API call, and in the response we got back an ID that we wanted to use as a unique key to track this resource in our terraform state file. To set the unique ID we call d.SetId("123"), where 123 is the unique ID we got back from the API response.

The mere existence of the ID, and lack of error returned, means Terraform will presume the CREATE operation was successful and store the user’s “foo” data in the local state file.

Finally, we call the READ function (resourceRead()) and pass through the d and m arguments. In this case, the way we’ve setup our CREATE function means that if the READ operation fails, then the error it returns will be returned through our CREATE function and will cause the create operation to fail as well.

It’s up to you whether that’s something you want to do, as a READ could fail due to a network issue and not necessarily mean there was an error with the CREATE operation.

Read

The READ operation must handle three things:

Calling out to the API to get the latest data for our resource.
Flatten the API response (or at least extract the values we need) and marshal the data into a format that Terraform understands (i.e. map[string]any).
Set the latest data into Terraform’s state file so it can identify if there were any differences between what the user has defined in their configuration and what actually exists on the remote data side.

Let’s look at the code in our ‘mock’ provider to see what we’re doing…

func resourceRead(d *schema.ResourceData, m any) error {
  resourceID := d.Id()
  foo := d.Get("foo")

  for _, f := range foo.([]any) {
    f := f.(map[string]any)

    for _, b := range f["bar"].([]any) {
      b := b.(map[string]any)
      b["version"] = uuid.New().String()
    }
  }

  if err := d.Set("foo", foo); err != nil {
    return err
  }

  d.Set("last_updated", time.Now().Format(time.RFC850))

  return nil
}

OK, so there’s a little more going on here, but nothing too complicated. First thing we do is get the ID we set into the Terraform state after we had initially created the resource (d.Id()).

Next, because we don’t have a real API to get live data from, I’m going to hardcode data to be set into terraform state. To do that I’m going to get the data out from the Terraform state using d.Get("foo") and I’m going to dynamically update the nested version attribute, which if you remember was defined as a ‘computed’ value (meaning its an attribute the Terraform provider must set).

But as this is a nested attribute I have to iterate over each parent structure until I reach the relevant level of the data structure where I can then set a value on the version attribute. That’s effectively what the two for loops are doing, and you’ll see this is where the github.com/google/uuid dependency comes in.

Once that’s done we’re able to set the updated data structure into the local state file using d.Set() and be aware that we can’t just set the version attribute directly, we must set the root foo attribute and this means constructing its complete data structure.

Remember that the last_updated attribute was also marked as ‘computed’ as so we need to set that dynamically too (and for that I just set the value using the current timestamp, and to be fair I could of done the same for the version attribute rather than use an external uuid package).

Update

The UPDATE operation is similar to the READ but we typically use additional SDK methods to help us know if there’s been a change that would require a modification to the data.

Let’s look at the code in our ‘mock’ provider to see what we’re doing…

func resourceUpdate(d *schema.ResourceData, m any) error {
  resourceID := d.Id()

  if d.HasChange("foo") {
    foo := d.Get("foo").([]any)
    d.Set("last_updated", time.Now().Format(time.RFC850))
  }

  return resourceRead(d, m)
}

So some similar stuff here. First we get the resource’s ID with d.Id() but now we’re using d.HasChange() to see if the foo attribute has changed, and if it has, then we modify the local state. Of course the mock provider is just setting a value for last_updated as if there was a change but in reality your real provider would be making an ‘update’ API call to update the resource.

Lastly, we do a READ operation to be sure we get the updated API changes stored into our local state.

Delete

The DELETE operation is very simple. In order to delete a resource from the local Terraform state we simply need to set the ID for the resource to an empty string…

func resourceDelete(d *schema.ResourceData, m any) error {
  resourceID := d.Id()
  d.SetId("")
  return nil
}

Now in our mock provider you’ll see we get the resource ID and although that looks pointless, in a real provider, you would want to get the resource ID because remember you’d need to make an API call to delete the resource first before deleting it from the Terraform state.

Data Sources

OK, before we wrap up, let’s look at our example data source to see the schema structure:

func dataSourceExample() *schema.Resource {
    return &schema.Resource{
        Read: dataSourceRead,
        Schema: map[string]*schema.Schema{
            "things": {
                Type:     schema.TypeList,
                Computed: true,
                Elem: &schema.Resource{
                    Schema: map[string]*schema.Schema{
                        "id": {
                            Type:     schema.TypeInt,
                            Computed: true,
                        },
                        "version": {
                            Type:     schema.TypeString,
                            Computed: true,
                        },
                    },
                },
            },
        },
    }
}

So there’s a lot of similarities between a ‘resource’ and a ‘data source’, and that’s because a data source is a subset of a resource. So the main difference is we have a READ operation/function but none of the other CRUD functions (e.g. Create, Update, Delete) because a data source is only used for querying data and not something you create/update/delete.

As far as the schema is concerned we’ve defined a things attribute, which is ‘computed’ along with all its nested attributes, and that makes sense as this isn’t something a user defines in their configuration.

Now let’s take a look at the READ function to see what that’s doing:

func dataSourceRead(d *schema.ResourceData, m any) error {
    const jsonStream = `
        [
            {"id": 1, "version": "a"},
            {"id": 2, "version": "b"},
            {"id": 3, "version": "c"},
        ]`

    things := make([]map[string]any, 0)
    err := json.NewDecoder(strings.NewReader(jsonStream)).Decode(&things)
    if err != nil {
        return err
    }

    if err := d.Set("things", things); err != nil {
        return err
    }

    d.SetId(strconv.FormatInt(time.Now().Unix(), 10))

    return nil
}

The jsonStream variable is creating stubbed data. This is as if we had made an actual API call and received back a bunch of data from the API.

In order for us to store the returned data into Terraform we need to marshal the data into a format that matches what the schema expects (e.g. []map[string]any) and in this case we’re imagining our API returned JSON formatted data and so we use json.NewDecoder to decode the API response into the things variable.

We then set the things data back into the local state using d.Set().

Now we’ve reached the final interesting part, which is the d.SetId() call. So to explain, we don’t have a unique ID for this data resource so we create one using a timestamp format. I typically see people use a hash of the returned API data as a unique key, so pick your poison.

How to use this provider

Now if my mock provider was published to the Terraform registry it would be very simple to consume, but I’ve not done that because I wanted an opportunity to explain how to test your own provider locally to be sure it’s working first.

So, to consume this mock provider without it being published to the Terraform registry, follow these steps:

Clone the repo and build the terraform-provider-mock binary:
```
make build
```
Create a separate directory for your own Terraform project.
- e.g. cd ../ && mkdir example-tf

Create a dev.tfrc file in your own Terraform project’s directory:


provider_installation {
dev_overrides {
  "integralist/mock" = "../terraform-provider-mock" // the directory where the binary was built.
}
direct {}
}

In that shell instance set the TF_CLI_CONFIG_FILE environment variable.
- e.g. export TF_CLI_CONFIG_FILE=/example-tf/dev.tfrc
Create Terraform project files.
- e.g. see the Example Terraform Consumer Code in my mock provider’s README.
Initialize your Terraform project and then execute a plan.
- e.g. terraform init && terraform plan

ℹ️ NOTE

every time you make a change to the Terraform provider code, you’ll need to rebuild the binary and then go to your consuming Terraform project and reinitialize (i.e. terraform init) so it picks up the latest version of the terraform-provider-mock binary.

Local Development

When running:

$ TF_LOG=TRACE terraform init

You should notice a couple of things different from what you’d normally see when initializing a new Terraform project.

The first is a message highlighting the fact that a provider ‘override’ is in place:

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - integralist/mock in /Users/integralist/Code/terraform/terraform-provider-mock

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

That is expected in this case we’ve followed the instructions above, which tells us how to implement an override for the sake of local testing of the provider code.

The other thing you’ll notice is an error:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
integralist/mock: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/integralist/mock

If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.

This error is expected because we’ve not actually published this provider to the Terraform registry, so indeed it cannot be found. But the error doesn’t prevent you from consuming the local provider binary still.

ℹ️ NOTE

don’t use Print functions from the fmt package in the Terraform provider as, depending on the execution flow, Terraform can treat it as input to its internal program and treat it as an error. So use Print functions from the log package instead.

UPDATE: I’ve since discovered https://pkg.go.dev/github.com/hashicorp/terraform-plugin-log/tflog

Debugging a Terraform Provider

There are essentially two approaches:

Log-Based Debugging
Debugger-Based Debugging.

Refer to the official Hashicorp plugin documentation and also the Fastly Terraform provider documents, which demonstrates the latter approach.

Conclusion

There we have it! A simple Terraform provider broken down as simply as I could manage.

Hopefully you can now mentally model your own API into a provider for your users.

Rust Smart Pointers

Mon, 20 Jun 2022 00:00:00 +0000

Summary

The five types covered: Box, Rc, Arc, Cell, RefCell.

ℹ️ NOTE

Not discussed are Mutex<T> and RwLock<T>,
which provide mutual-exclusion.

Box<T>: A pointer type for heap allocation.
Rc<T>: Single-threaded ‘multiple owners’ pointer (immutable).
Arc<T>: Thread-safe ‘multiple owners’ pointer (immutable).
Cell<T>: Single-threaded mutability for Copy types.
RefCell<T>: Single-threaded mutability for reference types.

Quick notes

Rc<T> and Arc<T> both support mutability by wrapping the inner value with another type. Cell<T> or RefCell<T> for Rc, and Mutex<T>, RwLock<T> or one of the Atomic* types for Arc.

Arc<T>’s thread-safety comes at a cost: additional performance overhead.
So choose Rc whenever possible.

Cell<T> behaves like an exclusive borrow, aka a &mut T (meaning the compiler won’t let you have multiple mutable borrows), while RefCell<T> removes the compile-time borrow-checks so that it can provide more flexibility than Cell<T> has (which is constrained to Copy types only).

Subsequently RefCell introduces the possibility of ending up with multiple mutable borrows. The borrow rules for RefCell aren’t statically checked at compile time but dynamically checked at runtime, which means this type is capable of triggering a panic if multiple mutable borrows is detected.

IMPORTANT

I’m not the author of the following content.

My own summary/quick notes (above) are sufficient for my own reference, but in case it’s not, then the following content may be useful to you.

Rather than link off to various intertwined documentation pages, with lots of extra cruft that can muddy the essentials of what you need to know. I read through the official Rust documentation and cherry-picked useful subsets of that information and grouped it in a way that helped me to more easily make sense of the topic.

This means I take no credit for the following content.
It was written by many people much smarter than me.

Context

A pointer is a general concept for a variable that contains an address in memory. This address refers to, or “points at,” some other data. The most common kind of pointer in Rust is a reference. References are indicated by the & symbol and borrow the value they point to. They don’t have any special capabilities other than referring to data, and have no overhead.

Smart pointers, on the other hand, are data structures that act like a pointer but also have additional metadata and capabilities.

ℹ️ NOTE

Both String and Vec<T> types count as smart pointers because they own some memory and allow you to manipulate it. They also have metadata and extra capabilities or guarantees.

Smart pointers are usually implemented using structs. Unlike an ordinary struct, smart pointers implement the Deref and Drop traits. The Deref trait allows an instance of the smart pointer struct to behave like a reference so you can write your code to work with either references or smart pointers. The Drop trait allows you to customize the code that’s run when an instance of the smart pointer goes out of scope.

`Box<T>`

Boxes allow you to store data on the heap rather than the stack. What remains on the stack is the pointer to the heap data.

Usage:

When you have a type whose size can’t be known at compile time and you want to use a value of that type in a context that requires an exact size (e.g. recursive data types).
When you have a large amount of data and you want to transfer ownership but ensure the data won’t be copied when you do so (e.g. only the small amount of pointer data is copied around on the stack, while the data it references stays in one place on the heap).
When you want to own a value and you care only that it’s a type that implements a particular trait rather than being of a specific type (i.e. trait objects used for dynamic dispatch).

`Rc<T>`

This type is an abbreviation for reference counting and it enables multiple ownership. It lets us have multiple “owning” pointers to the same data, and the data will be freed (destructors will be run) when all pointers are out of scope.

We use the Rc<T> type when we want to allocate some data on the heap for multiple parts of our program to read and we can’t determine at compile time which part will finish using the data last.

ℹ️ NOTE

Rc<T> is only for use in single-threaded scenarios. When shared ownership between threads is needed, Arc<T> (Atomic Reference Counted) can be used.

`Arc<T>`

The same as Rc<T> but thread-safe (it has the same API as Rc<T> to make them interchangeable). But thread-safety comes with a performance cost so if you don’t need thread-safety, then definitely opt for Rc<T> instead.

`Cell<T>`/`RefCell<T>`

Rust memory safety is based on this rule: Given an object T, it is only possible to have one of the following:

Having several immutable references (&T) to the object.
Having one mutable reference (&mut T) to the object.

This rule can be bent using Cell<T> and is referred to as interior mutability.

Cell<T> implements interior mutability by moving values in and out of the Cell<T>. To use references instead of values, use the RefCell<T> type, and acquire a write lock before mutating.

Borrows for RefCell<T>s are tracked at runtime, unlike Rust’s native reference types which are entirely tracked statically, at compile time.

Because RefCell<T> borrows are dynamic it is possible to attempt to borrow a value that is already mutably borrowed; when this happens it results in thread panic.

ℹ️ NOTE

Neither Cell<T> nor RefCell<T> are thread-safe.
The Sync marker trait is not implemented.

Many shared smart pointer types, including Rc<T> and Arc<T>, provide containers that can be cloned and shared between multiple parties. The contained values can only be borrowed with &, not &mut. Without cells it would be impossible to mutate data inside of these smart pointers at all.

It’s very common then to put a RefCell<T> inside shared pointer types to reintroduce mutability. But as RefCell<T>s are for single-threaded scenarios, consider using RwLock<T> or Mutex<T> instead of RefCell<T> if you need shared mutability in a multi-threaded situation.

New Laptop Configuration (Second Edition)

Thu, 26 May 2022 00:00:00 +0000

Introduction

This is the second edition to “New Laptop Configuration”.

Backup

When moving laptops I will temporarily backup my existing GPG and SSH keys (encrypted) to an external data storage device.

We start by making a directory to hold the files temporarily:

mkdir /tmp/keys

Next I start backing up my GPG data:

# Export the secret key that encrypts all the data in my 'password store'.
# I encode the binary data in a ASCII armored file (.asc).
gpg --export-secret-keys --armor <NAME> > /tmp/keys/<NAME>.asc

# Once exported I add a password to the file so people are unable to open it.
# This will produce a .asc.gpg file.
gpg --symmetric /tmp/keys/<NAME>.asc

# To prevent having to trust all the same keys as before I'll export the trust database.
gpg --export-ownertrust > /tmp/keys/trustdb.txt 

# Lastly, I move the files to my external flash drive (USB).
mv /tmp/keys/<NAME>.asc.gpg /Volumes/.../<NAME>.asc.gpg
mv /tmp/keys/trustdb.txt /Volumes/.../trustdb.txt

Next, I backup my SSH data:

# Recursively copy all files into a zip archive.
zip -r /tmp/keys/sshbackup ~/.ssh/

# I list the contents of the zip archive to be sure I have everything in there.
unzip -l /tmp/keys/sshbackup.zip

# Once archived I add a password to the file so people are unable to open it.
# This will produce a .zip.gpg file.
gpg --symmetric /tmp/keys/sshbackup.zip

# Lastly, I move the file to my external flash drive (USB).
mv /tmp/keys/sshbackup.zip.gpg /Volumes/.../sshbackup.zip.gpg

Then I clear out the temporary directory:

rm -rf /tmp/keys

Steps

Install Rust.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Install Go.

https://go.dev/dl/

Import GPG/SSH keys.

mkdir /tmp/keys
cd /tmp/keys

gpg --decrypt /tmp/keys/<NAME>.gpg > <NAME>
gpg --import <NAME>.asc

rm ~/.gnupg/trustdb.gpg
gpg --import-ownertrust < /tmp/keys/trustdb.txt

gpg --decrypt /tmp/keys/sshbackup.zip.gpg > sshbackup.zip
unzip /tmp/keys/sshbackup.zip
mv /tmp/keys/.ssh/ ~/

rm -rf /tmp/keys

Setup SSH.

eval "$(ssh-agent -s)"
ssh-add --apple-use-keychain ~/.ssh/github

ℹ️ NOTE

I’ve since moved to https://www.warp.dev/ (see my Dev Tools post)
so I no longer use Alacritty or Fig (steps 5 and 6 below).

Install Alacritty.

# https://github.com/alacritty/alacritty/releases
mkdir .bash_completion
curl https://raw.githubusercontent.com/alacritty/alacritty/master/extra/completions/alacritty.bash \
    -o ~/.bash_completion/alacritty

Install Fig.

https://fig.io/

Install Homebrew + packages.

# https://brew.sh
brew bundle --file /tmp/Brewfile install

Change default shell to Zsh.

echo /opt/homebrew/bin/zsh | sudo tee -a /etc/shells
chsh -s /opt/homebrew/bin/zsh

ℹ️ NOTE

I’ve since moved to https://neovim.io/ (see my Dev Tools post)
so I no longer use Vim-Plug (step 9 below).

Configure Vim-Plug.

sh -c 'curl -fLo "${XDG_DATA_HOME:-$HOME/.local/share}"/nvim/site/autoload/plug.vim --create-dirs \
     https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'

ℹ️ NOTE

I no longer use tmux (see my Dev Tools post)
so a bunch of the following step 10 has changed.

Setup dotfiles.

ℹ️ NOTE

Don’t forget to execute ‘prefix + I’ in tmux to install plugins.

cd /tmp
git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm
git clone https://github.com/Integralist/dotfiles.git
curl https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh -o ~/.git-prompt.sh
curl https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.zsh -o ~/.zsh/_git
cp -R .alacritty.yml .zsh .zshrc .config .gitconfig .gitignore .gnupg .ignore .inputrc .leptonrc .tmux.conf ~/
chown -R $(whoami) ~/.gnupg/
chmod 600 ~/.gnupg/*
chmod 700 ~/.gnupg

Setup password store.

KEY_ID=$(gpg --list-keys <NAME> | head -n 2 | tail -n 1 | cut -d ' ' -f 7)
pass init $KEY_ID
pass git init
pass git remote add origin git@github.com:<private/repo>
pass git pull

ℹ️ NOTE

I no longer use Safari (see my Dev Tools post)
so step 12 is redundant now.

Safari extensions.

AdBlock One
Dark Reader for Safari
Super Agent for Safari (Cookie Consent Automation)
Tab Sessions

Configure OS.

- Dock (Automatically hide and show the Dock)
- Keyboard (Key Repeat = Fast, Delay Until Repeat = Short)
- Accessibility > Zoom (Use keyboard shortcuts to zoom)
- Date & Time > Clock (Show date + Display the time with seconds)
- Mission Control (disable "Automatically rearrange Spaces based on most recent use")
- Terminal Developer Mode (`spctl developer-mode enable-terminal`)
- Wake up from sleep (`sudo pmset -a standbydelay 7200`)

Manage multiple versions of the Go programming language

Fri, 22 Apr 2022 00:00:00 +0000

Installing Go

All versions of Go are available to download here: https://go.dev/dl/

The installation directory will be: /usr/local/go.

Once you have Go installed, you can then use the go command to either install go based tools/binaries or other versions of Go.

ℹ️ NOTE

Additionally, you can automate the install via a terminal using the following shell function (only tested on macOS):

# you can swap `ag` for `grep` if you prefer
alias golatest="curl -L https://github.com/golang/go/tags 2>&1 |\
ag '/golang/go/releases/tag/go[\w.]+' -o | cut -d '/' -f 6 | awk NR==1 | ag '\d.+' -o"

function go_install {
  if [ -z "$1" ]; then
    echo USAGE: go_install 1.18.1 OR go_install \$\(golatest\)
    return
  fi
  v=$1
  osname=$(uname -s | tr '[:upper:]' '[:lower:]')
  hardware=$(uname -m)
  mkdir -p ~/goversions
  if ! test -f "~/goversions/go$v.$osname-$hardware.pkg"; then
    printf "\nDownloading %s\n\n" "go$v.$osname-$hardware"
    curl -L -o ~/goversions/go$v.$osname-$hardware.pkg https://go.dev/dl/go$v.$osname-$hardware.pkg
  fi
  echo ""
  sudo rm -rf /usr/local/go; sudo installer -pkg ~/goversions/go$v.$osname-$hardware.pkg -target /usr/local/
  clear
  go version
}

Installing binaries

As of version go 1.16 go install is now responsible only for installing binaries, not modifying a go.mod file (that’s what go get is for).

go install example.com/cmd@v1.0.0
go install example.com/cmd@latest

ℹ️ NOTE

See reference documentation.

Installing different go versions

The following is the ‘official’ approach…

ℹ️ NOTE

See reference documentation.

go install golang.org/dl/go1.18@latest
go1.18 download
go1.18 version # go version go1.18 darwin/amd64
alias go=go1.18

If you want a simple binary overwrite of the global go command (for example, an alias doesn’t work with Makefiles that reference go because make targets run in a subshell):

go install golang.org/dl/go1.18@latest
go1.18 download
sudo cp $(which go1.18) $(which go)

ℹ️ NOTE

This requires sudo as it’s copying into /usr/local/....

If you want the latest ‘tip’ release (and maybe with additional/unreleased features, at the time of writing that might have included something like ‘fuzzing’ which was made available in go1.18):

go install golang.org/dl/gotip
gotip download dev.fuzz # also download a dev/tip specific tool/feature
gotip test -fuzz=FuzzFoo

Basic switcher using `go.mod` as reference

# This function identifies the go version specified in a project's go.mod
# It then attempts to switch to a binary of that version.
# If none exists it will instruct you how to download it.
#
# NOTE: Some tools, e.g. TinyGo, won't work with this approach because we're
# just replacing the go binary and the VERSION file, where the originally
# installed version of go will have things like CGO files that TinyGo will try
# to use and if those don't align with the version of the binary we've switched
# to, then it means TinyGo will fail to compile. 
#
# In that scenario you're better off using the `go_install` shell function 
# approach at the top of the page.
function go_version {
    if [ -f "go.mod" ]; then
        v=$(grep -E '^go \d.+$' ./go.mod | grep -oE '\d.+$')
        if [[ ! $(go version | grep "go$v") ]]; then
          echo ""
          echo "About to switch go version to: $v"
          if ! command -v "$HOME/go/bin/go$v" &> /dev/null
          then
            echo "run: go install golang.org/dl/go$v@latest && go$v download && sudo cp \$(which go$v) \$(which go)"
            return
          fi
          sudo cp $(which go$v) $(which go)
          echo -n go$v | sudo tee $(dirname $(dirname $(which go)))/VERSION > /dev/null
        fi
    fi
}

# To support the configuring our go environment we will override the cd
# command to call the go logic for checking the go version.
#
# NOTE: We use `command` and not `builtin` because the latter doesn't take into
# account anything available on the user's $PATH but also because it didn't
# work with the Starship prompt which seems to override cd also.
function cd {
  command cd "$@"
  RET=$?
  go_version
  return $RET
}

Unofficial: goenv

https://github.com/syndbg/goenv

The nice thing about goenv is that it lets you more easily automate a switch between projects using either GOENV_VERSION or .go-version (docs).

Setting up Neovim for Rust and Go development

Thu, 14 Apr 2022 00:00:00 +0000

UPDATE November 2022

Not long after I wrote this post I had switched from VimScript to using Lua and also making large sets of changes and tweaks to my configuration. The source of truth is:

https://github.com/integralist/nvim (which is a submodule within https://github.com/integralist/dotfiles)

This post is being kept for posterity, but ultimately I would recommend you look at the above dotfiles repo instead.

This is going to be a very focused post because when you’re looking to get your code editor configured you typically just want answers. Let’s go…

ℹ️ NOTE

I configure Neovim with ~/.config/nvim/init.vim.

Requirements

Executables.
Plugin manager.
An LSP (Language Server Protocol) client.

Executables

There’s a bunch of Rust and Go based tools we’ll want to have installed first.

Add the following to your .bashrc:

export PATH="/usr/local/go/bin:$PATH"
export PATH="$HOME/go/bin:$PATH"
export PATH="$HOME/.cargo/bin:$PATH"

# rustup
#
# avoid https://github.com/rust-analyzer/rust-analyzer/issues/4172
#
# NOTE: Has to be defined after PATH update to locate .cargo directory.
#
export RUST_SRC_PATH="$(rustc --print sysroot)/lib/rustlib/src/rust/src"

# To support the configuring our go environment we will override the cd
# command to call the go logic for checking the go version.
#
# We also make sure to call ls when changing directories as it's nice to see
# what's in each directory.
#
# NOTE: We use `command` and not `builtin` because the latter doesn't take into
# account anything available on the user's $PATH but also because it didn't
# work with the Starship prompt which seems to override cd also.
function cd {
  command cd "$@"
  RET=$?
  ls
  go_version
  return $RET
}

# configure go environment
#
# Custom go binaries are installed in $HOME/go/bin.
#
function go_version {
    if [ -f "go.mod" ]; then
        v=$(grep -E '^go \d.+$' ./go.mod | grep -oE '\d.+$')
        if [[ ! $(go version | grep "go$v") ]]; then
          echo ""
          echo "About to switch go version to: $v"
          if ! command -v "$HOME/go/bin/go$v" &> /dev/null
          then
            echo "run: go install golang.org/dl/go$v@latest && go$v download && \
              sudo cp \$(which go$v) \$(which go)"
            return
          fi
          sudo cp $(which go$v) $(which go)
        fi
    fi
}
if [ ! -f "$HOME/go/bin/gofumpt" ]; then
    go install mvdan.cc/gofumpt@latest
fi
if [ ! -f "$HOME/go/bin/revive" ]; then
    go install github.com/mgechev/revive@latest
fi

# configure rust environment
#
# - autocomplete
# - rust-analyzer
# - cargo audit
# - cargo-nextest
# - cargo fmt
# - cargo clippy
# - cargo edit
#
source $HOME/.cargo/env
if [ ! -f "$HOME/.config/rustlang/autocomplete/rustup" ]; then
  mkdir -p ~/.config/rustlang/autocomplete
  rustup completions bash rustup >> ~/.config/rustlang/autocomplete/rustup
fi
source "$HOME/.config/rustlang/autocomplete/rustup"
if ! command -v rust-analyzer &> /dev/null
then
  brew install rust-analyzer
fi
if ! cargo audit --version &> /dev/null; then
  cargo install cargo-audit --features=fix
fi
if ! cargo nextest --version &> /dev/null; then
  cargo install cargo-nextest
fi
if ! cargo fmt --version &> /dev/null; then
  rustup component add rustfmt
fi
if ! cargo clippy --version &> /dev/null; then
  rustup component add clippy
fi
if ! ls ~/.cargo/bin | grep 'cargo-upgrade' &> /dev/null; then
  cargo install cargo-edit
fi

Plugin manager

I use vim-plug:

call plug#begin()

Plug 'folke/trouble.nvim'
Plug 'hrsh7th/cmp-buffer'
Plug 'hrsh7th/cmp-nvim-lsp'
Plug 'hrsh7th/cmp-nvim-lsp-signature-help'
Plug 'hrsh7th/cmp-path'
Plug 'hrsh7th/cmp-vsnip'
Plug 'hrsh7th/nvim-cmp'
Plug 'hrsh7th/vim-vsnip'
Plug 'hrsh7th/vim-vsnip-integ'
Plug 'j-hui/fidget.nvim'
Plug 'kosayoda/nvim-lightbulb'
Plug 'm-demare/hlargs.nvim'
Plug 'neovim/nvim-lspconfig'
Plug 'nvim-treesitter/nvim-treesitter', {'do': ':TSUpdate'}
Plug 'simrat39/rust-tools.nvim'
Plug 'weilbith/nvim-code-action-menu'
Plug 'williamboman/nvim-lsp-installer'

call plug#end()

Define the above and run :PlugInstall. Once everything is installed, continue to the next section.

LSP

I use Neovim’s built-in LSP.

Add the following plugin configuration:

" ------------------------------------
" j-hui/fidget.nvim
" ------------------------------------
"
lua require("fidget").setup()

" ------------------------------------
" kosayoda/nvim-lightbulb
" ------------------------------------
"
autocmd CursorHold,CursorHoldI * lua require('nvim-lightbulb').update_lightbulb()

" ------------------------------------
" weilbith/nvim-code-action-menu
" ------------------------------------
"
let g:code_action_menu_window_border = 'single'

" ------------------------------------
" folke/trouble.nvim
" ------------------------------------
"
lua require("trouble").setup()

" ------------------------------------
" Neovim LSP
" ------------------------------------
"
" Configure Rust LSP.
"
" https://github.com/simrat39/rust-tools.nvim#configuration
"
lua <<EOF
local opts = {
  -- rust-tools options
  tools = {
    autoSetHints = true,
    hover_with_actions = true,
    inlay_hints = {
      show_parameter_hints = true,
      parameter_hints_prefix = "",
      other_hints_prefix = "",
      },
    },

  -- all the opts to send to nvim-lspconfig
  -- these override the defaults set by rust-tools.nvim
  -- https://github.com/rust-analyzer/rust-analyzer/blob/master\
     /docs/user/generated_config.adoc
  -- https://rust-analyzer.github.io/manual.html#features
  server = {
    settings = {
      ["rust-analyzer"] = {
        assist = {
          importEnforceGranularity = true,
          importPrefix = "crate"
          },
        cargo = {
          allFeatures = true
          },
        checkOnSave = {
          -- default: `cargo check`
          command = "clippy"
          },
        },
        inlayHints = {
          lifetimeElisionHints = {
            enable = true,
            useParameterNames = true
          },
        },
      }
    },
}
require('rust-tools').setup(opts)
EOF

" Configure Golang LSP.
"
" https://github.com/golang/tools/blob/master/gopls/doc/settings.md
" https://github.com/golang/tools/blob/master/gopls/doc/analyzers.md
" https://github.com/golang/tools/blob/master/gopls/doc/vim.md#neovim
" https://github.com/neovim/nvim-lspconfig/blob/master/doc/server_configurations.md#gopls
" https://github.com/golang/tools/blob/master/gopls/doc/vim.md#neovim
" https://www.getman.io/posts/programming-go-in-neovim/
"
lua <<EOF
require('lspconfig').gopls.setup{
    cmd = {'gopls'},
  settings = {
    gopls = {
      analyses = {
        nilness = true,
        unusedparams = true,
        unusedwrite = true,
        useany = true,
      },
      experimentalPostfixCompletions = true,
      gofumpt = true,
      staticcheck = true,
      usePlaceholders = true,
    },
  },
    on_attach = on_attach,
}
EOF

" Configure Golang Environment.
"
fun! GoFumpt()
  :silent !gofumpt -w %
  :edit
endfun
autocmd FileType go map <buffer> <leader>p :call \
  append(".", "fmt.Printf(\"%+v\\n\", )")<CR> <bar> :norm $a<CR><esc>j==$i
autocmd FileType go map <buffer> <leader>e :call \
  append(".", "if err != nil {return err}")<CR> <bar> :w<CR>
autocmd BufWritePost *.go call GoFumpt()
autocmd BufWritePost *.go :cex system('revive '..expand('%:p')) | cwindow

" Order imports on save, like goimports does:
"
lua <<EOF
  function OrgImports(wait_ms)
    local params = vim.lsp.util.make_range_params()
    params.context = {only = {"source.organizeImports"}}
    local result = vim.lsp.buf_request_sync(0, "textDocument/codeAction", params, wait_ms)
    for _, res in pairs(result or {}) do
      for _, r in pairs(res.result or {}) do
        if r.edit then
          vim.lsp.util.apply_workspace_edit(r.edit, "UTF-8")
        else
          vim.lsp.buf.execute_command(r.command)
        end
      end
    end
  end
EOF
autocmd BufWritePre *.go lua OrgImports(1000)

" Configure LSP code navigation shortcuts
" as found in :help lsp
"
nnoremap <silent> <c-]>     <cmd>lua vim.lsp.buf.definition()<CR>
nnoremap <silent> <c-k>     <cmd>lua vim.lsp.buf.signature_help()<CR>
nnoremap <silent> K         <cmd>lua vim.lsp.buf.hover()<CR>
nnoremap <silent> gi        <cmd>lua vim.lsp.buf.implementation()<CR>
nnoremap <silent> gc        <cmd>lua vim.lsp.buf.incoming_calls()<CR>
nnoremap <silent> gd        <cmd>lua vim.lsp.buf.type_definition()<CR>
nnoremap <silent> gr        <cmd>lua vim.lsp.buf.references()<CR>
nnoremap <silent> gn        <cmd>lua vim.lsp.buf.rename()<CR>
nnoremap <silent> gs        <cmd>lua vim.lsp.buf.document_symbol()<CR>
nnoremap <silent> gw        <cmd>lua vim.lsp.buf.workspace_symbol()<CR>

" Replaced LSP implementation with code action plugin...
"
" nnoremap <silent> ga        <cmd>lua vim.lsp.buf.code_action()<CR>
"
nnoremap <silent> ga        <cmd>CodeActionMenu<CR>

nnoremap <silent> [x        <cmd>lua vim.diagnostic.goto_prev()<CR>
nnoremap <silent> ]x        <cmd>lua vim.diagnostic.goto_next()<CR>
nnoremap <silent> ]s        <cmd>lua vim.diagnostic.show()<CR>

" Replaced LSP implementation with trouble plugin...
"
" nnoremap <silent> <space>q  <cmd>lua vim.diagnostic.setloclist()<CR>
"
nnoremap <silent> <space>q  <cmd>Trouble<CR>

" Setup Completion
" https://github.com/hrsh7th/nvim-cmp#recommended-configuration
"
lua <<EOF
local cmp = require('cmp')
cmp.setup({
  snippet = {
    expand = function(args)
        vim.fn["vsnip#anonymous"](args.body)
    end,
  },
  mapping = {
    ['<C-p>'] = cmp.mapping.select_prev_item(),
    ['<C-n>'] = cmp.mapping.select_next_item(),
    ['<S-Tab>'] = cmp.mapping.select_prev_item(),
    ['<Tab>'] = cmp.mapping.select_next_item(),
    ['<C-d>'] = cmp.mapping.scroll_docs(-4),
    ['<C-f>'] = cmp.mapping.scroll_docs(4),
    ['<C-Space>'] = cmp.mapping(cmp.mapping.complete(), { 'i', 'c' }),
    ['<C-e>'] = cmp.mapping.close(),
    ['<CR>'] = cmp.mapping.confirm({
      behavior = cmp.ConfirmBehavior.Insert,
      select = true,
    })
  },
  sources = {
    { name = 'nvim_lsp' },
    { name = 'vsnip' },
    { name = 'path' },
    { name = 'buffer' },
    { name = 'nvim_lsp_signature_help' },
  },
})
EOF

" Setup Treesitter and friends
"
" NOTE: originally used `ensure_installed = "all"` but an experimental PHP
" parser was causing NPM lockfile errors.
"
lua <<EOF
require('nvim-treesitter.configs').setup {
  ensure_installed = { 
    "bash", 
    "c", 
    "cmake", 
    "css", 
    "dockerfile", 
    "go", 
    "gomod", 
    "gowork", 
    "hcl", 
    "help", 
    "html", 
    "http", 
    "javascript", 
    "json", 
    "lua", 
    "make", 
    "markdown", 
    "python", 
    "regex", 
    "ruby", 
    "rust", 
    "toml", 
    "vim", 
    "yaml", 
    "zig"
  },
  highlight = {
    enable = true,
  },
  rainbow = {
    enable = true,
    extended_mode = true,
    max_file_lines = nil,
  }
}
require('hlargs').setup()
EOF

Fix broken Vim Themes

Thu, 14 Apr 2022 00:00:00 +0000

I’ve been around the houses on this problem long enough, and after many years of trying various tricks and having them fail I decided to document an actual working solution.

Install a terminal that has good colour support (e.g. Alacritty)
Install a better Vim (i.e. Neovim) †

† This should still work with standard Vim, but actually Neovim has so many performance improvements, and quality of life plugins it’s not worth fighting for purity (trust me, I had been using standard vim and the standard macOS terminal app for 10 years before admitting I was fighting the tide).

Now that you have the right tools, let’s explain the configuration.

~/.config/nvim/init.vim

" this will tell Neovim to enable 24bit true color
if (has("termguicolors"))
 set termguicolors
endif

~/.bashrc

# this isn't necessary when using Neovim in an appropriate terminal, 
# but it will help when running Neovim inside of tmux.
export TERM="xterm-256color"

~/.tmux.conf

set-option -g default-terminal 'screen-256color-bce'
set-option -ga terminal-overrides ",xterm-256color:Tc"

The tmux manual states that default-terminal should be set to screen, tmux or a derivative of them, which is why this doesn’t get set to the same value (xterm-256color) that TERM has in the .bashrc file.

Additionally, we use the -bce suffix (rather than just setting screen-256color) because otherwise tmux won’t use a transparent background. Having a transparent background allows the terminal colour palette to be utilised and this helps Vim themes to display correctly.

Lastly, terminal-overrides let’s you configure tmux to know about terminal capabilities it otherwise might not be able to detect by itself. So this is why we set it to the same value (xterm-256color) that TERM has in the .bashrc file.

Developer Tools

Wed, 09 Mar 2022 00:00:00 +0000

I don’t know who needs to hear this but:
I love my developer tools.

Core

Package Manager: Homebrew
Terminal: Ghostty
Code Editor: Neovim

History

Used stock macOS Terminal.app with tmux for 10+ years.
I tried iTerm2, Kitty, Alacritty, WesTerm but none really fit well for me.
Constant colour issues forced me to switch from tmux to Zellij.
I moved to using Warp for a few years, but it still had issues.
I finally landed on Ghostty which has been great. Yes, it still has minor issues but the community is active and open.

Workflow

I use Zsh Autocomplete with fzf-tab to replace Zsh’s default completion selection menu with fzf which works really nicely.

I mostly use terminal splits and tabs for dividing up projects and tasks, and on the occasion I’ll use Neovim’s built-in terminal across multiple splits or floating windows. Otherwise, the other part of Ghostty I use a lot is its floating terminal that can be triggered from anywhere via a hotkey (as long as Ghostty is open somewhere).

💡 You can find my Neovim config here and my dotfiles here.

Supplementary

Code Statistics: Tokei
DNS Client: Doggo
Directory Lister: Tree
Directory Navigator: Broot
Directory Switcher: Zoxide
Disk Usage: Duf
Feed Reader: Tuifeed
File Content Display: Bat
File Finder: Fd
File Lister: Exa
File Removal: Rip
File Space Usage: Dust
Network Reachability: Gping
Network Utilisation: Bandwhich
Process Status: Procs
Process/System Monitor: Htop †
Shell: Zsh
Shell Autocomplete: FZF-tab
Shell Prompt: Starship

† I had switched to Bottom but htop is so much more flexible and configurable, and although the graphs in btm are nice, I don’t need them.

Browser

Controversial, but I had moved away from Firefox with a whole suite of security/privacy minded tools to Apple’s Safari browser.

I was using Safari’s “Tab Group” feature, which I really liked along with the following extensions…

AdBlock One
Dark Reader for Safari
SimplyJSON for Safari
Super Agent for Safari (Cookie Consent Automation)

…but ultimately the Tab Groups feature ended up being super buggy and duplicating bookmarks and groups etc and I couldn’t get the issue resolved, so I ended up back with Google Chrome.

💡 In the future I’ll likely switch over to using Ladybird.

Go Style Guide

Tue, 11 Jan 2022 00:00:00 +0000

This is my own personal style guide for Go.

Reference Materials

The following reference materials are my ‘go to’ whenever I’m unsure of something (they’re mostly official resources).

ℹ️ NOTE

Refer to the specification if ever confused about what the expected behaviour is.
See also the go.mod reference.

Naming

The following is a summary of how to name things in Go, gleaned from either my own experiences over the years or from some of the above reference materials.

Choose package names that lend meaning to the names they export (see also Google’s Go style guide for examples).
Where types are descriptive, name should be short (1 or 2 char name).
If longer name required, consider refactoring into smaller functions.
Commonly used names:
- Prefer i to index.
- Prefer r to reader.
- Prefer buf to buffer.
- Prefer cfg to config.
- Prefer dst, src to destination, source.
- Prefer in, out when referring to stdin/stdout (more flexible for mocked objects).
- Prefer rx, tx when dealing with channels.
  - i.e. receiver, transmitter.
- Prefer data when referring to file content.
  - Regardless of it being a string or []byte.
- Use ok instead of longer alternatives.
Errors:
- Types: <T>Error (e.g. type ExitError struct {...}).
- Values: Err<T> (e.g. var ErrFoo = errors.New("bar: baz")).
Interfaces:
- When an interface includes multiple methods, choose a name that accurately describes its purpose.
- Interfaces defining one method are named the same as the method with ‘er’ appended.
  - Sometimes the result isn’t correct English, that’s OK.
  - Sometimes we use English to make it nicer.
Return values on exported functions should only be named for documentation purposes.
- Side effect is that the variable is initialised at start of function with zero value.
- This can, in some cases, lead to a nice code design.
Set<T> vs Register<T>
- Set: use when flipping a bit (e.g. setting an int, string etc).
- Register: use when operation is going into something (e.g. registering a CLI flag inside a command).
Tests:
- Prefer got, want over alternatives like have, want (official reference).

ℹ️ NOTE

Refer also to https://github.com/kettanaito/naming-cheatsheet

Whitespace

The go standard library has no strong conventions or idioms for how to handle whitespace. So try and be concise without leaving the user with a wall of text to digest. Additionally, you can use block syntax {...} to help group related logic:

// Simple code is fine to condense the whitespace.
if ... {
  foo
  for x := range y {
    ...
  }
  bar
}

// Complex code could benefit from some whitespace (also separate block syntax for grouping related logic).
if {
  ...

  {
    ...grouping of related logic...
  }

  ...
}

Quick note on Code Design

Not always obvious but be wary of returning concrete types when building a package to be used as a library.

Here is an example of why this might be problematic: we had a library that defined a constructor that returned a struct of type *T. This struct had methods attached and inside of those methods were API calls.

The reason the returning of that struct was a problem was because when we built a separate CLI to consume the package library, we realised our CLI’s test suite wasn’t able to mock the returned type appropriately as some of the fields on the struct were private (these would determine if an attached method would make an API call), and so we were forced to make real API calls!

The solution was for us to return an interface. This made it simple to mock the behaviours we wanted (e.g. we could write our tests to pretend there was an API error, and see how our CLI handled that scenario).

I recommend reading my other post “Thinking about Interfaces in Go”.

Quick guide to Errors

When you wrap errors your message should include:

A pointer to where within your method the failure occurred.
Values that will be useful during debugging (e.g ids).
(sometimes) Details about why the error occurred.
Other relevant info the caller doesn’t know.

And your message should NOT include:

The name of your function
Any of the arguments to your function
Any other information that is already known to the caller

Here is a BAD example where the caller of a function that fails is seeing duplicate information:

// Source
func MightFail(id string) error {
    err := sqlStatement()
    if err != nil {
        return fmt.Errorf("mightFail failed with id %v because of sql: %w", id, err
    }
    ...
    return nil
}

// Caller
func business(ids []string) error {
    for _, id := range ids {
        err := MightFail(id)
        if err != nil {
            return fmt.Errorf("business failed MightFail on id %v: %w", id, err)
        }
    }
}

The resolution to the above bad code is: only include information the caller doesn’t have. The caller is free to annotate your errors with information such as the name of your function, arguments they passed in, etc. There is no need for you to provide that information to them, as its obvious up front. If this same logic is applied consistently you’ll end up with error messages that are high-signal and to-the-point.

See also the article “When life gives you lemons, write better error messages”, from which the following images are sourced.

Bad error message:

Good error message:

Essentially, the error “message” shouldn’t necessarily be formatted like “error doing x” or “failed to do x” (as I had been led to believe the former was the standard way of writing an error message). After reviewing “good” examples of Go code (e.g. at the time of writing this was one such example) it seems there is no ‘format’ for the error message other than to be direct/specific.

Quick guide to `panic`

The use of panic is reserved for when an error is unrecoverable.
What constitutes an “unrecoverable” error is contentious. Here are some definitions:
- To indicate that something impossible has happened, such as exiting an infinite loop.
- During initialization, if the library truly cannot set itself up, it might be reasonable to panic.
- When something internally has fundamentally failed.
- When a programmer gives something to a function which the function explicitly states is invalid.
bytes.Truncate is an example of the last sub-point.
- The above example could be considered aggressive.
- Instead the standard library could have returned an error so the caller could decide the appropriate action to take.
The use (and conditions) of panic should be documented (example: bytes.Truncate)
The use of recover is for when you disagree with the library authors.
Wherever possible avoid panic and return an error for the caller to handle.

Quick guide to slice ‘gotchas’

The first gotcha to be aware of is that in Go assigning a new value to a parameter with = won’t affect the argument in any way. So consider the following simple code that tries to append a value to a slice…

package main

import "fmt"

func addTwo(s []int) {
    s = append(s, 2)
}

func main() {
    mySlice := []int{1}
    addTwo(mySlice)
    fmt.Println(mySlice)
}

This will print [1], which is not what we want.

Now if you already understand that a slice is really a pointer to an internal struct type, then this might make this even more confusing because you would be of the understanding that you can modify a argument if it’s passed as a pointer.

But in this case, although a slice really is just a pointer to a struct, you have to remember that is an implementation detail and so you still have to explicitly define the parameter as a pointer type and pass it as such…

package main

import "fmt"

func addTwo(s *[]int) {
    *s = append(*s, 2)
}

func main() {
    mySlice := []int{1}
    addTwo(&mySlice)
    fmt.Println(mySlice)
}

The above version of the code will now correctly print [1 2].

When taking a slice of a slice you might stumble into behaviour which appears confusing at first. The cap, len and data fields might change, but the underlying array is not re-allocated, nor copied over and so modifications to the slice will modify the original backing array.

ℹ️ NOTE

There are more examples/explanations in https://blogtitle.github.io/go-slices-gotchas/

Ghost update 1

The underlying array is modified after updating an element on the slice as there is no re-allocation of the underlying array:

a := []int{1, 2}
b := a[:1]     /* [1]     */
b[0] = 42      /* [42]    */
fmt.Println(a) /* [42, 2] */

It’s likely you’ll want to set the capacity when taking a slice of a to assign to b. This will cause a new backing array to be created for the b slice:

a := []int{1, 2}
b := a[:1:2]   // [1]
b[0] = 42
fmt.Println(a) // [42, 2]
fmt.Println(b) // [42]

ℹ️ NOTE

Refer to the golang language specification section on “full slice expressions” syntax ([low : high : max]) for controlling the capacity of a slice.

Ghost update 2

When data gets appended to b (a slice of the a slice), the underlying array has enough capacity to hold two more elements, so append will not re-allocate. This means that appending to b might not only change a but also c (a slice of the a slice).

a := []int{1, 2, 3, 4}
b := a[:2] /* [1, 2] */
c := a[2:] /* [3, 4] */
b = append(b, 5)
fmt.Println(a) /* [1 2 5 4] */
fmt.Println(b) /* [1 2 5]   */
fmt.Println(c) /* [5 4]     */

The ‘fix’, like shown earlier, is b := a[:2:2] which sets the capacity of the b slice such that append will cause a new array to be allocated. This means a will not be modified, nor will the c slice of a.

Quick guide to pass-by-value vs pass-by-pointer

Reference articles: goinbigdata.com and dave.cheney.net and alexedwards.net.

You’ll commonly hear people use the phrase ‘pass-by-reference’. The behaviour this phrase describes is: “You’re not receiving a copy of the thing being passed, you’re getting direct access to it”.

In Go this behaviour is called ‘pass-by-pointer’. Whereas the phrase ‘pass by reference’ is actually a very specific type of behaviour (not supported in Go), and it’s not the same thing as ‘pass-by-pointer’.

To understand pass-by-pointer we first thing to understand how arguments are passed to a function.

All the following primitive/basic types in Go are passed as a value (i.e. copied):

array
boolean
float
int
string
struct

Whereas maps, slices, and channels are all passed by pointer.

Now, here’s where I need to be more specific (and accurate):

In Go every function operates on a copy of the arguments passed into the function. No exceptions, that is what happens for every type.

With the primitive/basic types, their value is copied.

So why is it that maps, slices and channels are passed by pointer?

Well, maps, slices and channels are all pointers (you might not have realised that!). When you create an instance of one of these types, the Go language actually instantiates an internal struct (e.g. a runtime.hmap, runtime.slice or runtime.hchan) and returns a pointer to them.

A pointer is something that points to a memory address.

As we’ve already said, Go will always pass a copy of an argument so Go doesn’t pass a pointer. Just like the primitive/basic types, it will create a copy of the pointer and pass that. This still means the receiver can deference the copy of the pointer its given, to get at the underlying memory address (because the underlying address is still the same, even if the pointer is a copy).

Now going back to a phrase that people commonly use when talking about Go: pass-by-reference. Go does not have pass-by-reference semantics because Go does not have ‘reference variables’, which is something you’d find in C++.

In C++ you can define:

a = 10

Then you can ‘alias’ b to a:

&b = a

In C++ this would mean updating b would affect a. Go doesn’t have this behaviour.

In Go, every variable is stored in its own memory space. Meaning, if we had b := &a and updated b then we wouldn’t cause any change to a.

In Go, imagine we define a function with a parameter p which is of a pointer type:

func changeName(p *Person) {
    //
}

Now imagine we pass a pointer to that function:

changeName(&person)

We now know that Go will not pass the &person pointer, but a copy of the pointer.

So, if the changeName function were to modify the p argument variable it receives, this would actually cause the person variable to be updated. This happens because although the &person pointer was copied into the function argument, the &person and p are two different pointers to the same struct which is stored at the same memory address. This is quite different to C++’s reference variables.

Quick guide to functions with large signature

Your functions should have concise/relevant arguments passed in.

Don’t, for example, pass in an argument whose type is a large and deeply nested object. Firstly, this means the consuming function has to know the structure well enough to dip into it (and arguably it could be argued that this violates the Law of Demeter). Secondly, it makes testing such a function tedious, and thirdly managing such a data structure is equally tedious. Instead choose a field from the object to pass in as it’ll likely have a simpler type (like a string or int).

There are four approaches to dealing with functions that potentially could have a large number of arguments…

Make multiple functions to help reduce the number of arguments.
Pass in a <T>Options struct.
Variadic arguments that accept a func type (aka “functional options pattern”).
Chaining methods on the configuration object (aka “builder pattern”).

I would say go with option 1 whenever possible, and almost never choose option 2 over option 3 as the latter is much more flexible.

The problem with option 2 is that it can become quite cumbersome to construct an object with lots of fields, and more importantly it can be hard to know which fields are required and which are optional. Yes it’s nice that you can easily omit optional fields easily, but then option 3 also provides that benefit while also solving the problem of knowing what arguments are required vs optional.

Using option 3 can be helpful when you want to make the function signature clear, by accepting a couple of concrete arguments that are required for the function to work, while shifting optional arguments into separate functions, as demonstrated below…

// Client is the complex type that needs to be constructed.
type Client struct {
  host string
  port int
}

// Option is a function that is passed a pointer to the Client type.
// This is so the function can modify the Client.
type Option func(*Client)

// WithPort returns an Option
func WithPort(port int) Option {
  return func(c *Client) { c.port = port }
}

// NewClient will set the 'host' while attempting to call all 'Option' functions.
func NewClient(host string, options ...Option) *Client {
  c := &Client{host: host, port: 80} // default values
  for _, option := range options {
    option(c) // apply the options by calling each one of them

    // Essentially, for this example code, there is noly one Option passed.
    // The Option being called is: 
    // func(c *Client) { c.port = port }
  }
  return c
}

c := NewClient("httpbin.org") // only 'host' is required 
c := NewClient("httpbin.org", WithPort(443)) // WithPort is called and returns an Option type

The downside to option 3 is that it’s hard to know the With* functions exist as they live at the package level and aren’t directly associated with the *Client type.

So option 4 helps with that:

package main

import (
    "fmt"
)

// Client is the complex type that needs to be constructed.
type Client struct {
    host string
    port int
}

// WithPort returns a mutated *Client type.
func (c *Client) WithPort(port int) *Client {
    c.port = port
    return c
}

// NewClient returns the initial *Client type with required host set and the
// optional port set to port 80.
func NewClient(host string) *Client {
    return &Client{host: host, port: 80} // default value for port
}

func main() {
    c := NewClient("httpbin.org")
    fmt.Printf("%#v\n", c) // &main.Client{host:"httpbin.org", port:80}
    c = NewClient("httpbin.org").WithPort(443)
    fmt.Printf("%#v\n", c) // &main.Client{host:"httpbin.org", port:443}
}

I don’t like the “chaining” pattern because it reminds me of jQuery (which made this approach famous in the world of JavaScript) but it’s hard to deny that the methods are definitely more discoverable than the functional options pattern.

GitHub Actions

Wed, 01 Dec 2021 00:00:00 +0000

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub.

I’ve been using GitHub Actions a lot recently and I’ve found it to be immensely flexible and feature rich. I think it’s well worth your time learning how to run your CI/CD pipelines via GitHub Actions, and in this post that’s exactly what we’re going to dig into.

Terminology

It all starts with a ‘workflow’.

A workflow is a yaml configuration file that defines:

Jobs: a job represents a collection of ‘steps’.
Steps: each ‘step’ does something useful (e.g. executes a piece of code).
Events: an event determines when your ‘jobs’ should run.

GitHub has a nice visualisation of this…

ℹ️ NOTE

Each job you define will run in parallel. If you need jobs to run sequentially, then you’ll need to configure a job to depend on another job using the needs property (we’ll see an example of this later).

Documentation

I would strongly suggest bookmarking the official documentation because it’s very thorough. Unfortunately it’s so thorough that it can be a bit overwhelming, but don’t worry, once you’ve gotten familiar with the various concepts you’ll start to remember where the important information is located.

The pages I use the most are:

Expressions: explains how to use GitHub’s builtin functions, like contains(), fromJSON(), and functions like success() and failure() that tell you the state of previous steps that have run.
Contexts: there are lots of contextual objects your jobs can use, such as objects for getting information about git (what branch we’re dealing with, the commit SHA etc), environment variables, secrets, data exposed by other jobs and lots more.
Syntax: this is the most important page as it details all the yaml configuration syntax (so I come here often).
Commands: when executing a shell command (or script) you can use echo with a specific format and, depending on the format used, it’ll take on special meaning to the workflow runner and can be used (among other things) for displaying rich messaging in the GitHub UI.
Environment variables: most of the ‘context’ properties are also exposed as environment variables to make it easier for your shell commands/scripts to utilise.
Reusing workflows: If you have a bunch of jobs all doing a similar thing (e.g. you’ve three jobs and each of them setup a specific set of environment variables before installing the rust programming language), then you can move that duplication into a separate workflow file that your main workflow can call out to.

Experimenting

The best thing to do when it comes to learning GitHub Actions is to create a test repo as a playground, like I did here: https://github.com/Integralist/actions-testing.

You can of course use an existing repo if you want, but I prefer to use something completely decoupled when I’m testing a new tool.

Workflows

As I mentioned earlier: everything starts with a workflow.

To create a workflow you must save a yaml file into the .github/workflows directory of your project’s git repo.

In the following example you’ll see I have created three separate workflows:

├── .github
│   └── workflows
│       ├── my-first-workflow.yaml
│       ├── my-second-workflow.yaml
│       └── my-third-workflow.yaml

You can create as many workflows as you need to, and you can name a workflow file anything you like.

Events

Once we create a workflow file, let’s see how we can trigger any jobs that are defined within the workflow. Be aware that in the following example workflow I have not defined any jobs, I’m focusing on the events configuration only:

name: My Workflow
on:
  push:
  schedule:
    - cron: "0 0 1 * *"   # https://crontab.guru/every-month
    - cron: "*/5 * * * *" # https://crontab.guru/every-5-minutes

So we can see I’ve given my workflow a name using the name key, and I’ve defined some events using the on key:

push: any push to your repo, whether it be to your main branch or a pull-request branch, will trigger your job(s) to run.
schedule: run your job(s) on a schedule using cron syntax, which in this example triggers job(s) every five minutes and monthly.

ℹ️ NOTE

https://crontab.guru/ makes dealing with cron syntax easy.

Refer to GitHub’s events documentation to learn more about the various events you can configure. For example, you can restrict a workflow to only execute against a specific branch.

Environment

Each job runs inside its own virtual machine ‘runner’, which means (just as an example) files you create, or data you produce, will not persist across jobs. This includes things like environment variables.

Example Job

Let’s take a look at a very simple job:

name: Test Workflow
on: push
jobs:
  simple-job:
    runs-on: ubuntu-latest
    steps:
      - name: Say Hello
        run: echo 'hello'

Let’s now break apart this workflow to understand what it’s doing…

name: the name for the workflow.
on: the event(s) we want to have trigger our job(s).
jobs: a list of jobs we want to be executed under this workflow.
- simple-job: a job that consists of nested configuration.
  - runs-on: the environment I want the job to run in.
  - steps: a list of steps to run under the job.
    - name: the name of the step, which can be omitted but it makes the output in the GitHub UI nicer.
    - run: the shell command/script I want to run (in this example, my command prints the string hello).

ℹ️ NOTE

Refer to the GitHub runner documentation to see what other environments are available. Also refer to the virtual-environments repo to see what is installed on each runner’s operating system.

As you can see, defining a workflow and its jobs/steps is actually very simple and intuitive. Now we can start looking at using more features and how to take advantage of the platform.

Environment Variables

GitHub Actions let you configure environment variables in multiple places depending on the scope they should have.

You can define environment variables globally, so they’re available to all jobs (and all steps within those jobs), or at a job level (so they can be accessed by all steps within the specific job) or at a step level, meaning only a specific step will have access to them.

I’ll demonstrate environment variable configuration in the following example:

name: Testing Environment Variables
on: push
env:
  FOO: bar
jobs:
  validate-env-vars:
    runs-on: ubuntu-latest
    env:
      LITERAL: whatever
      INTERPOLATION: ${{ github.ref_name }}
      EXPRESSION: $(echo ${{ github.ref_name }} | \
        perl -pe 's/[^a-zA-Z0-9]+/-/g' | \
        perl -pe 's/(\A-|-\Z)//g' | \
        awk '{print tolower($0)}')
    steps:
      - name: Print Global Environment Variables
        run: echo $FOO
      - name: Print Job Environment Variables
        run: |
          echo ${{ env.LITERAL }}
          echo ${{ env.INTERPOLATION }}
          echo ${{ env.EXPRESSION }}
          echo $LITERAL
          echo $INTERPOLATION
          echo $EXPRESSION
      - name: Print Step Environment Variables:
        env:
          STEPVAR: my step
        run: |
          echo ${{ env.STEPVAR }}
          echo $STEPVAR

OK, so there’s a few things to unpack from the above example workflow.

The first thing I want to clarify is that when running shell commands/scripts using steps.run you can either define multiple steps like so:

- run: echo hello
- run: echo world

Or alternatively you can use the pipe | character to indicate the value spans multiple lines, like so:

- run: |
  echo hello
  echo world

I would tend towards using separate run steps rather than one long multi-line run because it’s harder to handle errors (or know where an error occurred) in the latter approach.

The next thing to clarify is that there are two ways to access an environment variable:

Interpolation: ${{ ... }} (e.g. echo ${{ env.LITERAL }}).
Variable reference: $VARNAME (e.g. echo $LITERAL).

It’s important to understand that although the output between the two approaches is the same, there is still a distinction worth being aware of.

With interpolation the value is acquired by looking up the relevant key within the env context object, and then the value is injected into the shell command to be executed, while the more traditional variable reference approach works by the shell instance looking up the environment variable to access the value.

So when you look at the GitHub Actions GUI (e.g. https://github.com/<USER>/<REPO>/actions/runs/<ID>) and you look through the job output for the LITERAL example, then you’ll see something like:

echo whatever
echo $LITERAL

This is because the first line used interpolation, so really the shell command was just echo’ing the literal value, where as the second line was echo’ing the result of looking up the environment variable.

In our example workflow you can see we defined a global environment variable called FOO, we also defined a job level set of environment variables (LITERAL, INTERPOLATED and EXPRESSION), and finally we defined a step level environment variable called STEPVAR.

Let’s take a moment to clarify the job level environment variables as these demonstrate something important, which is although the EXPRESSION variable was assigned a shell command, that command isn’t evaluated and so the literal characters are assigned as the value.

If you thought the result of the shell command (e.g. acquire the branch name from the github context object, and then do some normalisation of the name using a combination of perl and awk) would be assigned to the environment variable, then you would have been wrong.

Only the steps.run key will actually evaluate a given shell command/script.

Now if the only way to evaluate a shell command is via steps.run, then how can we dynamically assign a value to an environment variable?

Consider this example, you’re a Node.js developer and you’re using the Node version manager nvm along with a .nvmrc file to control which version of Node your project uses.

You’re also using GitHub actions and you want to use a pre-existing action to manage installing Node on your job runner (we’ll dig into third-party actions later, but for now just know that they are a thing).

This is where things get a little funky, because third-party actions can’t evaluate shell commands given as inputs. So the most popular third-party action for installing Node is actions/setup-node and it allows you to specify the version of Node to install but it has to be a literal value.

The following example demonstrates how to side-step this restriction by using a steps.run to dynamically access the value inside the .nvmrc file and to then update the local environment with a new variable holding that value. This means you can then use interpolation to access that value and pass it into the actions/setup-node action:

name: Testing Dynamic Environment Variable Value
on: push
jobs:
  validate-env-vars:
    runs-on: ubuntu-latest
    steps:
      - name: Generate Dynamic Environment Variable
        run: echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV
      - name: Print NODE_VERSION
        run: |
          echo ${{ env.NODE_VERSION }}
          echo $NODE_VERSION
      - uses: actions/setup-node@v2
        with:
          node-version: "${{ env.NODE_VERSION }}"

The trick is to update a GitHub Actions provided environment variable called $GITHUB_ENV. All the following steps will then have an environment that includes whatever is in $GITHUB_ENV.

ℹ️ NOTE

You could also use the output of a step as the input to the actions/setup-node action, but we’ll look at that feature later.

Secrets

The previous Node.js example workflow actually leads us quite nicely into this section about secrets.

Let’s see a common solution to accessing a private NPM repository…

name: Testing GITHUB_TOKEN access restrictions
on: push
jobs:
  testing:
    runs-on: ubuntu-latest
    steps:
      - name: Acquire Node Version
        run: echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV
      - uses: actions/setup-node@v2
        with:
          node-version: "${{ env.NODE_VERSION }}"
      - name: Authorize access to private packages
        run: echo "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}" > ~/.npmrc

What we’re doing here is installing Node.js and then modifying a .npmrc file so that it knows, when we do an npm install, to use an authentication token because the repository we want to use to get at our dependencies is otherwise private.

You’ll see we’re using a secrets context object to access a GITHUB_TOKEN value to use as our authentication token. This should be fine, but it doesn’t work and our authentication with the private NPM repository fails.

Let’s take a moment to learn a bit about GitHub ‘secrets’…

In the GitHub UI for a repo/project you can add secrets that are encrypted and made accessible to your workflow via the secrets context object. But additionally GitHub Actions provides access to a secret called GITHUB_TOKEN.

I’ll just refer you to the GitHub documentation, as it explains it best:

At the start of each workflow run, GitHub automatically creates a unique GITHUB_TOKEN secret to use in your workflow. When you enable GitHub Actions, GitHub installs a GitHub App on your repository. The GITHUB_TOKEN secret is a GitHub App installation access token. You can use the installation access token to authenticate on behalf of the GitHub App installed on your repository. The token’s permissions are limited to the repository that contains your workflow.

That last sentence is the important bit! What essentially it means is that you can’t use secrets.GITHUB_TOKEN to access things outside of the project repo.

So to solve our problem we still need to use the secrets context object, but instead of using the default GITHUB_TOKEN we’re going to need to create a new Personal Access Token (PAT).

Once you create a PAT, as long as the user account that creates the PAT has access to the private repository, we can paste the PAT into the GitHub secrets UI.

For the sake of an example let’s say you create a new secret called NPM_AUTH_TOKEN and paste the PAT value into it. We can now reference the secret token value via the secrets context object in our workflow file:

name: Testing with Personal Access Token
on: push
jobs:
  testing:
    runs-on: ubuntu-latest
    steps:
      - name: Acquire Node Version
        run: echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV
      - uses: actions/setup-node@v2
        with:
          node-version: "${{ env.NODE_VERSION }}"
      - name: Authorize access to private packages
        run: echo "//npm.pkg.github.com/:_authToken=${{ secrets.NPM_AUTH_TOKEN }}" > ~/.npmrc

Notice that the workflow file basically hasn’t changed, except we swapped secrets.GITHUB_TOKEN for secrets.NPM_AUTH_TOKEN.

Third Party Actions

An action is just code that interacts with your repository. It’s possible to write your own custom actions and to share them with the community.

If you want to learn more about creating your own actions, then refer to GitHub’s “About custom actions”.

There are a bunch of third-party actions that you’ll see used a lot…

All of the above actions, with the exception of the last, are official GitHub maintained actions. This means they are considered safe to use in your workflows (remember that an action runner will be able to use your secrets.GITHUB_TOKEN). See also https://github.com/actions for more official/verified actions you can use.

ℹ️ NOTE

I’ve no idea why they don’t provide a Rust action.

Conditions

You can use an if conditional statement to control whether a job or step is run. It’s best to use this feature if you need to skip a job/step apposed to using an exit code from within a run shell command/script to stop a job/step that has already started to run.

One important caveat to using if in a workflow is that you must use single quotes and not double quotes. Consider the following example, which you might expect to only run the job if the GitHub branch affected is main…

name: Testing with Double Quotes
on: push
jobs:
  run-if-main:
    if: ${{ github.ref_name == "main" }}
    runs-on: ubuntu-latest
    steps:
      - run: echo hello

ℹ️ NOTE

Expressions can omit the surrounding ${{ ... }} but I tend to include it.

The above example won’t work simply because the string "main" is using double quotes. You’ll find the requirement for using single quotes is mentioned in the GitHub documentation for “Literals”.

If you want to learn more about the available operators, like ==, != and && etc, then refer to the operators documentation.

Persisting Data

A GitHub Action job does not persist data by default, as each job runs within its own ‘runner’. To persist data we can use either a:

Cache (actions/cache)
Artifact (actions/upload-artifact, actions/download-artifact)
Job Output (outputs documenation)

Caching is quick but isn’t always ideal as it requires the use of I/O to create files to be cached, and then also we have to coordinate reading values back from disk and parsing the data contained within those files etc. Caching is best used for simple situations such as caching installed programming language dependencies.

Artifacts are slow as they need to upload and download files from GitHub’s servers and also this requires two separate external actions to configure, making them not ideal for quick data persistence (and also makes ‘simple’ data persistence scenarios tedious).

To persist data using a job’s output requires a job to produce some data and to expose that data via the job’s outputs field. A consumer can then use and parse that data however they see fit. This approach can also be useful for dynamically generating job matrix variants using a job’s strategy.matrix field.

ℹ️ NOTE

A job has an outputs field, but also individual steps can access the output from a previous step by way of the same mechanism, which is a step needs to set an id field which either another step or the job itself can reference.

The way your run code (or an external shell script) can produce data that the GitHub job can reference is to echo a specially formatted string:

echo "::set-output name=<name>::<data>"

So in the following example we produce the data bar and we make it accessible via the name foo:

echo "::set-output name=foo::bar"

Now in the following example our GitHub job has two steps, and the latter step is accessing data produced by the first step:

name: Produce Data
on: push
jobs:
  example:
    runs-on: ubuntu-latest
    steps:
      - id: produce-data
        run: echo "::set-output name=foo::bar"
      - run: echo ${{ steps.produce-data.outputs.foo }}

ℹ️ NOTE

Be sure to use the id field so the second step can use it to reference the first step’s output!

Using shared job data to determine if subsequent job should run

In the following example the bar job will not run if the required fields foo and baz aren’t set to true.

Notice the data I produce within job foo is a simple array/list whose elements are strings who have a format of key=value:

name: Produce Data To Control Job Run
on: push
jobs:
  foo:
    runs-on: ubuntu-latest
    outputs:
      data: ${{ steps.footest.outputs.data }}
    steps:
      - run: |
          echo "FOO=true" >> $GITHUB_ENV
          echo "BAR=false" >> $GITHUB_ENV
          echo "BAZ=true" >> $GITHUB_ENV
      - id: footest
        run: echo ::set-output name=data::[\"foo=$FOO\", \"bar=$BAR\", \"baz=$BAZ\"]

  bar:
    needs: foo
    if: ${{ contains(needs.foo.outputs.data, 'foo=true') && contains(needs.foo.outputs.data, 'baz=true') }}
    runs-on: ubuntu-latest
    steps:
      - run: echo 'yay! we ran because the fields were set to true'

  build:
    needs: bar
    runs-on: ubuntu-latest
    steps:
      - run: echo 'yay! this job ran as the bar job was successful'

The key part to getting job bar to determine if it should run is to use an if along with one of GitHub’s native functions called contains(). You can see I use it twice to check if the persisted data contains the values I’m looking for.

ℹ️ NOTE

You’ll likely want to use the needs property to help make the jobs run sequentially. Otherwise without it the jobs will run in parallel and this means there would be a data race in bar trying to access the foo job’s output which might not yet be available and this would cause both the bar job and its dependant build job to not be run.

Persist data using `strategy.matrix`

Below is an example of using JSON data from job1 and exposing it to job2. This approach of using fromJSON with data from another job to create a matrix is really cool, but otherwise this approach (as far as data persistence is concerned) isn’t ideal because it requires the JSON data produced by the first job to have a very specific structure that the strategy.matrix expects. So to reiterate I’m not using strategy.matrix in this example because I want a matrix but just to demonstrate a clever way to persist data without having to resort to using either caching or artifacts.

With this in mind I needed to ensure I set the value of each matrix field to be a list type that contains a single entry.

If I didn’t use a list type then the strategy.matrix would fail to parse my JSON data. I purposely ensure there is only a single value within the list because I don’t actually need my data to cause the job to be run multiple times. This is because a strategy.matrix is typically used to generate multiple ‘variants’ of a job. By setting a single value inside a list, we ensure there is only ever one job variant generated (i.e. only one job is created), and that single job can simply reference the fields within the matrix as data points of interest.

name: example
on: push
jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - id: set-matrix
        run: echo "::set-output name=matrix::{\"FOO\":["abc"],\"BAR\":["xyz"]}"
  job2:
    needs: job1
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{fromJSON(needs.job1.outputs.matrix)}}
    env:
      FOO: ${{ matrix.FOO }}
      BAR: ${{ matrix.BAR }}
    steps:
      - run: echo ${{ matrix.FOO }} # abc
      - run: echo ${{ matrix.BAR }} # xyz

Clarify the cache action

OK, so as far as data persistence is concerned, we have the actions/cache action we can use as an option. It’s actually not very obvious how this action works and so I thought I’d take a brief moment to clarify my understanding, which then helped me to better understand how I could use it for data persistence outside of just caching language dependencies (which is what most examples are based on).

The actions/cache@v2 works like so, you have to define a new step like so:

- uses: actions/cache@v2
  with:
    path: path/to/be/cached
    key: ${{ runner.os }}-my-cache-key

When the step that implements the action is executed (see above snippet), the cache action simply looks up the cache key (e.g. Linux-my-cache-key) and if it finds something in the cache, then it restores the cache to the path you specified (e.g. path/to/be/cached).

If the cache action doesn’t find anything in the cache, then nothing happens.

Now the important bit: the cache action has a ‘post run’ event that executes once your job has finished successfully. The cache action will be run again and this time it stores whatever was in your given path into the cache using the key you said it should be stored under.

This means, when it comes to running another job, you need to ensure you define the cache action again (the same as you defined it in your first job). This is so all of what I’ve just explained will happen again in your second job (i.e. it’ll lookup the key but this time it’ll find something in the cache thanks to the ‘post run’ step from the first job). The only difference now in the second job is that in the ‘post run’ event, when the action gets run again, you’ll now see something like…

Cache hit occurred on the primary key Linux-my-cache-key, not saving cache.

Meaning there was nothing else to do. I imagine if there were changes to the files in the given path then it would indicate the cache was updated with the latest files.

Reusable Workflows

If you have a bunch of setup configuration that is the same between jobs, then you can move that configuration into a separate workflow file that can then be imported and used by each job in your main workflow file.

The following example, demonstrates how to call (i.e. import) a reusable workflow:

jobs:
  build:
    ...

  deploy:
    ...

  validate-foo:
    uses: integralist/actions-testing/.github/workflows/resuable-setup@main # install node, rust, setup env vars etc
    steps: 
      - ...

  validate-bar:
    uses: integralist/actions-testing/.github/workflows/resuable-setup@main # install node, rust, setup env vars etc
    steps: 
      - ...

An example implementation of a reusable workflow would be:

name: Reusable workflow for validation scripts
on:
  workflow_call:
    inputs:
      install_node:
        type: bool
      name:
        required: true
        type: string
      description:
        required: true
        type: string
      script:
        required: true
        type: string

jobs:
  validation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - if: ${{ github.event_name != 'schedule' }}
        run: exit 1
      - if: ${{ inputs.install_node }}
        name: Environment
        run: |
          echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV
      - if: ${{ inputs.install_node }}
        uses: actions/setup-node@v2
        id: node-yarn
        with:
          node-version: "${{ env.NODE_VERSION }}"
          cache: yarn
      - name: status update
        uses: ouzi-dev/commit-status-updater@v1.1.2
        with:
          name: ${{ inputs.name }}
          description: ${{ inputs.description }}
          status: pending
      - id: validator
        run: ${{ inputs.script }}
      - if: ${{ success() }}
        name: status update
        uses: ouzi-dev/commit-status-updater@v1.1.2
        with:
          name: ${{ inputs.name }}
          description: ${{ steps.validator.outputs.description }}
          status: ${{ steps.validator.outputs.status }}

Notice that in the reusable workflow we have a new event workflow_call that helps us to define inputs for the job being called.

In this reusable job we define a bunch of inputs which helps us to control whether the steps in the job are run (making the reusable job even more flexible) by utilising a step’s if condition and interpolating the input value.

For example, we don’t always install the Node.js programming language, only if the caller requests it:

- if: ${{ inputs.install_node }}
  uses: actions/setup-node@v2
  id: node-yarn
  with:
    node-version: "${{ env.NODE_VERSION }}"
    cache: yarn

Now one important consideration is that the reusable workflow will not inherit the parent workflow’s environment. This means secrets and environment variables need to be passed in via either workflow_call.inputs or workflow_call.secrets.

Annoyingly you can’t just set an input with ${{ env.FOO }} because the env context object can’t be referenced. So if your reusable job is used a lot then you either have to hardcode the value as an input to the reusable workflow or you store the environment variable as a secret in the GitHub UI so that you can reference it from the workflow_call.secrets property.

Conclusion

There is so much more to explore with GitHub Actions. The bits I’ve mentioned here are just the tip of the iceberg. I strongly recommend you read the documentation and have a play around with these, and other features. Let me know on twitter what you think of GitHub Actions and whether you’re using it in your projects.

Advanced Vim topics, tips and tricks

Tue, 15 Jun 2021 00:00:00 +0000

I see a lot of posts on Vim ‘tips and tricks’ and decided I’d have a go at putting together my own list of things that don’t typically see the light of day, but are super powerful and useful to know about.

❗ IMPORTANT

I want people to realise that they don’t need super complex Vim configurations with lots of third-party plugins, and this entire post is built on that motivation. This means you’ll find nearly everything described here is just plain Vim (no plugins). Don’t get me wrong, I use a few plugins, but I try to keep them to a minimum and rely more on the fundamentals of how Vim works.

[!NOTE] Remember that :help <some_phrase> is your friend! So if you don’t know about any features I mention (and I don’t explain them in this post), then use Vim’s help feature. You can also try https://vimhelp.org/ too, which is a nice UI with autocomplete search box.

Let’s take a look at what we’ll be covering…

Using Vim with no plugins

OK we’re going to start super basic here and demonstrate a simple, but completely usable, Vim configuration which will keep purists happy.

set nocompatible number cursorline expandtab hlsearch visualbell tabstop=2 shiftwidth=2
syntax on

nocompatible: Stops odd issues †.
number: Turn on line numbers.
cursorline: Highlight the current line.
expandtab: Convert tabs to spaces.
hlsearch: Highlight all search matches.
visualbell: Stop Vim from beeping at you when you make a mistake.
tabstop: Set tab size in spaces (this is for manual indenting).
shiftwidth: The number of spaces inserted for a tab (used for auto indenting).
syntax on: Enable basic syntax highlighting.

† Example: Using arrow keys in INSERT mode will send key sequences that are misinterpreted by vi.

To try out this basic configuration use Vim’s -u flag. For example, you can start Vim with no configuration vim -u NONE and then manually apply the configuration as shown above, or you can put it into a separate file and start Vim with that configuration instead of your normal one vim -u ~/.vimrc-basic.

In the above screenshot you can see I’m just using the basic configuration (looks quite nice) along with a couple of split buffer windows (:vs).

I’m also a big user of Vim’s built-in tabs feature :tabnew (not shown in the screenshot) and also :lcd for changing each tab’s root location (which allows me to easily switch between multiple projects).

In the screen shot you can see I’m also using :vimgrep to search for code in my current project (e.g. :vimgrep /func/j **/* followed with :copen).

Although not shown in the screenshot I also use the built-in file/directory explorer, AKA ‘netrw’, by executing :Ex. This let’s me manually traverse the current project directory (see also :Vex for vertical split and :Sex, no sniggering at the back, for a horizontal split). When using netrw I like to configure it to ignore certain files, which you can do like so:

" don't display .swp files
let g:netrw_list_hide= '.*\.swp$,.*\.DS_Store'

As mentioned earlier, using :h is a great way to learn about native features. As an example, I only recently discovered :cq which quits Vim without writing to any files and it will return an error exit code (this is useful in scenarios like writing git commit messages where you want to bail out at the last minute).

Another core feature of Vim I like to use is :marks which is a bookmark feature (see :h bookmark for more details). This can be useful for jumping around multiple locations within a single large file. The cool thing about marks is that they are unique to each file, so I tend to create marks using the registers a, b, c …etc as it’s easier to remember, and then I can use those same registers in each file (meaning I don’t have to jump to another file and be like “oh was I using d, e, f in this file or a, b, c?).

As well as using ! which lets me filter content through an external program. So for example, if I have the following lines…

foo
foobar
baz
quxfoo

…and I want to filter out any lines that contains foo, then I can visually select those lines and pass the selection through to my shell’s grep command like so:

:'<,'>!grep -v foo

This would result in those lines being replaced with the single line containing baz, as all the other lines were containing foo. Now a more Vim idiomatic approach to this particular problem is demonstrated in the section “Modifying content with global command”, but the takeaway is that the ! command is awesome.

Now a more practical example of using ! might be to take a single line of JSON and pipe it into a tool that pretty prints the data (the following example uses the % range to send the whole buffer to the program):

:%!python -m json.tool

ℹ️ NOTE

If you use jq you can both prettify %!jq and minify %!jq -c.

Or if you didn’t want to replace the current buffer, but instead append a pretty printed version on the line(s) after the current line, then you would use :read like so:

:read !cat % | python -m json.tool

Want to populate the Vim quickfix list with output from some shell command? No problem: Vim already supports make by default so that :make <target> will execute the specified Makefile target for you (you’ll need to manually open the quickfix window :copen).

But you can also change the default program using set makeprg=<whatever>! Now you can pass your current buffer to it using :make %.

You can also do clever things like:

autocmd BufWritePost *.go :cex system('revive '..expand('%:p')) | copen

…which executes a command (in this case revive, a golang code linter) and passes it the current file path, and opens the results up in the quickfix window!

And the great thing about all of this is that there are no plugins required. It’s all standard Vim features. You just need to know they exist.

ℹ️ NOTE

If you do decide to use plugins, then don’t forget to vim -c ":helptags ALL" -c ":q" to ensure you get all the relevant help information loaded.

Now, there may be times where you want a minimal config but with some extra treats, I’ll typically have a ~/.vimrc-core file with the following configuration that gives me the above ‘basic’ configuration (with some other configuration which isn’t essential but also isn’t superfluous either) along with some core plugins I like to use):

set nocompatible number autoread cursorline expandtab hlsearch visualbell 
set tabstop=2 shiftwidth=2 clipboard+=unnamed wildmenu hidden noswapfile
syntax on
packadd cfilter

call plug#begin('~/.vim/plugged')
Plug 'junegunn/fzf', { 'do': { -> fzf#install() } }
Plug 'junegunn/fzf.vim' " <Tab> to select multiple results
Plug 'mileszs/ack.vim'
Plug 'unblevable/quick-scope'
call plug#end()

map <leader>f :FZF<CR>
map <leader>b :Buffers!<CR>
map <leader>g :GFiles?<CR>
map <leader>w :Windows<CR>
map <leader>l :Lines<CR>
map <leader>t :AgC<CR>

set wildignore+=*/.git/*,*/node_modules/*,*/.hg/*,*/.svn/*.,*/.DS_Store
set wildmode=list:longest,list:full

autocmd VimEnter * command! -nargs=* -bang AgC \
call fzf#vim#ag(
      \ <q-args>, 
      \ '--path-to-ignore ~/.ignore '
      \ . '--hidden '
      \ . '--ignore "node_modules" '
      \ . '--ignore-dir="vendor" '
      \ . '--skip-vcs-ignores', 
      \ <bang>0
      \)

let g:ackprg = 'ag --vimgrep --smart-case '
      \ . '--path-to-ignore ~/.ignore '
      \ . '--hidden '
      \ . '--ignore-dir=node_modules '
      \ . '--ignore-dir=vendor '
      \ . '--skip-vcs-ignores'

let g:ack_mappings = {
  \ "h": "<C-W><CR>:exe 'wincmd ' (&splitbelow ? 'J' : 'K')<CR><C-W>p<C-W>J<C-W>p",
  \ "v": "<C-W><CR>:exe 'wincmd ' (&splitright ? 'L' : 'H')<CR><C-W>p<C-W>J<C-W>p"}
  
silent! tnoremap <Esc> <C-\><C-n>

Yes it’s bigger than the ‘basic’ version, but remember this is still only 30 lines of configuration and it gives me a much richer experience. Again, the motivation for this post is about not needing plugins, and you should aim for understanding the fundamentals of Vim, but this gives you a little bit of both worlds 😉

Recording repeatable steps with macros

Vim allows you to record any typed characters so that you can then replay them again. This makes applying a set of complex changes multiple times across a code base much easier.

The workflow is actually really simple…

Press q followed by any valid register to start recording (you should see something like recording @<register> in the status bar).
Start typing the changes you need to make.
Press q again to stop recording.
Press @<register> to replay the recorded steps.

I typically use the q register as it’s the quickest way to start recording a macro, as my finger already has to be over the q key to start recording any way (e.g. qq to record into q).

Now you might be wondering what happens if your steps need to include the q character (if that character is what stops a macro recording). Well luckily q only triggers the macro to stop recording if it’s the first character pressed as part of a new operation.

So to demonstrate, if I want a macro that deletes all occurrences of the letter q from the current line, then I would type the following:

qq0V:s/q//g<Enter>q

Let’s break these steps down…

qq: record into the q register.
0: move to the start of the line.
V: select the entire line.
:: start Ex mode
s/q//g: a substitution that deletes q ‘globally’ (i.e. multiple times)
<Enter>: I press the Enter key to have the substitution applied to the current line.
q: I stop recording.

If you execute :reg you’ll see the q register contains something very similar…

0V:s/q//g^M

ℹ️ NOTE

Once you’ve replayed a macro (e.g. @q) you can trigger it again without having to specify the register by typing @ again, e.g. @@ will rerun the last used macro.

If you need to run a macro a certain number of times then just prefix it with that number. For example, to run a macro six times I’d type 6@<register>.

Modifying content with `global` command

There are times when you want to execute an Ex command for any lines that match a specific pattern. That is where the :global comes in handy (this is different to substitution, which we’ll look at after).

Imagine you have a file with the following content:

foo
foobar
barfoo

You want to delete any lines that start with foo. To do that using the global command, would look like:

:g/^foo/d

ℹ️ NOTE

If you just want to see what would match, you could either use the :p command (e.g. :g/^foo/p) or just leave off the command altogether as :p is the default behaviour.

What’s cool about the :global command is that because the command can be any Ex command, it means you can also use macros by way of the :norm command. In the following example we search for foo anywhere in the content and then apply the @q register to the matches…

:g/foo/norm @q | update

With :norm it also means the first example could be mimicked with it:

# both are the same
:g/^foo/d
:g/^foo/norm dd

Now want if you needed to use controls keys like <Shift>? Well, you can do that with the :global command too, but this time the command to be called would be the :execute command. The :execute command will enable you to provide a string that will be evaluated into an Ex command.

So imagine you had a file like:

foo
bar
baz
qux

And you wanted it to be like:

foo bar
baz qux

Then you would use the following (note that <Shift-j> in normal mode causes the following line to be joined to the current line):

:g/^/exe "norm \<s-j>"

You can also use the ! to cause :global to behave in the reverse (i.e. anything that doesn’t match the given pattern, apply the command to).

Substitutions, magic regex mode and other flags

Most people know how to use Vim’s :substitute command, but it seems people are less familiar with the use of \v as a way to enable ‘magic mode’.

For me ‘magic mode’ really just means my regex pattern doesn’t require escaping characters like + or () which is quite frustrating. I just prefix my pattern with \v and I can forget that for the most part (†). more like I’d expect it to from an engine supporting PCRE (Perl Compatible Regular Expressions, probably the most common implementation).

† One caveat when using magic mode is that you do need to escape curly brackets as most regex engines treat {} as a ‘quantifier’ such as {1,3} and so if I’m programming/coding and I need to search for a { (which is common in programming languages), then I have to escape it \{. You can learn more about Vim’s regex engine via the help but also via vimregex.com.

Let’s start off by looking at \v not using a substitution but with a standard / search.

Imagine we have a project that instruments logging with different levels throughout. In this scenario imagine we have three log levels: debug, error and info that are used multiple times across the project. The log calls look something like:

log('foo', level='debug')
log('bar', level='error')
log('baz', level='info')

Lastly, imagine we want to find every instance of a log call but only those that are of the ‘debug’ and ‘error’ level, we don’t want to get any results for an ‘info’ level log call.

To do this we need a lookaround assertion. For our purposes we’re going to use a lookaround ahead (specifically a negative lookahead). The way a negative lookahead assertion works is that you provide a pattern you don’t want to be matched.

For our use case, if using a normal PCRE engine, could look something like:

level='(?!info)

In Vim this translates to:

level='\(info\)\@!

Or if you’re using ‘magic mode’:

\vlevel\='(info)@!

Notice with magic mode we don’t have to escape a bunch of things like the capture groups (i.e. the parenthesis) or the @.

ℹ️ NOTE

If the number of permutations was small enough, then it’s arguably simpler to use an alternator pipe like /\vlevel\='(debug|error) because remembering the lookaround syntax like @! (and its friends) might be hard to recall.

Next we’ll look at \u and \U (they have lowercase equivalents: \l and \L).

Remember our example log instrumentation calls from earlier, imagine we need to make the word level capitalised. Here’s how we could achieve that (note: I’ll show a better way after) using substitutions with magic mode and the \u special flag:

:%s/\v(l)(evel)/\u\1\2/

In this example we’ve used the % range to represent the entire content buffer, and we’ve used two capturing groups, one around the letter l and another capture group around the remainder of the word. In the replacement section we use \u to trigger an uppercase on the capture group \1 and then we print out the second capture group \2 following it.

Now this is actually more work than it needs to be because no matter what you capture, \u will only ever uppercase the first character, so we only really need a single capture group:

:%s/\v(level)/\u\1/

But what about the \U equivalent? That will uppercase everything that follows, so if we needed the word level to be LEVEL then that’s exactly what we’d need to use:

:%s/\v(level)/\U\1/

You could also use \U for uppercasing just the first letter, but you’d need to use either \e or \E (as a terminator) along with the two capture group design we had originally, like so:

:%s/\v(l)(evel)/\U\1\e\2/

ℹ️ NOTE

For more details refer to :h sub-replace-special, but also :h whitespace which elaborates on some other special regex pattern flags.

Searching and filtering content

The following are native Vim solutions to finding files (and also searching multiple files for specific content).

Finding a single file

To find a single file you can use the :find Ex command and pass it a wildcard glob character to help search recursively for the specified file pattern.

Example: we want to find a file called next.config.js:

:find **/next.*.js

ℹ️ NOTE

We could have just done **/next.config.js but in case you weren’t familiar with the filename, then using another wildcard like we did helps to narrow things down.

Finding content within one or more files

You have two options for locating a string within a file (or multiple files) and that’s the following Ex commands…

:vimgrep
:lvimgrep

The difference is that lvimgrep opens the results in a ‘location’ window and every open split window can have its own location window, while vimgrep opens the results in a ‘quickfix’ window and there can only be one of those shown.

Meaning if you ran vimgrep in one split window and then ran it again (e.g. you’re looking for something different now) from another window, then your first set of results would be replaced with the latter results. If you instead used lvimgrep then you could have multiple search results displayed (one for each split window).

This is the basic syntax structure:

:vimgrep  /<searchTerm>/[gj] </path/to/project/**/*.go>
:lvimgrep /<searchTerm>/[gj] </path/to/project/*>

ℹ️ NOTE

j prevents Vim from trying to open the first file match (also, if you don’t use j then the location list won’t be populated with results as it’ll presume the first match was all you wanted), while g means “ensure every match on a single line is displayed”.

Example usage (we’re searching for any reference to class anywhere in the project):

:vimgrep /class/gj **/*
:copen

For anyone unfamiliar, the copen command will open Vim’s ‘quickfix’ window, while the lopen command will open the ‘location list’ (refer to :h copen and :h lopen to find related commands).

ℹ️ NOTE

a nice trick if you’re using the append version of vimgrep (i.e. vimgrepa), is if you make a mistake populating the quickfix window, then you can use :cex [] to clear it! See :cex for details.

One interesting feature of :vimgrep is that you can use the result of a backtick expression to be the file source:

:vimgrep /ssh/j `find . -type f -name 'tmux*'`

You can also use a prior / search pattern like so:

:vimgrep /<C-r>// *

To clarify the above command, imagine you have a complex pattern you want to play around with and test with a single file so you use / to get Vim to jump into search mode for the current buffer content and then type in your complex pattern.

Once happy with your pattern, you now want to use it again for multiple files but you don’t want to have to type the pattern out again (especially in case it’s complex enough to easily include an unexpected typo).

So you type :vimgrep / and after that is where you would typically start typing your search pattern, at this point press <Ctrl-r> followed by / and Vim will automatically insert the last search pattern for you.

Imagine ... was the last search pattern, this would mean the Ex mode command would currently look like :vimgrep /... so you would need to finish the command / * (so it’s almost like you wrote the command in its entirety).

ℹ️ NOTE

if you use another plugin like :Ack! then <C-r>/ works to insert the last search pattern still (e.g. :Ack! '<C-r>/')

Using external shell tool

The :vimgrep and :lvimgrep commands use an internal Vim search implementation. Which might not be as performant as using a separate/external search tool.

This is why Vim also provides a :grep command, which allows you to utilise an external search program.

The default program it uses can be seen by running:

:set grepprg

Which should return something like the following (which is the system provided grep tool):

grepprg=grep -n $* /dev/null

I have mine set to use ag (i.e. the Silver Searcher):

set grepprg=ag\ --nogroup\ --nocolor\ --skip-vcs-ignores

ℹ️ NOTE

spaces have to be escaped with a backslash \.

You can now use the new program like so (e.g. to find any reference to the word class using the Silver Searcher tool):

:grep class
:copen

ℹ️ NOTE

you still need to open the ‘quickfix’ window manually afterwards to see the results.

All this said, you can improve the performance of :vimgrep by prefixing it with :noautocmd.

:noautocmd vimgrep /{pattern}/[flags] {file(s)}

This is because :vimgrep uses Vim’s procedures to read files, which can involve execution of several autocommands. So this disables autocommands.

Search plugins

OK, the built-in tools are great and flexible, but I’ll be honest with you and say that in my day-to-day Vim usage you’ll find me using :FZF to find files and :Ack! '<regex>' <path> to find files that contain a particular string.

ℹ️ NOTE

Although I use the Ack Vim plugin, I actually configure it to use the ag ‘Silver Searcher’ shell command.

" Plugin Managment
" https://github.com/junegunn/vim-plug#example
"
" Reload .vimrc and :PlugInstall to install plugins.
" Use single quotes as requested by vim-plug.
"
" Specify a directory for plugins
call plug#begin('~/.vim/plugged')

Plug 'junegunn/fzf', { 'do': { -> fzf#install() } }
Plug 'junegunn/fzf.vim' " <Tab> to select multiple results
Plug 'mileszs/ack.vim'

" Initialize plugin system
call plug#end()

" PLUGIN CONFIGURATION...

" FZF (search files)
"
" Shift-Tab to select multiple files
"
" Ctrl-t = tab
" Ctrl-x = split
" Ctrl-v = vertical
"
" We also set FZF_DEFAULT_COMMAND in ~/.bashrc
" Also we use --ignore-dir multiple times there
" Using --hidden to allow searching hidden directories like .github
" The --hidden still respects .ignore where we ignore things like .git
" NOTE: you need --path-to-ignore ~/.ignore otherwise ag only uses local ignore ./.ignore
"
" Note use :map command to see current mappings (also :vmap, :nmap, :omap).
" Can also restrict to specific mapping `:map <Leader>w`
" https://vi.stackexchange.com/questions/7722/how-to-debug-a-mapping
map <leader>f :FZF!<CR>
map <leader>b :Buffers!<CR>
map <leader>g :GFiles!?<CR>
map <leader>w :Windows!<CR>
map <leader>t :AgC!<CR>

" Files matched are ignored when expanding wildcards
set wildignore+=*/.git/*,*/node_modules/*,*/.hg/*,*/.svn/*.,*/.DS_Store 
set wildmode=list:longest,list:full

" configure FZF text search command to have default flags included
autocmd VimEnter * command! -nargs=* -bang AgC call fzf#vim#ag(
      \ <q-args>, 
      \ '--path-to-ignore ~/.ignore '
      \ . '--hidden '
      \ . '--ignore "node_modules" '
      \ . '--ignore-dir="vendor" '
      \ . '--skip-vcs-ignores', 
      \ <bang>0
      \)

" ack
let g:ackprg = 'ag --vimgrep --smart-case --path-to-ignore ~/.ignore \
--hidden --ignore-dir=node_modules --ignore-dir=vendor --skip-vcs-ignores'

" help Ack mappings to respect my split settings
let g:ack_mappings = {
  \ "h": "<C-W><CR>:exe 'wincmd ' (&splitbelow ? 'J' : 'K')<CR><C-W>p<C-W>J<C-W>p",
  \ "v": "<C-W><CR>:exe 'wincmd ' (&splitright ? 'L' : 'H')<CR><C-W>p<C-W>J<C-W>p"}

Now it’s probably worth me quickly mentioning that there are two small plugins I really like that’s related to this section:

vim-searchindex which shows how many times a search pattern, such as /your_pattern, occurs in the current buffer. After each search, it displays total number of matches, as well as the index of a current match.
quick-scope which highlights a unique character in every word on a line to help you more efficiently jump around on a single line using the standard f, F, t and T motions.

Processing search results with `cdo` and `cfdo`

So you’ve run a search using something like :vimgrep or :grep or maybe a plugin like :Ack, and you’ve populated a ‘quickfix’ window (or maybe a ‘location list’ if using something like :lvimgrep) with search results. Now you want to apply some sort of change to each search result.

This is where the <c|l>[f]do commands come in handy:

quickfix:
- cdo
- cfdo
location list:
- ldo
- lfdo

The first thing you need to think about is whether you want your ‘action’ to be applied to each ‘entry’ in the list or to each ‘file’ in the list. Think about the list of results you have… depending on what you searched for it’s possible you’ll see the same file appear multiple times because there were multiple entries found within the file.

So commands like cdo and ldo will operate on each entry, while cfdo and lfdo will only operate on each file. Meaning if a file appears 10 times, then you’ll have your action applied to the file just once rather than having the action applied to each entry.

Which command type you choose will depend on what your ‘action’ is intending to achieve.

Let’s consider an example scenario such as having a quickfix window containing two files:

example1.txt
example2.txt

The file example1.txt shows up multiple times, while example2.txt only shows up once.

The file example1.txt shows up multiple times because we searched for a phrase such as foo and that phrase happened to appear multiple times within example1.txt, while it only appeared once within example2.txt.

If you wanted to replace foo with bar using a subtitution like s/foo/bar/, and you used cdo, then all occurrences of foo would be replaced (i.e. across both example1.txt and example2.txt) because the substitution would be executed across each entry in the quickfix window.

But if you used cfdo then the substitution would only be applied once to each file, which would result in example2.txt having its only instance of foo replaced with bar while example1.txt would have only one instance of foo replaced with bar and the other nine instances left unchanged. This is because we didn’t provide a ‘range’, in this case %, in our substitution operation (e.g. :%s/foo/bar/ meaning apply the substitution across the entire buffer).

You could still use cfdo but you would need to specify %.

Be careful with tools that update the quickfix dynamically

The summary of this subsection is: be aware of any active plugins/tools that might dynamically update your quickfix window while your cdo command is still running.

To elaborate: I’ve found that my quickfix window is updated frequently/dynamically when using certain build tools.

For example, vim-go using gopls will update the quickfix list every time a file is written to. This is because it wants to display an updated list of files that might potentially contain broken code. That’s great when I’m generally developing on a go program, but not when I’m in the middle of trying to apply a set of changes globally using cdo.

The reason this causes problems when trying to apply changes is because cdo gets confused when it attempts to process the next quickfix entry but discovers the quickfix list of results has changed!

One way I could work around this is by using a location list, but most tools I use put results into a quickfix window so that’s not going to help me.

In this case I’ve found I’m better off using cfdo with %s/foo/bar/e | update which will write the buffer once, rather than the multiple times when using cdo with s/foo/bar/e | update. It’s also much more efficient using cfdo as it won’t write the buffer multiple times.

Your mileage may vary, but it’s important to think about this if you have certain build tools active that might mess around with the quickfix window half way through you processing the current list of results.

Let’s now look at an example where we execute a substitution for every ‘entry’ listed in the quickfix window:

:cdo s/foo/bar/ | update

You don’t need to pipe to update. I just do that because, most of the time, when I modify a file it’s because I want to save the changes 🙂

Now, to execute a macro introduces some interesting things to think about. For example, macros only execute once so if you need them to be executed multiple times, then you need to tell them to execute across a ‘range’ (e.g. the entire buffer or a section of lines, like %norm! @q). Having a macro run multiple times is fine when you’re dealing with content that is grouped together sequentially, but when wanting to apply a macro on a ‘search pattern’, then there’s no guarantees that the search results will be in the required order/grouping.

This is why I always use cdo when needing to apply a macro across multiple files (you’ll notice I also need to use execute so I can evaluate my expression norm @q as an Ex command):

:cdo execute "norm @q" | update

Filtering quickfix and location list results

What’s interesting about ‘quickfix’ and ‘location’ lists is that you can further filter their results. Now, admittedly if you’re using a third-party plugin like Ack, then yes you do have more control over the search pattern using additional flags provided by a third-party plugin that will help to filter the number of results, but I’d argue there are still situations where you need even more fine grained control over what’s displayed in the quickfix/location lists.

If you’ve taken the purist (i.e. no plugins) approach, then using the built-in Vim search (e.g. vimgrep/lvimgrep or even a custom configured grep to use an external command) will not offer much in the way of configuring the filtering of results (at the end of the day these built-in tools are useful but limited).

So what additional tools does Vim provide to us? In this case you can utilise either Cfilter or Lfilter to filter your search results. These two commands are internal Vim plugins that need to be loaded using Vim’s packadd command (I actually have this added to my .vimrc as I always forget to call packadd).

Here is an example of how to use Cfilter. I’m going to search for the phrase “vim” across my blog, which ends up returning quite a few results across multiple files. Turns out I’m only interested in files that are Markdown files and so I need to filter the results to only show me those files…

ℹ️ NOTE

Sure you could use wildignore (or maybe even suffixes, although it won’t solve our problem, only alleviate it slightly) but it’s really a hammer solution, where we want a scalpel.

:vimgrep /vim/ **/*
:packadd cfilter
:Cfilter /\.md$/

You’ll see I’ve used / as delimiters around the regex pattern, but you can also use other single-byte characters (refer to :h pattern-delimiter or the online help).

OK, so what’s interesting here is that Cfilter/Lfilter will search (using a regex pattern) both the filename and the text entry itself. So trying to filter for just \.md would still include more files than just Markdown files because the content of the file is also searched and I have .txt files that include both vim and the filter pattern .md.

This is why I also use the regex anchor $ because it’s less likely for me to accidentally hit a match, and so I’m more likely to just get back results that are Markdown files containing the phrase vim.

As you can see the default behaviour of the filter is to keep only those results that match your pattern. If you want the filter to work in reverse so it removes anything that matches your pattern, then add ! to the command like so :Cfilter! /your_pattern/.

ℹ️ NOTE

As this is a regex pattern you can still use \v to get ‘magic’ mode. For example /\vyour_pattern/.

Processing files with `<T>do`

We’ve already seen :cdo and :cfdo, but there’s so many more ‘do’ commands that can enrich your Vim experience:

:bufdo: apply action to all buffers (:h ls).
:tabdo: apply action to all tabs.
:windo: apply action to all windows (multiple windows might all display the same buffer).
:argdo: apply action to all arguments in the :h arglist.

Now admittedly I mostly use :cdo/:cfdo over any of the above because they are, for me, the most practical tools that give me massive value in my day-to-day work. That said, I do like to use :bufdo as well (I don’t think I ever use :tabdo or :windo to be honest, although it’s good to know they exist in my toolbox), and the action I do the most when using :bufdo is when I want to clear out a bunch of buffers. So I’ll run :bufdo bd (:h bd).

I’ll give you another example of how you might want to use something like :bufdo which came in handy for me recently. I had a git project which caused a bunch of conflicts in my files when I had rebased another branch. So I opened up all the conflicting files so I could locate the specific git syntax <<< HEAD that surrounds where the conflicts were.

At this point I went through each file and was fixing the issues, but then I realised I wasn’t sure if I had fixed all occurrences of <<< HEAD in each file I had edited. I guess I could have just run vimgrep over the whole project, but it’s a large project and it would have taken a lot longer than I would have liked because of the fact it was a very large project.

the solution in this case was to search over the files that I had open using :bufdo mixed with the :execute command and the append specific version of vimgrep (i.e. vimgrepa):

:bufdo execute "vimgrepa /<<< HEAD/ %"

As you can see I was also able to utilise the % register to access the current file name. So for each iteration of bufdo through the buffer list, the % register would be updated to reflect the name of the file.

Each of these ‘do’ commands are all fairly self-explanatory, but let’s take a moment to look at argdo as it has some interesting extra considerations. When you open Vim for the first time there is an ‘argument list’ that gets populated. All files open in the argument list will match what’s in your buffer list, unless you start opening/closing buffers. But regardless of changes to the buffer list, the arglist will stay the same, which means just because you have a file name in the argument list doesn’t mean you have a buffer open for that file.

That last bit is important because it can be a reason for using :argdo over :bufdo. Typically they produce the same end result because when you open Vim with three files given as arguments, then the arglist will contain those three file names, but there will also be three buffers opened with those files as content.

You can’t accidentally add a new argument to the arglist because you have to explicitly call :argadd (for deleting you need :argdelete). Where as with the list of buffers, that gets updated whenever you open a new file (and that is very easily done, especially when searching for content), but importantly the arglist will stay unchanged.

Automating content modification using Ex commands from the shell

Vim has a nice feature where it allows you to easily run Ex commands against a file from the shell. The summary of which is:

$ vim -E -s some_file.txt <<-EOF
  :%s/foo/bar/
  :update
  :quit
EOF

But stick around for a few more details…

Vim doesn’t handle stdin like other posix commands:

$ echo foo | vim

Vim: Warning: Input is not from a terminal
Vim: Error reading input, exiting...
Vim: Finished.

If you pass - to Vim, then it will accept the stdin and copy it to a new buffer…

$ echo foo | vim -

Before we look ahead at how to handle stdin a bit better, let’s consider the + flag which tells Vim what line to start on (the following example tells Vim to jump to line 10):

$ vim ~/.vimrc +10

This will become relevant when we look at two other flags -e and -s (:h -e, :h -s and :h -s-ex)…

$ echo foo | vim - -es +'%p' +'qa!'

Vim: Reading from stdin...
foo

When using the -e and -s flags, we’re able to use + to execute Ex mode commands.

ℹ️ NOTE

If you don’t use +'qa!' then Vim will cause the terminal to hang. You also need the ! otherwise qa would (if dealing with a traditional Vim UI) show a message saying the buffer has been edited and can’t be quit.

To avoid the Vim: Reading from stdin... message we need an additional flag --not-a-term:

$ echo foo | vim - -es +'%p' +'qa!' --not-a-term

foo

So now if we want to manipulate the content (let’s say uppercase the word foo to FOO) we can do:

$ echo foo | vim - -es --not-a-term +'norm VgU' +'%p' +'qa!'

FOO

ℹ️ NOTE

norm says execute the following characters as if the user is typing them, so V selects the entire line and gU uppercases the selection. We then print the output to stdout %p and then quit without trying to save the modifications.

Starting Vim with your last workspace

Now this feature I’m going to describe I don’t use that often, but there’s been a few occasions where it’s saved me a lot of hassle. The feature i’m going to talk about is :mksession. There are lots of things you can do with :mksession so read the help documents, but I’m only going to cover the basics here because that’s mostly all I ever need to use.

The summary of this Ex command is that it will take a snapshot of your working environment inside Vim and will reproduce it the next time you startup Vim.

The way it does this is to stick every buffer, window, window size and layout configuration into a Session.vim file (don’t forget to add this file to your .gitignore).

So if you need to stop Vim suddenly (for whatever reason) and you don’t want to lose your workspace layout because you’ve got lots of files open that you’d have to write down filename/paths to, and maybe you have specific window splits setup to work on multiple files in a specific order and you don’t want to have to remember what the layout was (especially if like me you have multiple Vim tabs open and each one has its own window splits etc) then you just need to run :mksession before closing Vim.

To start Vim with the session snapshot you would run:

vim -S Session.vim

That’s it! Your entire workspace is spun up exactly how you left it.

Once you open Vim again you might want to change some of your workspace. If that’s the case and you want the Session.vim file to record that updated layout in its snapshot, then run the :mksession! command (notice the extra ! at the end to force overwriting the original Session.vim file).

Vim’s start-up process

The Vim documentation explains all the various steps that are gone through during ‘start-up’, see :h startup.

In short, Vim executes :runtime! plugin/**/*.vim meaning any directories listed in the runtime path (:h runtimepath) will be searched for a plugin sub-directory and all files ending in “.vim” will be sourced (in alphabetical order per directory).

If you want to see what’s in your runtime path you can execute:

:set runtimepath?

ℹ️ NOTE

if you want to debug the start-up process: vim --startuptime some_log_filename.

To learn more about the various directories Vim uses, then refer to the :help documentation, for example:

~/.vim/autoload/... (:h autoload)
~/.vim/plugin/... (:h plugin)
~/.vim/ftplugin/... (:h ftplugin)
~/.vim/after/... (:h after-directory)

Although I will take a brief detour through that last item…

The `after` directory

The after directory can be used by both Vim ‘users’ and by Vim ‘plugin authors’ to override specific plugin configuration (that could be either ~/.vim/plugin/... or ~/.vim/ftplugin/...).

For example, the Vim plugin author for vim-polyglot adds this file: ~/.vim/plugin/vim-polyglot/after/ftdetect/rspec.vim which overrides the filetype configuration for rspec files.

Where as a Vim user might want to override the behaviour of a plugin they’re using (e.g. the FZF plugin) by adding the file ~/.vim/after/plugin/config/fzf.vim, and due to how Vim loads ‘after’ scripts, that file would get loaded. Although it’s important to add a guard into the code to ensure it only executes if the FZF plugin actually is loaded (otherwise this after script could cause an error)…

" include guard; quit if fzf isn't loaded
if ! exists(':FZF')
    finish
endif

Debugging Vim issues

First thing first, run :se to see all the Vim options that have changed from the Vim defaults. This can be a good ‘at a glance’ perspective on what might be wrong or why you’re seeing some sort of unexpected behaviour.

To check a specific setting and who (i.e. which plugin or script) last modified it, use :verbose set <setting>?.

For example, :verbose set shiftwidth? returns…

shiftwidth=2
      Last set from ~/.vimrc

You can also see what mappings have been configured using the map command.

For example, to see all mappings with the leader key…

:verbose map <leader>

x  \y            :Buffers<CR>
        Last set from ~/.vimrc
   \t            :FZF<CR>
        Last set from ~/.vimrc
        
n  \z            :ALEPrevious<CR>
        Last set from ~/.vimrc
n  \x            :ALENext<CR>
        Last set from ~/.vimrc

ℹ️ NOTE

see :h map-listing for the various modes (n = normal, x = visual, etc).

The same principle works with other mappings like <Ctrl-k> and <Ctrl-j…

:verbose map <c-k>

n  <C-K>         <Plug>MoveLineUp
        Last set from ~/.vim/plugged/vim-move/plugin/move.vim
v  <C-K>         <Plug>MoveBlockUp
        Last set from ~/.vim/plugged/vim-move/plugin/move.vim

:verbose map <c-j>

n  <NL>          <Plug>MoveLineDown
        Last set from ~/.vim/plugged/vim-move/plugin/move.vim
v  <NL>          <Plug>MoveBlockDown
        Last set from ~/.vim/plugged/vim-move/plugin/move.vim

If you want to know what the default commands that Vim defines, then take a look at :h index and drill down into the various commands per ‘mode’.

ℹ️ NOTE

Vim also has a debugger you can use: vim -D ~/.vimrc (see reference below for details).

Lastly, there is the -V<N> flag that sets the verbosity of Vim output when starting up…

" >= 1  When the viminfo file is read or written.
" >= 2  When a file is ":source"'ed.
" >= 5  Every searched tags file and include file.
" >= 8  Files for which a group of autocommands is executed.
" >= 9  Every executed autocommand.
" >= 12 Every executed function.
" >= 13 When an exception is thrown, caught, finished, or discarded.
" >= 14 Anything pending in a ":finally" clause.
" >= 15 Every executed Ex command (truncated at 200 characters).

ℹ️ NOTE

see :h vbs for details.

Usage example: vim -V9 ~/.vimrc, but you can also write the output to a log file instead (pro tip: use the log file approach) such as vim -V9foo ~/.vimrc which will write the output to the log file foo.

OK, time for the short sections… 🙂

Autocomplete with no plugins

So you want some form of auto-complete feature without the need for a third-party plugin, no problem…

filetype plugin on
set omnifunc=syntaxcomplete#Complete

Now you can execute <C-x><C-o> to get a very basic form of native code autocompletion.

Vim also provides the native command <C-n> for autocompletion based on words existing in all opened buffers.

Understanding line feed and carriage returns

This isn’t necessarily a Vim specific thing but it used to catch me out all the time. Looking for line breaks and inserting line breaks are two different things.

The term CRLF refers to Carriage Return (ASCII 13, \r) Line Feed (ASCII 10, \n). They’re used to note the termination of a line, however, dealt with differently in today’s popular Operating Systems.

Windows: CRLF (\r\n)
Linux/Unix: LF (\n)

Imagine in Vim you have a buffer like:

a
b
c
d

If you wanted to add an extra line space between each line, so it looked like:

a

b

c

d

You would need to use a substitution like:

:%s/\n/\r\r/

Notice we’re looking for a ‘line feed’ \n (because that’s how macOS denotes a new line), while to get Vim to insert a line break we need it to insert two separate ‘carriage returns’ \r.

Auto highlighting keywords (and creating your own custom highlighting)

Vim can highlight certain words inside of code comments, such as…

BUG (Golang)
FIXME
NOTE
NOTES (Python)
TODO
XXX

The NOTE works in both Go and Python files and yet it’s not defined in the Go syntax file, which means it’s likely inherited from a default syntax file.

See the syntax files for…

Python
Go

You can add your own, see this StackOverflow post for the full details, but in summary it looks something like:

augroup myTodo
  autocmd!
  autocmd Syntax * syntax match myTodo /\v\_.<(TODO|FIXME).*/hs=s+1 containedin=.*Comment
augroup END

highlight link myTodo Todo

Sorting and filtering duplicates

This is such a small ‘tip’, and is probably obvious to most Vim users, but just in case it isn’t…

Imagine you have some content that consists of a list that might also contain some duplicates, like:

foo
bar
baz
foo
bar
baz

To sort the contents you just need:

:sort

Resulting in:

bar
bar
baz
baz
foo
foo

While if you wanted to remove the duplicates too you would pass u:

:sort u

Resulting in:

bar
baz
foo

Miscellaneous tips and tricks

If you need your cursor to always be in the middle of the screen (this is modifying the ‘scroff offset’ option):

:set so=999

Move the cursor to the top, middle, bottom of the visible screen:

H: high (i.e. top of the screen)
M: middle
L: low (i.e. bottom of the screen)

Conclusion

So there you have it, my collection of useful Vim features that I use regularly. Let me know what you think. Anything surprising? What do you use that maybe I’ve missed?

Thanks for reading! ❤️

Rust Ownership, Borrowing, and Lifetimes

Sun, 28 Mar 2021 00:00:00 +0000

I’ve been learning Rust recently. This will probably be my third (lazy) attempt to learn the language. The reason I’ve failed previously is simply because I had no reason to learn it.

Other than the memory safety aspects, which I like a lot, I don’t actually like the design of the language at all (but that’s a conversation for another day).

This time around I want to learn the language as it’s pertinent to my job. A few months ago I started working for Fastly as a Staff Software Engineer in their Developer Relations team, and their powerful Compute@Edge platform has strong support for Rust.

I want to start building things on the Compute@Edge platform, so here I am giving Rust another go (at least until TinyGo support matures).

So I know from prior experience the real learning curve involved with this language is going to be its memory management model, which is broken down into three sections:

Understanding the first section (ownership) requires an understanding of ‘stack’ vs ‘heap’ memory, which you may be unfamiliar with depending on your programming experience and if you’ve used only high-level programming languages. So here’s a quick rundown…

Stack Memory

The stack is memory that is available to your program at runtime. Data that is assigned to variables or passed as arguments to function calls are allocated onto the stack in a ‘Last In, First Out’ (LIFO) model, much like a stack of plates.

The only type of data that can be stored on the stack is data that has a known and fixed size. All other ‘unknown’ data must go onto the heap.

The stack is very fast to access data because the data is close together, unlike heap memory, and also because the data that is stored is literal (i.e. primitive types like string literals, booleans, integers etc) it means the data itself can be hard coded into the compiled Rust binary.

ℹ️ NOTE

Rust primitive/scalar types (int, bool, float, char, string literal etc) are stored in stack memory.

Heap Memory

The heap is memory that is also available to your program at runtime but is much slower to access than stack memory. This is because it requires a ‘pointer’ to locate the stored data, of which the storage area is large and very unorganized.

The heap is for memory that grows or has unknown size (such as when accepting user input, you don’t know what data will be provided), and it will be allocated onto the heap by a ‘memory allocator’ that needs to first find a space in the heap large enough to hold the data, and then return a pointer to that space in the heap.

ℹ️ NOTE

**Rust complex types (String, Box etc.) are stored into heap memory.

Ownership

The rules for ownership are quite simple:

Data is assigned to a variable.
The variable becomes the ‘owner’ of the data.
There can only be one owner at a time.
When the owner goes out of scope, the data will be dropped.

Primitive types are popped from stack memory automatically when they go out of scope (e.g. when a function block ends), while complex types must implement a drop function which Rust will call when out of scope (to explicitly deallocate the heap memory).

So here are some of the gotchas that trip people up:

Primitive types are copied (because it’s cheap to copy stack memory).
Primitive types have a Copy trait that enable this behaviour.
Complex types move ownership.
Complex types do not have a Copy trait.

As an example, consider the following code, which compiles correctly because we’re dealing with primitives and so the println!() macro used is safely able to reference both the variable a and b.

fn main() {
    let a = 123;
    let b = a;
    
    println!("a: {}, b: {}", a, b); // a: 123, b: 123
}

Now consider a similar example which doesn’t work because we’re dealing with a complex type (String). The value assigned to a is moved to b. The b variable has now become the new owner of the data, and this means a is not allowed to be used again (e.g. we can’t reference it in println!()).

fn main() {
    let a = String::from("foo");
    let b = a;
    
    println!("a: {}, b: {}", a, b); // COMPILER ERROR
}

This will generate the following compiler error:

error[E0382]: borrow of moved value: `a`
 --> src/main.rs:5:30
  |
2 |     let a = String::from("foo");
  |         - move occurs because `a` has type `String`, which does not implement the `Copy` trait
3 |     let b = a;
  |             - value moved here
4 |     
5 |     println!("a: {}, b: {}", a, b);
  |                              ^ value borrowed here after move

The only solution here is to manually copy the value using the .clone() method of the String type, which means b no longer becomes the new owner of the data (the data itself is duplicated and so it’s new data that b is the owner of):

fn main() {
    let a = String::from("foo");
    let b = a.clone();
    
    println!("a: {}, b: {}", a, b); // a: foo, b: foo
}

Using clone() will duplicate the heap memory, which isn’t cheap. This forces the programmer to opt into this more expensive behaviour, and is a performance/design decision most users of high-level programming languages don’t often have to think about.

It’s important to realise that passing a variable to a function will either move or copy (just as assignment does), and that even a function’s return value can transfer ownership in the same way. So for example, returning a complex type will move ownership to the caller (and the variable the returned value is assigned to becomes the new owner).

In that scenario the complex type’s drop function is not called, as would normally be the case if a variable went out of scope at the end of a function (even if the original variable/owner was created within the function) as the compiler is able to determine that the value shouldn’t be dropped and instead is being moved to a new owner somewhere else in your program.

Borrowing

The concept of borrowing is designed to make dealing with ownership changes easier. It does this by avoiding the moving of owners. The way it does this is by letting your program provide a ‘reference’ to the data. This means the receiver of the reference (e.g. a function, struct field or a variable etc) can use the value temporarily without taking ownership of it.

To pass a reference instead of passing over ownership, all you have to do is prefix your variable with an ampersand:

fn main() {
    let s = String::from("foo");
    
    borrow(&s) // pass a 'reference' to s
}

fn borrow(s: &String) { // accept a 'reference type'
    println!("s: {}", s);
}

In the above example, the s argument variable of the borrow function will be created on the stack, and will point to data in heap memory. When the borrow function finishes and the s variable is popped off the stack it won’t delete any data because it never owned the data that s was pointing to.

The next thing you’ll likely want to do in your programs is to have functions that borrow data to be able to mutate it. That’s simple enough by using the mut keyword.

Notice in the below example code we not only define the main function’s s variable to be mutable but we also have to change the reference in the call to borrow() as well as the borrow function’s signature to accept a mutable reference.

Also notice that after borrowing the value, we call take_ownership() and we don’t pass a ‘reference’, meaning the function is the new owner of the data that was belonging to s:

fn main() {
    let mut s = String::from("foo");
    
    println!("s: {}", s); // s: foo
    
    borrow(&mut s);
    
    println!("s: {}", s); // s: foobar
    
    take_ownership(s);
    
    println!("s: {}", s); // COMPILER ERROR (ownership of s was moved)
}

fn borrow(s: &mut String) {
    s.push_str("bar");
}

fn take_ownership(s: String) {
    println!("s: {}", s); // s: foobar
}

It’s also important at this point to understand that defining a variable as being ‘mutable’ and passing a ‘mutable reference’ are two different things. You can see in the below example code that we have said s is mutable and then we pass it as an immutable reference to borrow_no_mut() and then as a mutable reference to borrow_with_mut():

fn main() {
    let mut s = String::from("foo");
    
    borrow_no_mut(&s);
    borrow_with_mut(&mut s);
    
    println!("s: {}", s) // foobar
}

fn borrow_no_mut(s: &String) {
    println!("s: {}", s) // foo
}

fn borrow_with_mut(s: &mut String) {
    s.push_str("bar");
}

Gotchas

You can’t take a reference and then modify the original variable’s value.

Here’s an example that doesn’t compile:

fn main() {
    let mut x;
    x = 42;
    let y = &x;
    x = 43;
    println!("{:?}", y);
}

In the above example we define x and assign the value 42. Next we define y and take a reference to x (i.e. we ‘borrow’ it). Lastly we try to reassign a new value to x while still holding a reference to it, which isn’t allowed because it would potentially invalidate the reference.

This would result in the following compiler error:

$ cargo run
   Compiling chapter1 v0.1.0 (/Users/integralist/Code/rust/rust-for-rustaceans/chapter1)
warning: value assigned to `x` is never read
 --> src/main.rs:5:5
  |
5 |     x = 43;
  |     ^
  |
  = note: `#[warn(unused_assignments)]` on by default
  = help: maybe it is overwritten before being read?

error[E0506]: cannot assign to `x` because it is borrowed
 --> src/main.rs:5:5
  |
4 |     let y = &x;
  |             -- borrow of `x` occurs here
5 |     x = 43;
  |     ^^^^^^ assignment to borrowed `x` occurs here
6 |     println!("{:?}", y);
  |                      - borrow later used here

For more information about this error, try `rustc --explain E0506`.
warning: `chapter1` (bin "chapter1") generated 1 warning
error: could not compile `chapter1` due to previous error; 1 warning emitted

To solve this problem you need y to fall out of scope (or alternatively create a function so that when the function is called with a reference the reference drops out of scope at the end). This can be done using block scope syntax like so:

fn main() {
    let mut x;
    x = 42;
    {
        let y = &x;
        println!("{:?}", y); // 42
    }
    x = 43;
    println!("{:?}", x); // 43
}

Another gotcha:

You can have only one mutable reference (i.e. this prevents data races).

…unless! the scope allows for it.

So here is an example where it isn’t allowed:

fn main() {
    let mut s = String::from("foo");
    
    let a = &mut s;
    
    borrow(a);
    
    let b = &mut s;
    
    borrow(b);
    
    println!("a: {}", a); // COMPILER ERROR
    println!("b: {}", b);
}

fn borrow(s: &mut String) {
    s.push_str("bar");
}

To make this example work we need the scope rules to allow for it, which means moving the first mutable reference assignment into its own block where the end of the newly defined block’s scope will cause a to be dropped:

fn main() {
    let mut s = String::from("foo");
    
    {
        let a = &mut s;
        borrow(a);
        println!("a: {}", a); // foobar
    } // <-- a is dropped
    
    let b = &mut s;
    borrow(b);
    println!("b: {}", b); // foobarbar
}

fn borrow(s: &mut String) {
    s.push_str("bar");
}

Another gotcha:

You cannot have a mutable reference and an immutable reference.

Here’s an example where the compiler complains:

fn main() {
    let mut s = String::from("foo");
    
    let a = &s;     // immutable reference
    let b = &mut s; // mutable reference
    
    borrow(b);
    
    println!("a: {}", a); // COMPILER ERROR
    println!("b: {}", b);
}

fn borrow(s: &mut String) {
    s.push_str("bar");
}

Multiple immutable references are safe because you’re only able to read the data and not mutate it, but you cannot have an immutable reference while also holding a mutable reference because this otherwise could change the value of the immutable reference (and that would be unexpected for the part of the program using the immutable reference).

The only way this would be allowed is if the immutable reference goes out of scope before the mutable reference(s) were assigned:

fn main() {
    let mut s = String::from("foo");
    
    {
        let a = &s;
        println!("a: {}", a); // foo
    } // <-- immutable reference `a` is dropped
    
    let b = &mut s;
    borrow(b);
    println!("b: {}", b); // foobar
}

fn borrow(s: &mut String) {
    s.push_str("bar");
}

One last gotcha:

You can’t return a function defined variable as a reference.

Here’s what that might look like:

fn main() {
    let r = return_ref();
    println!("r: {}", r); // foo
}

fn return_ref<'a>() -> &'a String {
    let s = String::from("foo");
    return &s; // COMPILER ERROR
}

ℹ️ NOTE

the <'a> and &'a syntax will be explained in the next section called “Lifetimes”.

The above code will cause the following compiler error:

error[E0515]: cannot return reference to local variable `s`
 --> src/main.rs:8:12
  |
8 |     return &s; // COMPILER ERROR
  |            ^^ returns a reference to data owned by the current function

If you look at the compiler explanation for the error (rustc --explain E0515) it describes the reason, and the solution:

Local variables, function parameters and temporaries are all dropped before the end of the function body. So a reference to them cannot be returned. Consider returning an owned value instead.

So the solution to this problem is to ‘move’ ownership of the data to whoever is calling the function, like so:

fn main() {
    let r = return_ref();
    println!("r: {}", r); // foo
}

fn return_ref() -> String {
    let s = String::from("foo");
    return s;
}

In the above example the r variable is now the new owner of the data.

Lifetimes

Lifetimes are tightly coupled to ‘references’.

A ‘lifetime’ is how long a reference lives for, and the compiler wants to be sure that any reference that is currently active doesn’t refer to data that no longer exists (i.e. a ‘dangling pointer’).

The compiler needs a way to track a reference so it can be sure the reference lives long enough for no errors to occur. To achieve this goal, the Rust compiler uses a “borrow checker” to validate a reference’s lifetime.

So how do we identify the lifetime of a reference? Well, it begins when the reference is created and it ends when the reference is last used (you might have expected it to be when the reference was dropped/out of scope, which would be incorrect).

What does a lifetime look like? It’s just an annotation that has a specific naming convention: '<T> where <T> is a letter like 'a or 'b. The letters don’t mean anything special, they’re just a way for the compiler to track a reference.

The Rust online book has a good example of this, where they define a function that accepts two arguments x and y (both of type &str, i.e. a string literal) and depending on the length of the given strings you’ll get back either the x or y string.

Without the lifetime feature this would be a problem for the compiler because it wouldn’t be able to statically determine which variable (x or y) is going to be returned. That can only be determined at runtime not compile time.

The below code example highlights how defining a single lifetime called 'a and assigning it to both arguments (and to the return value) allows the compiler to track these references and ensure they both live long enough to prevent any errors at runtime.

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}

The above code states all the references in the signature must have the same lifetime, and it tells the borrow checker it should reject any values that don’t adhere to these constraints.

This means that if either of the arguments x or y don’t live long enough to be used safely, the compiler will let you know about it.

Here is an example that demonstrates the potential error when the code is poorly designed:

fn main() {
    let string1 = String::from("a very long string");
    let result;
    {
        let string2 = String::from("short string");
        result = longest(string1.as_str(), string2.as_str());
    }
    println!("The longest string is {}", result);
}

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}

The above code will cause the following compiler error:

error[E0597]: `string2` does not live long enough
 --> src/main.rs:6:44
  |
6 |         result = longest(string1.as_str(), string2.as_str());
  |                                            ^^^^^^^ borrowed value does not live long enough
7 |     }
  |     - `string2` dropped here while still borrowed
8 |     println!("The longest string is {}", result);
  |                                          ------ borrow later used here

Conclusion

Although I’m very new to these concepts that Rust defines, I get the feeling that although understanding them at a high-level (as I’ve described them in this post) is reasonably easy. Being able to fully appreciate them and more importantly getting comfortable with them is just something that’s going to take time.

Let me know if this helped you. Reach out to me on twitter.

Reflection in Go

Mon, 17 Aug 2020 00:00:00 +0000

I’m going to walk you through how to understand reflection in Go using the reflect package. We’ll do this by looking at an open-source package I created called go-flags that utilizes reflection heavily.

The use of reflection is often frowned upon because it side steps the ‘compile time’ type safety we get in Go. So I’ll also explain how we can still ensure a level of type safety even at runtime.

What is reflection?

Reflection in computing is the ability of a program to examine its own structure, particularly through types; it’s a form of metaprogramming. It’s also a great source of confusion.
– The Laws of Reflection

The above quote comes from the opening of the Go Blog post The Laws of Reflection and I strongly recommend you read it, as it breaks down how types and interfaces work in Go (which is a prerequisite to understanding reflection) before leading into how reflection itself works.

There will be a summarized version of that information here, but ultimately what differentiates their post from mine is that I’ll be discussing examples that are more practical and based on a real project (e.g. go-flags).

Understanding Interfaces in Go

In Go an ‘interface type’ is a collection of methods which can be defined like so:

type Foo interface {
    Bar(s string)
    Baz(i int)
}

If you define a variable to be of type Foo, then you’re saying the value that is assigned to the variable can be any type that implements the methods as defined by the Foo interface.

Conceptually the variable containing the assigned value can be thought of as a tuple that contains the underlying concrete value and the concrete type.

This is best demonstrated in an example. Below we create an interface type Foo, and then we define a struct type that will implement that interface. We’ll assign an instance of the struct to a variable defined as being an interface type of Foo and we’ll inspect the code at runtime using the reflect package to see what it can tell us about the variable and what it contains:

package main

import (
    "fmt"
    "reflect"
)

type Foo interface {
    Bar(s string)
    Baz(i int)
}

type T struct{
    s string
    i int
}

func (t T) Bar(s string) {
    //
}

func (t T) Baz(i int) {
    //
}

func main() {
    var f Foo
    f = T{"testing", 123}

    fmt.Printf("(%+v, %+v)\n", reflect.ValueOf(f), reflect.TypeOf(f))
}

The output of the above program is:

({s:testing i:123}, main.T)

OK, so we can see that the ‘value’ assigned to the f variable (of interface type Foo) is a struct with the relevant fields assigned with the appropriate value types. We can also see that the ‘type’ of the value is main.T, which is correct and expected.

So this clearly demonstrates that although we’ve stated the variable f should hold an interface type of Foo, it’s actually holding a different type entirely.

This probably sounds like common sense (and as such not that interesting) but if you really think about it, that’s incredible. We’ve not explicitly declared the T struct as implementing the Foo interface.

Instead the fact it does implement the interface, and doing so is being dynamically understood by the compiler is a super powerful feature.

Why do we need Reflection?

You need reflection whenever dealing with information that wasn’t known at the time of the program being written. One such example might be that you need to populate a struct with configuration data that comes from a external file.

There are many other examples, and the one I’ll be using today is similar in that I wanted to populate a struct with values provided by a user at runtime, specifically CLI flag values.

The building blocks of Reflection

So let’s consider what are probably the two primary reflection methods people learn about first when dealing with reflection (these were used in our example code earlier):

reflect.ValueOf(): returns a reflect.Value.
reflect.TypeOf(): returns a reflect.Type.

Both returned types have a bunch of methods associated with them that are unique to the purpose of those types. So for example, you want a reflect.Value if you’re interested in querying the concrete value, where as reflect.Type is more useful when you need to know information about the specific type.

So the use case for getting back a reflect.Type vs reflect.Value depends on what you’re trying to ascertain at runtime.

Once you have either a reflect.Value or a reflect.Type, then you can start digging further down into the data that is exposed via these types.

This is where exploring a real-world use case can be handy, as it helps to clarify why certain methods on these objects are called (compared to using a more contrived example).

Package Walk-through: Go-Flags

If you’ve ever written a CLI program in Go you’ll inevitably use its flag package. Most people find this package unintuitive and in some cases just downright difficult to work with or to build nice patterns of use from.

This leads people to build their own abstraction patterns on top of the standard library, but from my experience the majority (if not all) of third-party flag packages are convoluted and confusing to work with. The user experience feels very poor IMHO.

ℹ️ NOTE

I’m not knocking these packages. They all do very complex things, and solve real problems, and are written by much smarter people than myself. But it doesn’t change the fact that rarely do they come across as ‘simple’ and ‘easy to use’.

I wanted to solve the problem of handling flags and commands for a CLI based program in a simple way, that didn’t require me to construct a whole bunch of boilerplate code and have to negotiate lots of types.

In essence I wanted to define a ‘schema’ of what flags and commands (and flags for those commands) that I was expecting, and to have a package magically populate that struct with the values provided by a user of my program.

This is where utilizing reflection was the only real solution to my problem (if I indeed wanted the user experience to be as simple as possible).

So thus go-flags was born.

Here’s an example of how you would use it:

package main

import (
    "fmt"
    "os"

    "github.com/integralist/go-flags/flags"
)

type Schema struct {
    Debug   bool   `short:"d" usage:"enable debug level logs"`
    Number  int    `short:"n" usage:"a number field"`
    Message string `short:"m" usage:"a message field"`
    Foo     struct {
        AAA string `short:"a" usage:"does A"`
        BBB string `short:"b" usage:"does B"`
    }
    Bar struct {
        CCC bool `short:"c" usage:"does C"`
    }
}

func main() {
    var s Schema

    err := flags.Parse(&s)
    if err != nil {
        fmt.Printf("error parsing schema: %v\n", err)
        os.Exit(1)
    }
}

This is very simple and concise from a user’s perspective.

We can see that there are a bunch of top-level, non command specific, flags defined:

-d/-debug
-n/-number
-m/-message

You’ll notice that we utilize struct tags (the backticks that follow the struct field’s type) for defining the ‘short’ flag (while the field name itself is defining the ‘long’ flag variation) as well as defining the ‘usage’ description for each flag.

We can also see that we’ve defined two separate commands (Foo and Bar), and each command has its own set of flags:

foo: -a/-aaa (string), -b/-bbb (string).
bar: -c/-ccc (bool).

An example of how a user might then run this program would be:

my_cli_app -debug -n 123 -m "something here" foo -a beepboop -b 666

Explaining how go-flags uses Reflection

So how exactly does go-flags achieve this magic using reflection?

Let’s step through the code and find out…

ℹ️ NOTE

I’m not an expert on the reflect package, nor was a lot of time spent on this package outside getting it functional, so there’s likely many improvements that can be made to the code.

We’ll start with the Parse() function. Here we can see that the function accepts an argument of type interface{} which is the empty interface.

The reason for this is because we want the schema for the flags to be defined by the consumer of this package, and so at runtime we have no idea what that value will look like (hence the use of interface{} to mean: we’ll take any type).

Once we’ve got this unknown value we’ll execute the following code to get at the actual value:

v := reflect.Indirect(reflect.ValueOf(s))

We can see the familiar reflect.ValueOf() but we have also wrapped it in a call to reflect.Indirect(), so why do that? Well, the Indirect() method will dereference the struct pointer to the struct concrete value!

It’s also safe to use Indirect() because if you give it a non-pointer value it’ll just return the value from reflect.ValueOf(). Don’t worry, in just a moment we’ll do some validation of the given argument to ensure the user has followed our package instructions and indeed given us a pointer to a struct containing their flag schema.

Next we acquire a reflect.Type type from the given argument:

st := v.Type()

We get the type structure from the reflect.Value type returned from calling reflect.Indirect(), and we do this because later on we’ll pass it into a function called IterFields which iterates over the user’s schema struct. We pass it into that function because within there we’ll try to get the struct’s individual fields as a reflect.StructField (I’ll explain why we want that later).

Now it’s at this point we do some runtime type validation:

if v.Kind() != reflect.Struct {
  return ErrWrongType
}

We’re expecting a struct and there’s no point trying to go any further with something that’s not a struct, so we short-circuit the program if we’re given anything unexpected.

OK, at this point of our package we have a few stages that involve, among many things, iterating the given struct (we use a custom iterator function called IterFields and we pass it a callback function to execute for each ‘field’ in the given struct).

Here are the stages at a high-level:

iterate over the top level fields of the user provided struct, and dynamically create flags.
parse the flag values the user of the CLI program have provided (using the flags our package has now dynamically generated).
iterate over the top level fields of the user provided struct, and populate the fields with the parsed flag values.
identify the command the user provided when running the CLI program (e.g. foo or bar).
parse the flag values the user provided after specifying the command.
dynamically create a flag.FlagSet for the command.
parse the flagset values the user provided (using the flagset our package has now dynamically generated).
iterate over the command fields of the user provided struct, and populate the fields with the parsed flagset values.

Now we understand the general steps taken, we can dig into each of those and understand what reflection work needs to be carried out.

I won’t be discussing the general code logic, just picking out the bits related to reflection. So if you want to understand the full go-flags implementation, then I recommend you read through the code base in its entirety after finishing this post.

ℹ️ NOTE

if I was working with Go 2.0 then I’d have access to generics and parts of this code could be reduced/simplified. Hello future readers who are lucky enough to have Go 2.0.

As the majority of these steps are related to iterating over the user provided struct, let’s start with the IterFields function that enables that.

First thing we do in IterFields() is we call .NumField() on the reflect.Value, which gives us back the number of fields the struct contains.

This will panic if the reflect.Kind() is not a reflect.Struct type, hence why at the beginning of our Parse() function we do that validation there.

We could have moved the check for a struct down into the IterFields() function but then we’d end up doing extra unnecessary logic processing only to have to just stop the program any way (so best to short circuit the code as soon as possible).

Once we have the number of struct fields, we’ll create a loop for that number and we’ll use the incrementing i value to access each individual struct field by calling .Field() on the v variable’s reflect.Value type.

ℹ️ NOTE

you’ll find that when you call methods on a reflect.Value type you’ll likely end up with …another reflect.Value type!

for i := 0; i < v.NumField(); i++ {
  field := v.Field(i)

  ...
}

Next we call the .Field() method again but this time on the reflect.Type type. The reason we do this is because (as I mentioned earlier) we want a reflect.StructField type (which is what the .Field() call will give us) so we can inspect it later on in the callback function and parse out the ‘tags’ assigned to the current struct field.

We have at this point a if/else condition which I’m going to skip over the if block briefly and move onto discussing the else block which states:

if !recurse && field.CanSet() {
    callback(field, sf)
}

You’ll see we call .CanSet() on the field variable which (as noted earlier) is actually a reflect.Value type. The reason we do this is because one of the given callback functions will attempt to set a value onto the struct field and that is only valid if the struct field is public (i.e. exported) otherwise it would panic.

Let’s jump back up to the if statement, in there we’re checking the condition:

if field.Kind() == reflect.Struct {
  ...
}

This is a similar check to what we did at the start of the Parse() function, so why are we doing it again? Well, this time (as we iterate over the user’s schema struct and look at each field) we might discover one of the fields is itself a struct! So we want to also iterate over the nested struct’s fields looking for flag information.

All the code within this if block is effectively the same as what came before it with regards to getting access to the underlying struct ‘value’ that is assigned to the current struct’s field, and then looping over that nested structs own fields.

We could have done this with recursion but then it makes the IterFields() function more complex and as I was designing go-flags to work with only one level of nesting supported it wasn’t worth the effort to implementing a strict recursive function.

That said, there is one aspect of the if block we should look at which is the nested if block. The nested if block is what actually does the looping over the nested struct’s fields, and it does this by first getting at the nested struct concrete value (as I mentioned a moment ago).

To get at that struct value we do:

reflect.TypeOf(field.Interface())

OK, so this needs some explaining! Firstly, we call .Interface() on the field variable (which is a reflect.Value type) and then we pass that ‘result’ to reflect.TypeOf().

The .Interface() method will get the underlying concrete value as an empty interface (i.e. interface{}) type, and so that’s what we pass to reflect.TypeOf().

The reason we use .Interface() first is because if we had just passed the field variable into reflect.TypeOf(), then we would have gotten reflect.Value as the type (and that’s no use to us! we want the struct type).

So why did we need the struct type? Well, after we start iterating over the nested struct’s fields, we again need to get a reflect.StructField so that when we execute the given callback function it’ll be able to parse any ‘tags’ found on the nested struct fields.

At this point in time we’ve explained some key bits as far as looping safely over the user’s struct, so let’s go back to what we do in some of the ‘callback’ functions that get executed for each struct field.

One thing we do is access the struct field’s tags, and we do this by calling .Tag.Get() on the reflect.StructField type. So in go-flags we ask users to add struct tags like so:

short:"..." usage:"..."

That’s two tags ‘short’ and ‘usage’. So if I want the ‘short’ tag I’d call .Tag.Get("short") and if I want the ‘usage’ tag I’d call .Tag.Get("usage"). Nice and simple.

When it comes to setting the field values we have to check the kind of the field (remember this is a reflect.Value type). The following switch statement demonstrates this:

switch field.Kind() {
case reflect.Bool:
    if b, ok := getter.Get().(bool); ok {
        field.Set(reflect.ValueOf(b))
    }
case reflect.Int:
    if i, ok := getter.Get().(int); ok {
        field.Set(reflect.ValueOf(i))
    }
case reflect.String:
    if s, ok := getter.Get().(string); ok {
        field.Set(reflect.ValueOf(s))
    }
}

The call to .Kind() returns a reflect.Kind type, which is an integer iota that maps constants to a numerical value for easier comparison. Hence each case is a check against some of those constants like reflect.Bool, reflect.Int and reflect.String.

Once we know what type the field is we type assert the flag value to the relevant type and then call .Set() on the field. You’ll notice that we can’t just pass the value into the .Set() method as we need to provide an argument of type reflect.Value.

To fix that we call reflect.ValueOf() and pass it the value we’re trying to set, and then pass that resulting reflect.Value type into .Set().

Conclusion

Well, that’s it! There is sooooo much more to reflection than what I have had time to describe in this post, but hopefully this has been enough to help you in the future if you ever stumble across a need for reflection.

One thing I would say before I go is: avoid reflection wherever possible. It’s a nightmare to work with and although we can code defensively and get back some runtime safety, that’s still no comparison to compile time safety. There’s also a performance cost to runtime reflection that can’t be ignored.

So do please think twice before reaching for reflection.

Tips for Comparing Code Libraries

Sat, 18 Jul 2020 00:00:00 +0000

When writing software you will inevitably need to choose an external code library to assist you with some specific feature or behaviour.

You’re also likely to discover that there are many tools available that claim to solve the same problem. So how do you choose which one to use?

You should carefully compare the value of each individual tool that is under consideration, and to do that we typically utilize a comparative table.

Comparison Matrix

A comparison matrix table should take into consideration the following properties:

Activeness: is the repo/source code actively maintained (e.g. features, bug fixes etc).
Dependencies: how large is the dependency graph.
Design: does the interface follow golang syntax/language best patterns.
Documentation: is there sufficient documentation to support the code.
Extensibility: is it part of a larger ecosystem and is it easy to extend.
Performance: which implementation is more performant.
Popularity: which tool is more popular (based on GitHub stars, so not scientific!).
Stability: how buggy is the software (based on GitHub issues, so not scientific!). †
Suitability: how effective is it (e.g. what core APIs/features does it provide).
Usability: how easy to use it is (warning: this is subjective).

† stability: we can’t really judge the quality of issues raised, nor can we suggest that more issues is an indicator of poorly written software (e.g. a more popular tool is likely to have a larger number of issues raised compared to a tool with very few users).

But we can at least compare the number of issues that have been closed vs those that are left open and stale (e.g. a code base may be very active and still have poor user engagement by ignoring opened issues).

Although again, this is flawed because a code base with a much higher number of users (and thus a much higher number of issues raised) may not be able to review issues as quickly as a code base with very few users.

Emoji Key

We will use the following emojis to identify comparative outcomes:

✅: clear winner
❌: clear loser
⚖️: balanced results
🗑: no winner/no loser

ℹ️ NOTE

I will sometimes mark both tools as neither ‘winner’ or ‘loser’ due to the comparative value not necessarily being indicative of either. For example, the data point might just be ‘of interest’ (such as the date of when the project started).

Example

	Tool A	Tool B
Property A	✅	❌
Property B	⚖️	⚖️
Property C	🗑	🗑

Winner: Tool A

We would have these results stored inside some kind of shared document for historical purposes, and once a winner is identified, then we should declare what the proposed steps are for integrating the chosen tool (as that in itself can sometimes determine whether the tool is the winner or not, depending on deadlines and/or ‘time to market’ etc).

Rate Limiting at the CDN Edge

Wed, 08 Jul 2020 00:00:00 +0000

What is Rate Limiting?

Rate Limiting is a technique used to control the rate of requests received.

It exists to help online services to stay up and running even when clients of the service are issuing lots of requests.

Think of an API such as the GitHub API. If they didn’t rate limit their clients, then it’s very possible a single client could consume most of their resources (either accidentally or intentionally, i.e. a “bad actor”).

Problems with Rate Limiting

There are two key issues that I needed to address for my employer. The first is very specific to the needs we were trying to solve, and the other is a general issue…

our service is public, not private.
avoid overloading our internal infrastructure.

Let’s consider the first point: most exposed APIs will expect a client to provide a ‘key’ in order to access the API (this allows the API service to keep a track of individual clients and to ensure they aren’t being overly aggressive with their requests).

But I work for a well known online publisher, and so our customers are public users visiting our public website, which means we can’t rate limit using a ‘key’.

This is because we don’t know who our clients are, and so we can’t expect them to have signed up to get a key to access our content (unless our content was sat behind a paywall, which it isn’t).

This means we have to identify clients another way. The typical approach here is to identify a client by their IP address.

Identifying a client by an IP is problematic for rate limiting because you potentially will end up grouping together multiple clients who all happen to be using the same IP address (e.g. a college campus).

So what can we do to address these two issues?

Identify clients more granularly

The problem with IP identification is that it’s not granular enough. To workaround this issue we create a fingerprint of the client that’s actually a hash built from multiple contextual pieces of data:

<host>:<path>:<method>:<client-ip>:<user-agent>

This means we ultimately are rate limiting clients on a ‘path’ basis, meaning a client can request /foo enough to be rate limited for that path but still be allowed to access /bar.

But this doesn’t help us with web scrapers and fuzzers who present a potential Denial of Service (DoS) attack by hitting multiple endpoints at a very high rate (this is because the count for a single path could be low while their overall crawling impact could be much larger and damaging).

This forces us to define two types of clients:

Standard
Broad

The ‘standard’ clients are those we described originally (they hit a single endpoint at a high rate). An example of this are pentesters trying to compromise a sign-in page.

The ‘broad’ clients are those who have a less granular fingerprint (specifically <client-ip>:<user-agent>), and where we cross reference them against a list of known pentesting/scraper/fuzzer tools.

Get your CDN to help

The problem we had was that our internal services were struggling to handle the volume of uncacheable requests.

What would be better is if we could leverage the scale of our CDN provider (Fastly) rather than have our internal infrastructure front the burden.

To do this we needed a way for the CDN to be able to identify a client as one that should be rate limited. But we didn’t have enough programmatic access with Fastly to achieve this (their infrastructure is designed around Varnish which uses a specialized DSL called VCL).

I won’t explain why we didn’t have enough programmatic access with Fastly, because it requires an understanding of Fastly and their services, which is outside the scope of this post.

ℹ️ NOTE

Fastly does have a beta product called compute@edge but it’s still very early stages and they don’t recommend their customers use it for production workloads.

Architecture

With the above understanding clear in our minds, let’s take a look at the architecture I ended up designing/implementing:

Existing proxy tracks clients via Redis.
New HTTP service (written in Go):
- Periodically scans Redis.
- Identifies if client should be rate limited.
- Updates Fastly ‘edge dictionary’ via Fastly API.
Fastly CDN checks ‘edge dictionary’ and rejects clients as necessary.

Visually this looks something like…

Two things worth clarifying in the above architecture diagram…

We’ll attempt to serve a cached response from the CDN otherwise we’ll attempt to serve stale (if available), before finally falling back to a synthetic 429 Too Many Requests.
The “Perimeter” service directly behind the CDN was a pre-existing custom built reverse proxy (only slightly modified for the purpose of putting clients into redis).

Downsides

The biggest downside is that there is a delay in applying rate limits, as processing data isn’t happening at runtime (i.e. the ‘Rate Control’ service is decoupled from the perimeter proxy).

This additionally introduces some complexity in the sense that we have to limit requests per ‘minute’ rather than the more traditional ‘request per second’ (as we need to allow the external service time to process data).

Subsequently this design will allow more requests through to our internal infrastructure before rate limiting will be applied. In practice this hasn’t been an issue for us, but it’s worth calling out.

Lastly, although this works well for our purposes I appreciate it is not a perfect solution, and is merely a tourniquet.

Features

OK, so let’s review the benefits of this design…

Verifies if a client requires rate limit restriction.
- Then notifies CDN to action this behaviour.
Provides DoS level protection.
Implements granular level of client identification.
Supports identification/blocking of common scrapers/fuzzers:
Supports reduction of requests handled by our own infrastructure.
Supports reduction of additional latency.
Avoids overloading upstream proxy(s) responsibility.
Provides CLI tooling to:
- Clear a rate limited client from the Fastly edge dictionary.
- Toggle on/off rate limiting at the edge.

ℹ️ NOTE

although not done yet, I intend on building an internal UI service for ‘Rate Control’ that will make interfacing with the various components easier (inc. all features currently present in the CLI).

Example Code

Let’s see some example VCL code to understand the CDN edge implementation a bit better…

ℹ️ NOTE

for brevity I’ve removed chunks of logic so we can more easily focus in on the ‘control flow’.

sub ratelimit_trigger {
  // client is rate limited, so attempt to find cached version of the content
  //
  set req.http.X-Reject = "true";
  return(lookup);
}

sub ratelimit_recv {
  // client_key generates a key from the contextual data we're looking for
  //
  set var.client_hash = digest.hash_sha1(var.client_key);

  if (table.lookup(ratelimit, var.client_hash) == "true") {
    call ratelimit_trigger;
  }

  // if we're unable to find the current key, then before we allow the normal request flow to continue
  // we'll first check if we can find a 'bad actor' (e.g. web scraper/fuzzer) in the rate limit table
  //
  // NOTE: it's at this point we'd change the client_key and regenerate the client_hash.
  //
  if (table.lookup(ratelimit, var.client_hash) == "true") {
    call ratelimit_trigger;
  }
}

sub ratelimit_miss {
  // we were unable to find cached content, so move to vcl_error
  // where we'll attempt to serve stale if it exists, otherwise
  // we'll serve a synthetic '429 Too Many Requests'
  //
  error 601;
}

sub ratelimit_error {
  if (obj.status == 601) {
    if (stale.exists) {
      return(deliver_stale);
    }

    set obj.status = 429;
    set obj.response = "Too Many Requests";
    set obj.http.Content-Type = "text/html";
    synthetic {"<h1>Too Many Requests</h1>"};
    return(deliver);
  }
}

The above code would typically be a separate code file which we’d include in our standard VCL file and then call within the relevant state subroutines…

ℹ️ NOTE

again I’ve omitted chunks of code for the sake of brevity.

include "rate_limiting"

sub vcl_recv {
  unset req.http.X-Reject;

  call ratelimit_recv;
}

sub vcl_miss {
  if (req.http.X-Reject == "true") {
    call ratelimit_miss;
  }
}

sub vcl_error {
  call ratelimit_error;
}

Git Internals

Sun, 29 Mar 2020 00:00:00 +0000

Introduction

There are many version control systems, but git is undoubtedly the most popular, and regularly used, thanks to online social platforms such as GitHub and GitLab.

Yet, it is a tool that is still vastly misunderstood and feared. In this post I aim to take a look at some of the internal moving parts of git, primarily what’s inside the .git directory (inc the various subdirectories and files).

My hope is that by better understanding how git works, and the concepts it is built upon, readers will feel more empowered and confident when working with git (especially when they have issues and would normally be unsure of what to do).

ℹ️ NOTE

this article isn’t an introduction to git, and does presume that the reader is familiar with (i.e. a user of) git.

General Concept

I wanted to take a quick moment just to clarify the terminology associated with the general concepts of how git works (so we’re all on the same page):

Working Directory: your project files.
Staging Area: a file that tracks the changes to your project files.
Repository: the location where your project files are stored.

ℹ️ NOTE

these bullet points are just summarizations, but I would like to extend upon it slightly in that: your ‘working directory’ can change depending on what ‘version’ of the project you have ‘checked out’ from the git repository (i.e. this is what happens when you change your ‘branch’ with git checkout <branch_name>).

So for example, commands like git add will copy objects from the working directory into the staging area (aka the ‘index’), while git reset will remove objects from the staging area.

A command such as git diff compares your working directory to your staging area, while using the --staged flag will change this behaviour such that git will compare your staging area to your actual repository state.

Subcommands: Porcelain and Plumbing

The git version control system wasn’t initially designed to be a user-friendly interface, and so alongside the more commonly used subcommands are commands that can carry out very low-level operations.

This has resulted in much confusion around what commands are intended for use by general users and which commands exist for the purpose of internal use.

ℹ️ NOTE

although used internally, the low-level subcommands are also typically used by systems that require such granular operational control.

The git subcommands are generally split into one of two groups:

Porcelain: the user-friendly interface (e.g. git checkout, git pull etc.)
Plumbing: low-level interface (e.g. git cat-file, git rev-parse etc.)

Git Subcommands

Below is a list of the git subcommands (as of git version 2.22.0), and knowing which are meant to be ‘porcelain’ and which are meant to be ‘plumbing’ can be difficult.

$ man git-<tab>

git-add                       git-commit-tree               git-fsck                      
git-am                        git-config                    git-fsck-objects              
git-annotate                  git-count-objects             git-gc                        
git-apply                     git-credential                git-get-tar-commit-id         
git-archimport                git-credential-cache          git-grep                      
git-archive                   git-credential-cache--daemon  git-gui                       
git-bisect                    git-credential-store          git-hash-object               
git-blame                     git-cvsexportcommit           git-help                      
git-branch                    git-cvsimport                 git-http-backend              
git-bundle                    git-cvsserver                 git-http-fetch                
git-cat-file                  git-daemon                    git-http-push                 
git-check-attr                git-describe                  git-imap-send                 
git-check-ignore              git-diff                      git-index-pack                
git-check-mailmap             git-diff-files                git-init                      
git-check-ref-format          git-diff-index                git-init-db                   
git-checkout                  git-diff-tree                 git-instaweb                  
git-checkout-index            git-difftool                  git-interpret-trailers        
git-cherry                    git-fast-export               git-log                       
git-cherry-pick               git-fast-import               git-ls-files                  
git-citool                    git-fetch                     git-ls-remote                 
git-clean                     git-fetch-pack                git-ls-tree                   
git-clone                     git-filter-branch             git-mailinfo                  
git-column                    git-fmt-merge-msg             git-mailsplit                 
git-commit                    git-for-each-ref              git-merge                     
git-commit-graph              git-format-patch              git-merge-base                
git-merge-file                git-rebase                    git-show-index
git-merge-index               git-receive-pack              git-show-ref
git-merge-one-file            git-reflog                    git-stage
git-merge-tree                git-remote                    git-stash
git-mergetool                 git-remote-ext                git-status
git-mergetool--lib            git-remote-fd                 git-stripspace
git-mktag                     git-remote-testgit            git-submodule
git-mktree                    git-repack                    git-svn
git-multi-pack-index          git-replace                   git-symbolic-ref
git-mv                        git-request-pull              git-tag
git-name-rev                  git-rerere                    git-unpack-file
git-notes                     git-reset                     git-unpack-objects
git-p4                        git-rev-list                  git-update-index
git-pack-objects              git-rev-parse                 git-update-ref
git-pack-redundant            git-revert                    git-update-server-info
git-pack-refs                 git-rm                        git-upload-archive
git-parse-remote              git-send-email                git-upload-pack
git-patch-id                  git-send-pack                 git-var
git-prune                     git-sh-i18n                   git-verify-commit
git-prune-packed              git-sh-i18n--envsubst         git-verify-pack
git-pull                      git-sh-setup                  git-verify-tag
git-push                      git-shell                     git-web--browse
git-quiltimport               git-shortlog                  git-whatchanged
git-range-diff                git-show                      git-worktree
git-read-tree                 git-show-branch               git-write-tree

But there is a way to find out! Currently the man git page describes which commands are intended as porcelain and which are plumbing. Simple search for GIT COMMANDS and you’ll find the two groupings.

My own generalized way of making a distinction is to consider the day-to-day subcommands I use (e.g. git add, git diff) as being porcelain, while the more esoteric subcommands (e.g. git fsck, git multi-pack-index) as being more plumbing orientated.

In practice it doesn’t really matter which subcommands are porcelain and which are plumbing. If there’s a subcommand you feel you need to use, then go ahead and use it. My personal perspective on this is: if you’re ever unsure of what it is you’re doing you’re unlikely to use a subcommand.

Most users do not diverge from the well trodden path of: git add, git commit, git pull, git push, git diff (with an occasional git rebase).

What’s interesting about the plumbing subcommands is that some of them are used internally by git when you’re calling the porcelain subcommands (e.g. git read-tree, git update-index, git update-ref will be called by other porcelain commands such as git add or git commit).

ℹ️ NOTE

although we’ll be looking at a couple of plumbing commands in this article, I’ll refer you to the git book for a look at the different plumbing commands available and how they’re used.

The `.git` directory

When you start a new project that you want to use version control for, you’ll typically run the git init subcommand:

git init [dir]

Most people will know that there is now a .git directory created in the root of your project directory, but that’s about where their understanding of things stop.

Let’s see what’s initially inside the .git directory of a new project…

$ tree .git/

.git/
├── HEAD
├── config
├── description
├── hooks
│   ├── applypatch-msg.sample
│   ├── commit-msg.sample
│   ├── fsmonitor-watchman.sample
│   ├── post-update.sample
│   ├── pre-applypatch.sample
│   ├── pre-commit.sample
│   ├── pre-push.sample
│   ├── pre-rebase.sample
│   ├── pre-receive.sample
│   ├── prepare-commit-msg.sample
│   └── update.sample
├── info
│   └── exclude
├── objects
│   ├── info
│   └── pack
└── refs
    ├── heads
    └── tags

8 directories, 15 files

OK, so there’s some important directories and files here that we need to learn a bit about in order to appreciate how git works.

ℹ️ NOTE

I’m not going to explain every file and directory, only those necessary to understand the fundamentals.

Here are some interesting ones:

HEAD: contains a pointer to the tip of the current branch.
config: contains project-specific configuration options.
info: contains a global exclude file †
objects: contains four types of ‘objects’ (commit, tree, blob, tag).
refs: contains pointers to ‘commit’ objects.

† this is separate from a local user’s .gitignore.

References and Objects

The two most important concepts in git are: references and objects.

For example, your branches, tags and remotes are all references to commits. While your commits are objects, your files are objects, your directories are objects.

References

Git is built upon the simple premise of using ‘pointers’ to data, and these pointers are typically referred to as ‘references’ (or ‘refs’ for short).

This is what the .git/refs directory stores: references.

As I mentioned earlier, these references all point to a ‘commit’ object…

remote    branch     tag
  |          |        |
  |          |        |
  |          V        |
  ------> commit <-----
             |
             |
             V
           tree
             |
             |
             V
           blob

ℹ️ NOTE

you can see from the above ascii graph that the ‘commit’ object itself points to a ‘tree’ object, and that tree object points to a ‘blob’ object. We’ll dig into these reference ‘object’ types in more detail in the “Object Types” section.

It’s worth clarifying now that although we conceptually talk in terms of ‘branches’ in git, the internal directory structure (where references to branches are stored) uses the term ‘heads’ instead. It’s a terrible name (like most things in git’s lexicon), but it’s best to just accept it and move on.

The reason git uses ‘references’ is it enables users to be able to refer to a specific commit without having to remember the full SHA1 hash.

Imagine wanting to checkout your master branch but instead of just executing git checkout master you had to remember the specific hash.

git checkout b5d34b608ce697f0d20d011ee569529bca3feee8

Not very practical heh.

The HEAD reference

If you recall from earlier, we said the HEAD file contains a pointer to the tip of the current branch.

If we were to look at the .git/HEAD file we would find that by default it has the following content:

ref: refs/heads/master

You can see it’s a pointer to another location (the reference .git/refs/heads/master), which means it’s a pointer to a pointer!

Remember that refs/heads/master is a reference file (which refers to our master branch), and the contents of that file is a pointer to a commit hash. So this is telling us that ultimately HEAD is pointing to our master branch.

But at this point in time I’ve only executed git init, and so I’ve not actually committed anything into git. This means that there isn’t actually a master file inside of the .git/refs/heads subdirectory.

If we look back at the earlier directory tree (which we printed after running git init), we’ll notice that although there is a .git/refs/heads directory, there is no master file. A file called master won’t exist in that subdirectory until I make my first commit.

ℹ️ NOTE

if you recall from earlier I said that the refs/heads subdirectory was essentially a synonym for ‘branches’ created locally for this project. Hence, the default file referenced by the HEAD file is master (because it’s referencing the master branch).

Let’s now create a commit so that we can see a refs/heads/master file and what it points to…

$ echo foo > foo.txt
$ git add foo.txt
$ git commit -m "foo"

[master (root-commit) b5d34b6] foo
 1 file changed, 1 insertion(+)
 create mode 100644 foo.txt

Once we do this we’ll find git has created a master file inside of .git/refs/heads and the contents of that file is the hash of my first commit (which indicates that the master reference file, or ‘branch’, is pointing at a specific commit snapshot):

b5d34b608ce697f0d20d011ee569529bca3feee8

When you execute a command (such as) git checkout master, internally git will resolve master into refs/heads/master and that is what tells git which commit object to now point to.

Subcommands and References

Although a reference is a pointer to a commit hash, it doesn’t mean you can use a reference within a git subcommand.

Here is an example subcommand that works fine with a reference: git log. We can use git log origin/master, and git will know to internally resolve that reference to the fully qualified path .git/refs/remotes/origin/master.

Knowing that, we would also know that it is possible to use a partial reference path such as git log refs/remotes/origin/master or maybe git log remotes/origin/master.

All these variations work fine, but we typically use git log origin/master for convenience (because it’s less typing).

But using a shorted ‘reference’ isn’t possible with commands like git checkout and git pull for different reasons. With git pull if we look at man git-pull we see we need to provide a <repository> <refspec> and that means the refspec we provide will be scoped to .git/refs/remote/.

If I look at .git/refs/remote/ I’ll see only a single directory origin, and inside of that are all the branches (i.e. refspecs) for the origin remote. So if I attempted to do something like git pull origin HEAD this wouldn’t work because there’s a HEAD file inside of that origin directory (and it points to a different commit from our local HEAD in .git/HEAD)!

This means we’d end up trying to pull the changes from the remote master!! Which happens because HEAD on the remote is setup to track the master branch…

$ git remote show origin

* remote origin
  Fetch URL: git@github.com:example/repo.git
  Push  URL: git@github.com:example/repo.git
  HEAD branch: master
  Remote branches:
    ...

So subsequently doing git pull origin HEAD would bring in lots of unexpected changes to your local branch 😬

ℹ️ NOTE

using HEAD isn’t a problem when doing something like git push origin HEAD because it’s a fundamentally different operation and so git knows to reference the local HEAD file to get the commit range before pushing to the remote.

Similarly, using a shortened ‘reference’ isn’t possible with a command like git checkout as its internal logic will cause a detached HEAD state (e.g. if you were to do something like git checkout refs/heads/master instead of git checkout master).

Let’s now understand what a ‘detacted HEAD’ means, and why it is a git checkout would cause that when using a refspec…

Detached HEAD

Internally git does recognize the reference and can resolve it to the appropriate .git/refs directory, but the behaviour of the checkout command changes when checking out a reference that is a qualified path such as refs/heads/master. What you would discover is you don’t checkout the branch but are placed into a ‘detached HEAD’ at the relevant commit.

Why is that? Well, if we look at the documentation for the checkout subcommand (man git-checkout) we would discover…

if it (the given branch name) refers to a branch (i.e., a name that, when prepended with “refs/heads/”, is a valid ref), then that branch is checked out. Otherwise, if it refers to a valid commit, your HEAD becomes “detached” and you are no longer on any branch.

Running git checkout master means you’ve given an identifier (i.e. master) that git can internally resolve to refs/heads/master and thus git will happily checkout that branch, while git checkout refs/heads/master is a direct reference that git first resolves to a commit.

Hence it’s like you had actually run the subcommand git checkout <commit-hash>, and so git puts you into a detached HEAD state.

If you’re unfamiliar with what a ‘detached HEAD’ state is, then it simply means the HEAD file no longer is pointing at a reference such as .git/refs/heads/master but directly to a commit hash. The purpose of a detached HEAD is to allow you to do work off a branch.

I’ve never had a need to work ‘off’ a branch (:shrugs:) and so I can only presume there are situations where you would want to do that.

OK, now that we have our first commit let’s dig a little deeping into the ‘objects’ git defines, and how the .git directory structure has changed…

Object Types

There are four main types of objects in git:

commit
tree
blob
tag

ℹ️ NOTE

we’ll primarily be covering the first three object types.

Since we committed a single file into git there has been a few new files and directories created:

index: a binary file containing a sorted list of path names.
COMMIT_EDITMSG: temporary file used to store latest commit message.
objects/25/7cc5642cb1a054f08cc83f2d943e56fd3ebe99: the foo.txt file (type: blob)
objects/b5/d34b608ce697f0d20d011ee569529bca3feee8: commit message data (type: commit)
objects/fc/f0be4d7e45f0ef9592682ad68e42270b0366b4: directory tree (type: tree)

You’ll notice that the new objects are stored in a subdirectory which uses the first two characters from the hash of the object’s contents.

For example, the foo.txt blob object’s content was hashed into 257cc5642cb1a054f08cc83f2d943e56fd3ebe99. Next git took the first two characters 25 and made a subdirectory, and then moved the object into that directory while naming the object file using the remaining characters (i.e. 7cc5642cb1a054f08cc83f2d943e56fd3ebe99).

In order to look at these files you’ll need a couple different plumbing commands: git ls-files and git cat-files.

Let’s start with the index file.

The index is a binary file which tracks our working directory and our staging area (use --stage flag to see staging area). The index enables fast comparisons between the tree object it defines and the working tree.

We’ll need to use git ls-files in order to read the contents:

$ git ls-files

foo.txt

It only has foo.txt tracked, which is correct. There are no other files or directories at this point in time (we’ll add more as we go).

To look at the different ‘objects’ we’ll use the git cat-files command which decompresses the file and displays the file contents (we’ll use the -t flag to return the ‘type’ and the -p flag to ‘print’ the contents).

ℹ️ NOTE

we don’t provide the path (e.g. objects/../...) as the argument, but the sha itself (shortened sha is acceptable too).

$ git cat-file -t 257cc5642cb1a054f08cc83f2d943e56fd3ebe99
blob

$ git cat-file -p 257cc5642cb1a054f08cc83f2d943e56fd3ebe99
foo

$ git cat-file -t b5d34b608ce697f0d20d011ee569529bca3feee8
commit

$ git cat-file -p b5d34b608ce697f0d20d011ee569529bca3feee8
tree fcf0be4d7e45f0ef9592682ad68e42270b0366b4
author Integralist <example@gmail.com> 1585480397 +0100
committer Integralist <example@gmail.com> 1585480397 +0100

foo

$ git cat-file -t fcf0be4d7e45f0ef9592682ad68e42270b0366b4
tree

$ git cat-file -p fcf0be4d7e45f0ef9592682ad68e42270b0366b4
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99    foo.txt

What’s also interesting is that when you execute command such as git add, git will ‘conceptually’ copy the file to your staging area, but internally it has created a ‘blob’ object. While a command such as git commit then creates the ‘commit’ and ‘tree’ objects to reference the already existing ‘blob’ object. I mention this because I wanted to be clear that these three objects don’t all get created at the same time.

Snapshots, Not Differences

We saw earlier an ascii graph that indicated the hierarchy of these objects. It showed that git reference types (e.g. remotes, branches and tags) all point to a ‘commit’ object. This commit object will include a pointer to a ‘tree’ object, and the tree object is a list of files (i.e. blobs) and directories (i.e. more trees).

It’s this graph that builds up the entire snapshot of the repository. This is why you shouldn’t think of a git commit as being a patch or set of changes to a bunch of files, but instead should see each commit as a complete snapshot of your entire project at a singular point in time.

If any files or directories change, then their commit hash will change and thus the HEAD commit will consist of different tree and blob objects (resulting in a different hash-tree graph).

With that in mind, let’s start by looking at the commit object we have (git cat-file -p b5d34b6). We can see the first line says tree followed by a hash (all other information is the typical commit information you’re used to seeing when you run git status).

If we look at the tree object git cat-file -p fcf0be4 (which the commit object linked to), then we can see it consists of a single line: a blob object with its hash and its filename foo.txt (this makes sense as our project only contains this single file).

Lastly, let’s look at the blob object git cat-file -p 257cc56 (which the tree object linked to), then we can see the contents of that blob object is the contents of the foo.txt file itself.

OK, so what happens if I add a new file bar.txt and a new subdirectory baz with another file qux.txt within that subdirectory…

$ tree
.
├── bar.txt
├── baz
│   └── qux.txt
└── foo.txt

1 directory, 3 files

Once I add baz/qux.txt and commit it I then inspect the new objects in my .git/objects folder. From there I locate the commit object (I do that by looking at the .git/refs/heads/master and seeing what commit hash it has) and once I cat-file -p that hash, I follow its tree pointer…

$ git cat-file -p edc6771b338b472d901358e530db7cede202c1c7

100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6    bar.txt
040000 tree 3d15e426c95bac2548d7255af9c5e240df786e03    baz
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99    foo.txt

$ git cat-file -p 3d15e426c95bac2548d7255af9c5e240df786e03

100644 blob 100b0dec8c53a40e4de7714b2c612dad5fad9985    qux.txt

We can see from the above output that the tree object not only includes my project files, but now a baz directory (itself a tree object). Looking at that tree object shows there is one file inside of it (a blob object for qux.txt).

If we review the index file again we’ll see our new set of files/directories:

$ git ls-files

bar.txt
baz/qux.txt
foo.txt

Remotes

When you add a remote like so:

git remote add origin git@github.com:Integralist/dotfiles.git

We can now look at the configuration of our remote:

$ git remote show origin

* remote origin
  Fetch URL: git@github.com:Integralist/dotfiles.git
  Push  URL: git@github.com:Integralist/dotfiles.git
  HEAD branch: master
  Remote branches:
    linux                                new (next fetch will store in remotes/origin)
    master                               new (next fetch will store in remotes/origin)
    minimal-mac-version-of-linux-version new (next fetch will store in remotes/origin)
  Local ref configured for 'git push':
    master pushes to master (local out of date)

You might be confused though if you were to look at .git/refs and don’t see a remotes subdirectory. This happens automatically if you clone an existing repository, but it’ll also be created when executing git fetch after manually adding a new remote to an existing repository.

I added my new origin remote (see above), but it was only once I had executed a git fetch was I then able to see a ‘remote’ reference:

refs/
| remotes/
| | origin/
| | | master

If I inspect the .git/refs/remotes/origin/master file, then I’ll see the latest commit my remote master branch is on. It’s also interesting to remember what we mentioned earlier about references that point to commits being interchangeable with commit hashes in various subcommands.

For example, git diff allows you to specify two branches to compare against each other (remember a branch is just a reference file that points to a commit hash), and so you might want to compare your local master against your remote master branch:

git diff master..origin/master

This is just a shortened way of doing:

git diff master..refs/remotes/origin/master

Which itself is just a shortened way of doing:

git diff master..c3865b72b019ced930cfc601b09b874685c29e72

ℹ️ NOTE

one last thing I wanted to mention (and there was no other place really to mention this) is that git comes with a UI! you can execute the command gitk to use it.

Python 101: Context Managers

Mon, 06 Jan 2020 00:00:00 +0000

Introduction

In this post I wanted to discuss a relatively simple, but important topic: Context Managers. I want to cover the what, the why and the how.

Summary

Content Managers abstract away ‘clean-up’ logic.
Define a class with __enter__/__exit__ methods.
The __enter__ method is similar to a try block.
The __exit__ method is similar to a finally block.
Reduce boilerplate with @contextlib.contextmanager.

What is a Context Manager?

Officially…

A context manager is an object that defines the runtime context to be established when executing a with statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the with statement, but can also be used by directly invoking their methods. – Python Docs

In simpler terms it means: a Context Manager can ensure code that requires ‘clean-up’ logic to be executed, is done so in a more idiomatic/Pythonic way.

Why use a Context Manager?

The classic example given is when opening lots of file in Python:

files = []

for _ in range(100000):
    f = open('foo.txt', 'w')
    files.append(f)
    f.close()

Notice each file object’s .close() method is called to ensure the file descriptor is released. If we didn’t do that, then your Operating System would exhaust its allowed limit of open file descriptors.

To make this code more Pythonic and cleaner, we can utilize Context Managers.

How to use a Context Manager?

There are two ways to utilize a Context Manager…

with
@contextlib.contextmanager

`with`

We can use the with statement to define a similar block of code to our earlier ‘open multiple files’ example.

The with statement expects a ‘Context Manager’ to be provided, and there are already a few built-in Python objects designed as Context Managers; such as the open function we saw used in our above example code.

ℹ️ NOTE

another example is threading.Lock.

Here is what the code might look like when using with:

files = []

for _ in range(100000):
    with open('foo.txt', 'w') as f:
        files.append(f)

Notice how we didn’t have to explicitly call .close() on each file object generated by open. That’s because open works as a Context Manager and knows how to clean-up after itself when called via the with statement.

We’ll show you how to implement your own Context Manager in the following section: How to implement a Context Manager?.

`@contextlib.contextmanager`

Python provides a decorator function @contextlib.contextmanager which is actually a callable class (i.e. it defines __call__ magic method) that enables custom context managers (e.g. your own code you want to act as a context manager) to use simpler code than the traditional ‘class-based’ implementation we previously mentioned.

This means if you have custom objects that need to implement clean-up logic (similar to how open does), then you can decorate your own function so it behaves like a Context Manager, while your function itself simply uses a yield statement, like so:

from contextlib import contextmanager

files = []

@contextmanager
def open_file(path, mode): 
    file = open(path, mode)
    yield file
    file.close()

for _ in range(100000):
    with open_file('foo.txt', 'w') as f:
        files.append(f)

In the above example code we’ve effectively recreated the open Context Manager just to demonstrate the principle.

How to implement a Context Manager?

Now we’ve already seen how to implement a Context Manager using the @contextlib.contextmanager decorator (see previous sub-section), but how do we implement a class-based version of a Context Manager?

That requires us to define a class which implements __enter__ and __exit__ methods. Below is an example, again replicating the open function to keep things simple:

files = []

class Open():
    def __init__(self, filename, mode):
        self.filename = filename
        self.mode = mode

    def __enter__(self):
        self.open_file = open(self.filename, self.mode)
        return self.open_file

    def __exit__(self, *args):
        self.open_file.close()

for _ in range(100000):
    with Open('foo.txt', 'w') as f:
        files.append(f)

ℹ️ NOTE

for more information, see Context Manager Types.

When to use one or the other?

One thing I noticed recently was that the contextmanager variation wouldn’t execute an ‘exit’ if an exception was raised during execution of the code, while the more verbose ‘class-based’ implementation would. See the following code for an example…

from contextlib import contextmanager

@contextmanager
def foo():
    print("enter!")
    yield "foobar"
    print("exit!")

try:
    with foo() as f:
        raise Exception("unexpected")
        print(f"f was: {f}")
except Exception as e:
    print(f"whoops: {e}")

The output is not what I expected:

enter!
whoops: unexpected

Notice how there is no exit! printed.

Now compare this to a ‘class-based’ example…

class Foo():
    def __enter__(self):
        print("enter!")

    def __exit__(self, *args):
        print("exit!", args)

try:
    with Foo() as f:
        raise Exception("unexpected")
        print(f"f was: {f}")
except Exception as e:
    print(f"whoops: {e}")

The output is as expected:

enter!
exit! (<class 'Exception'>, Exception('unexpected'), <traceback object at 0x108882d00>)
whoops: unexpected

i.e. we see both an enter and exit message.

We can get the contextmanager to behave as we might have expected it to (e.g. the same as the ‘class-based’ implementation) by ensuring the function that calls yield is wrapped in a try/finally block, like so:

from contextlib import contextmanager

@contextmanager
def foo():
    print("enter!")
    try:
        yield "foobar"
    finally:
        print("exit!")

try:
    with foo() as f:
        raise Exception("unexpected")
        print(f"f was: {f}")
except Exception as e:
    print(f"whoops: {e}")

The output of this is now what we might expect…

enter!
exit!
whoops: unexpected

When choosing between the two options contextmanager and ‘class-based’ implementation, it might be worth keeping this caveat in mind.

Multiple Context Managers in a single With statement

One interesting aspect of the with statement is that you can execute multiple context managers as part of its block control. Meaning when the with block completes, then all context managers will be cleaned up.

from contextlib import contextmanager

@contextmanager
def foo():
    print("enter!")
    try:
        yield "foobar"
    finally:
        print("exit!")


with foo() as f1, foo() as f2, foo() as f3:
    print(f"f1 was: {f1}")
    print(f"f2 was: {f2}")
    print(f"f3 was: {f3}")

Alternatively you can utilize contextlib.ExitStack:

from contextlib import contextmanager, ExitStack

@contextmanager
def foo():
    print("enter!")
    try:
        yield "foobar"
    finally:
        print("exit!")

with ExitStack() as stack:
    managers = [stack.enter_context(foo()) for cm in range(3)]
    print(managers)  # ['foobar', 'foobar', 'foobar']

Python 101: iterators, generators, coroutines

Sat, 28 Dec 2019 00:00:00 +0000

In this post I’m going to be talking about what a generator is and how it compares to a coroutine, but to understand these two concepts (generators and coroutines) we’ll need to take a step back and understand the underlying concept of an Iterator.

Each section leads onto the next, so it’s best to read this post in the order the sections are defined. Unless you’re already familiar with earlier segments and prefer to jump ahead.

Summary

The summary of everything we’ll be discussing below is this:

Iterators let you iterate over your own custom object.
Generators are built upon Iterators (they reduce boilerplate).
Generator Expressions are even more concise Generators †
Coroutines are Generators, but their yield accepts values.
Coroutines can pause and resume execution (great for concurrency).

† think comprehensions.

Iterators

According to the official Python glossary, an ‘iterator’ is…

An object representing a stream of data.

Why use Iterators?

An interator is useful because it enables any custom object to be iterated over using the standard Python for-in syntax. This is ultimately how the internal list and dictionary types work, and how they allow for-in to iterate over them.

More importantly, an iterator (as we’ll discover) is very memory efficient and means there is only ever one element being handled at once. Thus you could have an iterator object that provides an infinite sequence of elements and you’ll never find your program exhausting its memory allocation.

Iterator Implementation

An iterator is (typically) an object that implements both the __iter__ and __next__ ‘dunder’ methods, although the __next__ method doesn’t have to be defined as part of the same object as where __iter__ is defined. Let me clarify…

An ‘iterator’ is really just a container of some data. This ‘container’ must have an __iter__ method which, according to the protocol documentation, should return an iterator object (i.e. something that has the __next__ method). It’s the __next__ method that moves forward through the relevant collection of data.

So you could design a single class that contains both the __iter__ and __next__ methods (like I demonstrate below), or you might want to have the __next__ method defined as part of a separate class (it’s up to you and whatever you feel works best for your project).

ℹ️ NOTE

the Python docs for collections.abc highlight the other ‘protocols’ that Python has and the various methods they require (see an earlier post of mine that discusses protocols + abstract classes in detail). If you’re unfamiliar with ‘dunder’ methods, then I’ll refer you to an excellent post: a guide to magic methods.

By implementing these two methods it enables Python to iterate over a ‘collection’. It doesn’t matter what the collection is, as long as the iterator object defines the behaviour that lets Python know how to iterate over it.

Iterator Example

Below is a contrived example that shows how to create such an object. In this example we pass in a list of strings to a class constructor and the class implements the relevant methods that allow for-in to iterate over that collection of data:

class Foo:
    def __init__(self, collection):
        self.collection = collection
        self.index = 0

    def __iter__(self):
        """
        we return self so the 'iterator object' 
        is the Foo class instance itself,
        
        but we could have returned a new instance 
        of a completely different class, so long as
        that other class had __next__ defined on it.
        """
        return self

    def __next__(self):
        """
        this method is handling state and informing
        the container of the iterator where we are
        currently pointing to within our data collection.
        """
        if self.index > len(self.collection)-1:
            raise StopIteration

        value = self.collection[self.index]
        self.index += 1

        return value

# we are now able to loop over our custom Foo class!
for element in Foo(["a", "b", "c"]):
    print(element)

ℹ️ NOTE

raising the StopIteration exception is a requirement for implementing an iterator correctly.

With this example implementation, we can also iterate over our Foo class manually, using the iter and next functions, like so:

foo = Foo(["a", "b", "c"])
iterator = iter(foo)

next(iterator)  # 'a'
next(iterator)  # 'b'
next(iterator)  # 'c'

ℹ️ NOTE

iter(foo) is the same as foo.__iter__(), while next(iterator) is the same as iterator.__next__() – so these functions are basic syntactic sugar provided by the standard library that helps make our code look nicer.

This type of iterator is referred to as a ‘class-based iterator’ and isn’t the only way to implement an iterable object. Generators and Generator Expressions (see the following sections) are other ways of iterating over an object in a memory efficient way.

We can also realize the full collection by using the list function, like so:

iterator = Foo(["a", "b", "c"])
list(iterator)  # ["a", "b", "c"]

ℹ️ NOTE

be careful doing this, because if the iterator is yielding an unbounded number of elements, then this will exhaust your application’s memory!

Generators

According to the official Python documentation, a ‘generator’ provides…

A convenient way to implement the iterator protocol. If a container object’s __iter__() method is implemented as a generator, it will automatically return an iterator object.

Why use Generators?

They offer nice syntax sugar around creating a simple Iterator, but also help reduce the boilerplate code necessary to make something iterable.

A Generator can help reduce the code boilerplate associated with a ‘class-based’ iterator because they’re designed to handle the ‘state management’ logic you would otherwise have to write yourself.

Generator Implementation

A Generator is a function that returns a ‘generator iterator’, so it acts similar to how __iter__ works (remember it returns an iterator).

In fact a Generator is a subclass of an Iterator. The generator function itself should utilize a yield statement to return control back to the caller of the generator function.

The caller can then advance the generator iterator by using either the for-in statement or next function (as we saw earlier with the ‘class-based’ Iterator examples), which again highlights how generators are indeed a subclass of an Iterator.

When a generator ‘yields’ it actually pauses the function at that point in time and returns a value. Calling next (or as part of a for-in) will move the function forward, where it will either complete the generator function or stop at the next yield declaration within the generator function.

Generator Example

The following example prints a, then b, finally c:

def generator():
  yield "a"
  yield "b"
  yield "c"

for v in generator():
     print(v)

If we used the next() function instead then we would do something like the following:

gen = generator()
next(gen)  # a
next(gen)  # b
next(gen)  # c
next(gen)  # raises StopIteration

Notice that this has greatly reduced our code boilerplate compared to the custom ‘class-based’ Iterator we created earlier, as there is no need to define the __iter__ nor __next__ methods on a class instance (nor manage any state ourselves). We simple call yield!

If our use case is simple enough, then Generators are the way to go. Otherwise we might need a custom ‘class-based’ Iterator if we have very specific logic we need to execute.

Remember, Iterators (and by extension Generators) are very memory efficient and thus we could have a generator that yields an unbounded number of elements like so:

def unbounded_generator():
    while True:
        yield "some value"

gen = unbounded_generator()

next(gen)  # some value
next(gen)  # some value
next(gen)  # some value
next(gen)  # some value
next(gen)  # ...

So, as mentioned earlier, be careful when using list() over a generator function (see below example), as that will realize the entire collection and could exhaust your application memory.

def generator():
  yield "a"
  yield "b"
  yield "c"

gen = generator()
list(gen)  # [a, b, c]

Generator Expressions

According to the official PEP 289 document for generator expressions…

Generator expressions are a high-performance, memory–efficient generalization of list comprehensions and generators.

In essence they are a way of creating a generator using a syntax very similar to list comprehensions.

Below is an example of a generator function that will print "foo" five times:

def generator(limit):
    for i in range(limit):
        yield "foo"

for v in generator(5):
    print(v)

Now here is is the same thing as a generator expression:

for v in ("foo" for i in range(5)):
    print(v)

The syntax for a generator expression is also very similar to those used by comprehensions, except that instead of the boundary/delimeter characters being [] or {}, we use ():

(expression for item in collection if condition)

ℹ️ NOTE

so although not demonstrated, you can also ‘filter’ yielded values due to the support for “if” conditions.

Nested Generators (i.e. `yield from`)

Python 3.3 provided the yield from statement, which offered some basic syntactic sugar around dealing with nested generators.

Let’s see an example of what we would have to do if we didn’t have yield from:

def baz():
    for i in range(10):
        yield i

def bar():
    for i in range(5):
        yield i

def foo():
    for v in bar():
        yield v
    for v in baz():
        yield v

for v in foo():
    print(v)

Notice how (inside the foo generator function) we have two separate for-in loops, one for each nested generator.

Now look at what this becomes when using yield from:

def baz():
    for i in range(10):
        yield i

def bar():
    for i in range(5):
        yield i

def foo():
    yield from bar()
    yield from baz()

for v in foo():
    print(v)

OK so not exactly a ground breaking feature, but if you were ever confused by yield from you now know that it’s a simple facade over the for-in syntax.

Although it’s worth pointing out that if we didn’t have yield from we still could have reworked our original code using the itertool module’s chain() function, like so:

from itertools import chain

def baz():
    for i in range(10):
        yield i

def bar():
    for i in range(5):
        yield i

def foo():
    for v in chain(bar(), baz()):
        yield v

for v in foo():
    print(v)

ℹ️ NOTE

refer to PEP 380 for more details on yield from and the rationale for its inclusion in the Python language.

Coroutines

Coroutines (as far as Python is concerned) have historically been designed to be an extension to Generators.

Coroutines are computer program components that generalize subroutines for non-preemptive multitasking, by allowing execution to be suspended and resumed. – Wikipedia

Why use Coroutines?

Because coroutines can pause and resume execution context, they’re well suited to conconcurrent processing, as they enable the program to determine when to ‘context switch’ from one point of the code to another.

This is why coroutines are commonly used when dealing with concepts such as an event loop (which Python’s asyncio is built upon).

Coroutines Implementation

Generators use the yield keyword to return a value at some point in time within a function, but with coroutines the yield directive can also be used on the right-hand side of an = operator to signify it will accept a value at that point in time.

Coroutines Example

Below is an example of a coroutine. Remember! a coroutine is still a generator and so you’ll see our example uses features that are related to generators (such as yield and the next() function):

ℹ️ NOTE

refer to the code comments for extra clarity.

def foo():
    """
    notice we use yield in both the 
    traditional generator sense and
    also in the coroutine sense.
    """
    msg = yield  # coroutine feature
    yield msg    # generator feature

coro = foo()

# because a coroutine is a generator
# we need to advance the returned generator
# to the first yield within the generator function
next(coro)

# the .send() syntax is specific to a coroutine
# this sends "bar" to the first yield 
# so the msg variable will be assigned that value
result = coro.send("bar")

# because our coroutine also yields the msg variable
# it means we can print that value
print(result)  # bar

ℹ️ NOTE

coro is an identifier commonly used to refer to a coroutine. For more information on other available coroutine methods, please refer to the documentation.

Below is an example of a coroutine using yield to return a value to the caller prior to the value received via a caller using the .send() method:

def foo():
    msg = yield "beep"
    yield msg

coro = foo()

print(next(coro))  # beep

result = coro.send("bar")

print(result)  # bar

You can see in the above example that when we moved the generator coroutine to the first yield statement (using next(coro)), that the value "beep" was returned for us to print.

Asyncio: generator based coroutines

When the asyncio module was first released it didn’t support the async/await syntax, so when it was introduced, to ensure any legacy code that had a function that needed to be run concurrently (i.e. awaited) would have to use an asyncio.coroutine decorator function to allow it to be compatible with the new async/await syntax.

ℹ️ NOTE

refer to the documentation for information on this deprecated (as of Python 3.10) feature, as well as some other functions like asyncio.iscoroutine that are specific to generator based coroutines.

The original generator based coroutines meant any asyncio based code would have used yield from to await on Futures and other coroutines.

The following example demonstrates how to use both the new async coroutines with legacy generator based coroutines:

@asyncio.coroutine
def old_style_coroutine():
    yield from asyncio.sleep(1)

async def main():
    await old_style_coroutine()

Asyncio: new async coroutines

Coroutines created with async def are implemented using the more recent __await__ dunder method (see documentation here), while generator based coroutines are using a legacy ‘generator’ based implementation.

Types of Coroutines

This has led to the term ‘coroutine’ meaning multiple things in different contexts. We now have:

simple coroutines: traditional generator coroutine (no async io).
generator coroutines: async io using legacy asyncio implementation.
native coroutines: async io using latest async/await implementation.

Miscellaneous

There are a couple of interesting decorator functions provided by Python that can be a bit confusing, due to these functions appearing to have overlapping functionality.

They don’t overlap, but do appear to be used together:

types.coroutine: converts generator function into a coroutine.
asyncio.coroutine: abstraction ensuring asyncio compatibility.

ℹ️ NOTE

as we’ll see in a moment, asyncio.coroutine actually calls types.coroutine. You should ideally use the former when dealing with asyncio code.

More specifically, if we look at the implementation of the asyncio.coroutine code we can see:

If decorated function is already a coroutine, then just return it.
If decorated function is a generator, then convert it to a coroutine (using types.coroutine).
Otherwise wrap the decorated function such that when it’s converted to a coroutine it’ll await any resulting awaitable value.

What’s interesting about types.coroutine is that if your decorated function were to remove any reference to a yield, then the function will be executed immediately rather than returning a generator. See this Stack Overflow answer for more information as to where that behaviour was noticed.

Understanding the purpose of tox.ini

Wed, 18 Dec 2019 00:00:00 +0000

Introduction

I was confused by the Python tool tox for a long time, and was unsure what it was really used for considering lots of different Python packages appeared to rely on tox’s configuration file (e.g. tox.ini).

In this post I’m going to briefly explain what the tox tool is, and what a tox.ini configuration file looks like and why that configuration file can be used for more than just serving the purposes of the tox tool.

What is tox?

Tox is a tool that creates virtual environments, and installs the configured dependencies for those environments, for the purpose of testing a Python package (i.e. something that will be shared via PyPi, and so it only works with code that defines a setup.py).

PyPi == Python Package Index (i.e. where you typically install all your Python packages from, when executing pip install).

The file that you use to configure tox can be one of the following…

tox.ini
pyproject.toml (see PEP 518)
setup.cfg (see official guide to distributing packages)

ℹ️ NOTE

these ^^ are all ini file formats – which is information that will become more relevant/important later on.

Example tox.ini

All configuration options for tox can be found here: tox.readthedocs.io/en/latest/config.html but the below example is a simple demonstration of how you might configure tox for a real package (e.g. this is an actual tox.ini file I’ve used).

[tox]
envlist = 
    py37, lint
toxworkdir = 
    {env:TOX_WORK_DIR}

[testenv]
setenv =
    PYTHONDONTWRITEBYTECODE = 1
whitelist_externals =
    /bin/bash
deps = 
    -rrequirements-dev.txt
commands =
    py.test --cov={envsitepackagesdir}/bf_tornado -m "not integration"

[testenv:dev]
usedevelop=True
recreate = False
commands =
    # to run arbitrary commands: tox -e dev -- bash
    {posargs:py.test --cov=bf_tornado}

[testenv:lint]
deps =
    flake8==3.8.3
    mypy==0.782
commands =
    flake8 bf_tornado
    mypy --verbose --ignore-missing-imports --package bf_tornado

ℹ️ NOTE

the name after testenv: is the name of the virtual environment that will be created (e.g. testenv:foo will create a “foo” virtual environment).

Configuring other packages

A tox.ini file can be used to configure different types of packages, which is confusing at first because the tox home page suggests that tox is used to test your own packages you plan on distributing to PyPi.

What the tox authors mean by that description, is that the tox command itself is used to handle testing your packages, while the tox.ini configuration file is just one such file that can be used to contain configuration information.

This is why other packages, such as Flake8 allow you to configure Flake8 using the tox.ini file, as well as alternatives such as: setup.cfg or .flake8.

The key to understanding why this works is as follows: each of these files conform to the ini file format. So you’re free to use whatever file name you feel best suits your project, while the format of the file will stay consistent to what is expected of an .ini file.

Why do multiple tools support `tox.ini`?

The benefit of these various tools supporting the tox.ini filename specifically is to help reduce the number of configuration files a project requires.

Imagine you needed a unique configuration file for every Python command line tool you use in your project. How many would that result in? How hard would it be to remember all the various file names?

With that in mind, below is an example that shows various Python packages being configured within a tox.ini file.

And in case it’s unclear, the configuration inside of the tox.ini file is used instead of having to pass those configuration values via the command line.

So in the case of a tool such as flake8, instead of using flake8 --max-line-length=120 you could just call flake8 and the flag value is extracted from the configuration file.

[flake8]
max_line_length = 120
ignore = E261,E265,E402  # http://pep8.readthedocs.org/en/latest/intro.html#error-codes

[coverage:run]
branch = True

[coverage:report]
show_missing = True
exclude_lines =
    raise NotImplementedError
    return NotImplemented
    def __repr__
omit = bf_tornado/testing.py

[pytest]
addopts = 
    --strict -p no:cacheprovider --showlocals
markers =
    integration: mark a test as an integration test that makes http calls.

Bonus Section

Although not directly related to the conversation about tox.ini I thought it worth mentioning that I discovered recently the Python logging library allows you to be able to configure logging via an ini configuration file!

Reference: docs.python.org/3/howto/logging.html

The ini file might look something like the following:

[loggers]
keys=root,simpleExample

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_simpleExample]
level=DEBUG
handlers=consoleHandler
qualname=simpleExample
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

The Python program that uses this file might look like the following:

import logging
import logging.config

logging.config.fileConfig('logging.conf')

# create logger
logger = logging.getLogger('simpleExample')

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

Notice though, the name of the configuration file we imported was logging.conf (no .ini extension). It doesn’t matter that the file doesn’t use the .ini extension as long as the format of the file itself conforms to the ini format.

Python Management and Project Dependencies

Sun, 01 Dec 2019 00:00:00 +0000

Introduction

This blog post aims to demonstrate the most practical way to install multiple versions of Python, and of setting up ‘virtual environments’ for macOS userso

We’ll also dig into how to manage our project dependencies (e.g. we’ll be discussing the classic Pip and requirements.txt format) + our approach for avoiding newer tools such as Poetry and Pipenv, which I feel are just too complicated/overkill for the majority of use cases.

Why is this not a simple thing to do? I hear you ask! Well, because we don’t live in a perfect world and sometimes the overhead of ‘convenience’ can be worth the cost otherwise incurred, and sometimes it can’t.

I’ll start by describing briefly what a virtual environment is, and then I’ll move onto the point of this article and the options we have available to us.

Virtual Environments

A cooperatively isolated runtime environment that allows Python users and applications to install and upgrade Python distribution packages without interfering with the behaviour of other Python applications running on the same system. – Python Glossary

To simplify: if you work on multiple Python projects and each project uses the same external dependency (e.g. the Requests HTTP library), then it’s possible your application (for each project) will be written to support a specific version of that dependency.

Having a virtual environment setup for each project means you can have project specific Python package installs. For example, project ‘foo’ can use the Request library version 1.0 while project ‘bar’ can use version 2.0 (which might introduce a different API).

ℹ️ NOTE

the official documentation on Virtual Environments can be found here.

Creating Virtual Environments

If you installed Python using Homebrew (e.g. brew install python3) and you don’t care what version of Python you have, you only care about having virtual environments, then you can utilize the built-in venv module like so:

python3 -m venv /foo/bar
source /foo/bar/bin/activate
python3 -m pip install <dependencies>

This will ensure you only install third-party packages/modules into the specific virtual environment you’ve just activated.

The only downside of this very simple approach is that installing Python via Homebrew means you’ll have only a single version of Python installed and you have no control over what that version is, let alone have multiple versions installed (as the python3 command will be overwritten each time).

This consideration is actually critical to understand when you consider the Linux OS internals use the same Python version as exposed to the end user. Meaning if you mess up your Python install on Linux, then there is a high chance you’ll break the entire operating system!

ℹ️ NOTE

if you want to prevent accidentally executing pip install outside of a virtual environment then use export PIP_REQUIRE_VIRTUALENV=true (it can also be set in a ~/.pip/pip.conf)

If you require virtual environments across multiple Python versions, then read the following couple of sections…

Installing Python Versions

So there are in fact two ways of installing Python versions, but it’s much less practical in reality and so i’m only going to demonstrate one of them.

The two ways are:

Manual compilation
External build tool

As you can imagine, manually compiling Python would be the more flexible solution, but the harsh reality is that compiling Python requires very specific dependencies that can be hard to get right (and most often it goes wrong).

So instead we’ll focus on the latter: an external tool called pyenv which internally uses another external tool called python-build.

pyenv: let’s us switch Python versions easily
python-build let’s us install Python versions easily.

You don’t need to install python-build directly as it’ll be installed when you install pyenv.

To install pyenv execute:

brew install pyenv

ℹ️ NOTE

yes, this means you need to be using Homebrew, but let’s face it, it’s the defacto standard for macOS package management.

Once installed you’ll be able to use the following commands:

python-build --definitions: list all versions of Python available to be installed
pyenv install <version>: install the version of Python you need

Once you have installed the version of Python you need, now you just need to remember to ‘activate’ it whenever you’re working on your project that requires that specific version of Python. To do that, you’ll need to do two things (and you only need to do them once):

add eval "$(pyenv init -)" to your .bashrc (or shell of choice)
execute pyenv local <version> in your project directory

What the first point does is it’ll allow your shell to respond to any .python-version file found within a directory on your computer. This file will contain a Python version.

What generates the .python-version file is the latter point.

Virtual Environments for multiple Pythons

To setup virtual environments with Python is actually very simple (as we saw in the earlier part of this post), but not compatible when using an external build tool such as pyenv because of where pyenv installs Python binaries and how it switches between versions.

But luckily there is an extension to pyenv called pyenv-virtualenv which can be installed like so:

brew install pyenv-virtualenv

Once installed, setting up a new virtual environment is as simple as:

pyenv virtualenv foobar
pyenv activate foobar
pyenv deactivate foobar

ℹ️ NOTE

if you want you can specify the Python version to create the virtual environment for: pyenv virtualenv <version> <name>.

Shell Configuration

This is just a quick summary of the configuration lines added to my .bashrc:

eval "$(python3 -m pip completion --bash)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Managing Dependencies

When it comes to dealing with specific dependency versions, I like to use the method Kenneth Reitz published back in 2016.

ℹ️ NOTE

this method keeps with the traditional requirements.txt file as utilized by Pip. I mention this as you’ll notice with other tools (such as Pipenv or Poetry), that they move away from this established format and that can be a bit disruptive in terms of how Python teams have traditionally worked. I’m not saying it’s a bad thing, but change isn’t always good.

Problem Summary

Here is a summary of the problem we’re trying to solve:

The requirements.txt file typically doesn’t include the sub-dependencies required by your top-level dependencies (because that would require you to manually identify them and to type them all out, something that should be an automated process and so in practice never happens manually).

e.g. you specify a top-level dependency of foo (which might install version 1.0), but that internally requires the use of other third-party packages such as bar and baz (and specific versions for each of them).

But a pip install from a file that only includes the top-level dependencies could (over time) result in different sub-dependency versions being installed by either different members of your team or via your deployment platform.

e.g. if you don’t specify a version for foo, then in a months time when someone else (or your deployment platform) runs pip install it will attempt to install the latest version of foo which might be version 2.0 (and subsequently the third-party packages it uses might also change).

To avoid that people have come to use pip freeze after doing a pip install to overwrite their requirements.txt with a list of all dependencies and their explicit versions.

This solves the issue of installing from requirements.txt in a months time when lots of your top-level dependencies release new breaking versions.

The problem now is that you have to manually search for the top-level dependencies (in this new larger/more-indepth requirements.txt file) and update them manually. Doing this might break things as you now don’t know what the sub-dependency versions should be set to.

Solution

So the approach we take with any project is to define a requirements-to-freeze.txt file. This file will contain all your project’s top-level dependencies (inc. any explicit versions required), for example:

requests[security]
flask
gunicorn==19.4.5

Next we can generate our actual requirements.txt file based upon the contents of requirements-to-freeze.txt using the pip freeze command, like so:

python -m pip install -r requirements-to-freeze.txt
python -m pip freeze > requirements.txt

Which will result in a requirements.txt file that looks something like:

cffi==1.5.2
cryptography==1.2.2
enum34==1.1.2
Flask==0.10.1
gunicorn==19.4.5
idna==2.0
ipaddress==1.0.16
itsdangerous==0.24
Jinja2==2.8
MarkupSafe==0.23
ndg-httpsclient==0.4.0
pyasn1==0.1.9
pycparser==2.14
pyOpenSSL==0.15.1
requests==2.9.1
six==1.10.0
Werkzeug==0.11.4

This means you’ll never manually update requirements.txt again. Any time you need to update a dependency you’ll do it in requirements-to-freeze.txt, then re-run:

python -m pip install -r requirements-to-freeze.txt
python -m pip freeze > requirements.txt

Or instead of manually updating the dependencies in requirements-to-freeze.txt you could use the --upgrade flag:

python -m pip install -r requirements-to-freeze.txt --upgrade
python -m pip freeze > requirements.txt

To make the process easier (see notes below for details) we can utilize a bash shell script (and docker) to help us:

#!/bin/bash

# A quick and easy way to exhaustively freeze a set of "top level" Python
# dependencies for services following the "Better Pip Workflow" approach:
#
#     https://www.kennethreitz.org/essays/a-better-pip-workflow
#
# This script makes it easy to generate the frozen requirements without needing
# to juggle fresh virtualenvs or compile-time dependencies by instead
# installing the requirements in an ephemeral docker container.
#
# Example usage:
#
#     .freeze-requirements path/to/requirements-to-freeze.txt > path/to/requirements.txt
#
# To install extra base OS packages (e.g. mysql-dev), specify them in PACKAGES on the command line:
#
#     PACKAGES="mysql-dev" ./freeze-requirements ...

set -e
set -u
set -o pipefail

DEFAULT_PYTHON_VERSION=3.8
PYTHON_VERSION="${PYTHON_VERSION:-$DEFAULT_PYTHON_VERSION}"

DEFAULT_PACKAGES="gcc python-dev libssl-dev"
PACKAGES="${DEFAULT_PACKAGES} ${PACKAGES:-}"

TAG="freeze-requirements-${PYTHON_VERSION}"

requirements_file="${1:-}"

if [ "$requirements_file" == "" ]; then
    echo "Usage: $(basename $0) REQUIREMENTS_FILE" >&2
    exit 1
fi

if [ ! -f "$requirements_file" ] && [ "$requirements_file" != "-" ]; then
    echo "File not found: $requirements_file" >&2
    exit 1
fi

docker build -t $TAG - >&2 <<EOF
FROM python:${PYTHON_VERSION}-slim

RUN apt-get update && apt-get install -y ${PACKAGES}
RUN pip install virtualenv && virtualenv /venv
EOF

cat $requirements_file | exec docker run --rm -i -a stdin -a stdout -a stderr $TAG sh -c '
cat >/tmp/requirements-to-freeze.txt
/venv/bin/pip install -r /tmp/requirements-to-freeze.txt >&2
/venv/bin/pip freeze -r /tmp/requirements-to-freeze.txt'

Caching Dependencies

Starting with pip version 6.0 you can prevent having to reinstall dependencies that are used across multiple virtual environments by caching them (this is especially useful with Continuous Integration builds).

To do so, add the following to your ~/.bashrc file:

export PIP_DOWNLOAD_CACHE=$HOME/.pip/cache

Alternatively add it to your ~/.pip/pip.conf file:

[global]
require-virtualenv = true
download-cache = $HOME/.pip/cache

Command Line Packages

As a bonus section I’m going to quickly mention the tool pipx which allows us to install Python command line tools such that they are isolated binaries and so they don’t clutter up our top-level Python runtime space.

To install pipx, the official instructions state you can install it via either Homebrew or via an existing Python interpreter.

Homebrew install is as follows:

brew install pipx
pipx ensurepath

Using an existing Python interpreter:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

But I found it didn’t work unless I omitted the --user flag:

python3 -m pip install pipx
python3 -m pipx ensurepath

After that you can add the following to your ~/.bashrc (or similar for whatever shell you use):

pipx completions
# eval "$(register-python-argcomplete pipx)"

Now you’re able to safely install command line Python tools, like so:

pipx install pycowsay

ℹ️ NOTE

if you use the PIP_REQUIRE_VIRTUALENV setting (mentioned earlier in this post) but you also installed pipx via Homebrew, then you’ll find that doing so can cause problems because pipx can’t use its internal list function (as no virtual environment is currently activated). So to fix the issue always set the environment variable to false: PIP_REQUIRE_VIRTUALENV=false pipx list.

Guide to Concurrency in Python with Asyncio

Sat, 30 Nov 2019 00:00:00 +0000

This is a quick guide to Python’s asyncio module and is based on Python version 3.8.

Introduction

So let’s start by addressing the elephant in the room: there are many modules provided by the Python standard library for handling asynchronous/concurrent/multiprocess code…

In this post we’re going to focus on the last two. Primarily we will be focusing on asyncio, before wrapping up with a look at some useful features of concurrent.futures.

The motivation for this post is to understand why you will most likely want to use asyncio over the other available modules (e.g. _thread and threading) and when it’s actually more appropriate to use either multiprocessing or concurrent.futures.

Why focus on asyncio?

One of the issues with writing concurrent code (using either the _thread or threading modules) is that you suffer the cost of ‘CPU context switching’ (as a CPU core can only run one thread at a time) which although quick, isn’t free.

Multi-threaded code also has to deal with issues such as ‘race conditions’, ‘dead/live locks’ and ‘resource starvation’ (where some threads are over utilized and others are under utilized).

Asyncio avoids these issues, so let’s see how…

A quick `asyncio` summary

asyncio is a library to write concurrent code using the async/await syntax. – docs.python.org/3.8/library/asyncio.html

The asyncio module provides both high-level and low-level APIs. Library and Framework developers will be expected to use the low-level APIs, while all other users are encouraged to use the high-level APIs.

It differs conceptually from the more traditional threading or multiprocess approach to asynchronous code execution in that it utilizes something called an event loop to handle the scheduling of asynchronous ‘tasks’ instead of using more traditional threads or subprocesses.

Importantly, asyncio is designed to solve I/O network performance, not CPU bound operations (which is where multiprocessing should be used). So asyncio is not a replacement for all types of asynchronous execution.

Asyncio is designed around the concept of ‘cooperative multitasking’, so you have complete control over when a CPU ‘context switch’ occurs (i.e. context switching happens at the application level and not the hardware level).

When using threads the Python scheduler is responsible for this, and so your application may context switch at any moment (i.e. it becomes non-deterministic).

This means when using threads you’ll need to also use some form of ‘lock’ mechanism to prevent multiple threads from accessing/mutating shared memory (which would otherwise subsequently cause your program to become non-thread safe).

A quick `concurrent.futures` summary

The concurrent.futures module provides a high-level interface for asynchronously executing callables. – docs.python.org/3.8/library/concurrent.futures.html

The concurrent.futures provides a high-level abstraction for the threading and multiprocessing modules, which is why we won’t discuss those modules in detail within this post. In fact the _thread module is a very low-level API that the threading module is itself built on top of (again, this is why we won’t be covering that either).

Now we’ve already mentioned that asyncio helps us avoid using threads so why would we want to use concurrent.futures if it’s just an abstraction on top of threads (and multiprocessing)? Well, because not all libraries/modules/APIs support the asyncio model.

For example, if you use boto3 and interact with AWS S3, then you’ll find those are synchronous operations. You can wrap those calls in multi-threaded code, but it would be better to use concurrent.futures as it means you not only benefit from traditional threads but an asyncio friendly package.

The concurrent.futures module is also designed to interop with the asyncio event loop, making it easier to work with a pool of threads/subprocesses within an otherwise asyncio driven application.

Additionally you’ll also want to utilize concurrent.futures when you require a pool of threads or a pool of subprocesses, while also using a clean and modern Python API (as apposed to the more flexible but low-level threading or multiprocessing modules).

Green threads?

There are many ways to achieve asynchronous programming. There’s the event loop approach (which asyncio implements), a ‘callback’ style historically favoured by single-threaded languages such as JavaScript, and more traditionally there has been a concept known as ‘green threads’.

In essence a green thread looks and feels exactly like a normal thread, except that the threads are scheduled by application code rather than by hardware (so effectively working around the same issue of deterministic context switching as an event loop does). But the problem of handling shared memory still exists.

So let’s take a quick look now at what the ‘event loop’ is, as it’s the foundation of what makes asyncio work and why we can avoid ‘callback hell’ and the problems inherent with ‘green threads’…

Event Loop

The core element of all asyncio applications is the ‘event loop’. The event loop is what schedules and runs asynchronous tasks.

Image Credit

What makes the asyncio event loop so effective is the fact that Python implements it around generators. A generator enables a function to be partially executed, then halt its execution at a specific point, maintaining a stack of objects and exceptions, before resuming again.

I’ve written about iterators, generators and coroutines recently, so if you’re interested in those concepts, then I’ll refer you to that post.

ℹ️ NOTE

for more API information on the event loop, please refer to the official Python documentation.

Awaitables

The driving force behind asyncio is the ability to schedule asynchronous ‘tasks’. There are a few different types of objects in Python that help support this, and they are generally grouped by the term ‘awaitable’.

Ultimately, something is awaitable if it can be used in an await expression.

There are three main types of awaitables:

Coroutines
Tasks
Futures

ℹ️ NOTE

Futures is a low-level type and so you shouldn’t need to worry about it too much if you’re not a library/framework developer (as you should be using the higher-level abstraction APIs instead).

Coroutines

There are two closely related terms used here:

a coroutine function: an async def function.
a coroutine object: an object returned by calling a coroutine function.

Generator based coroutine functions (e.g. those defined by decorating a function with @asyncio.coroutine) are superseded by the async/await syntax, but will continue to be supported until Python 3.10 – docs.python.org/3.8/library/asyncio-task.html.

Refer to my post “iterators, generators, coroutines” for more details about generator based coroutines and their asyncio history.

Tasks

Tasks are used to schedule coroutines concurrently.

All asyncio applications will typically have (at least) a single ‘main’ entrypoint task that will be scheduled to run immediately on the event loop. This is done using the asyncio.run function (see ‘Running an asyncio program’).

A coroutine function is expected to be passed to asyncio.run, while internally asyncio will check this using the helper function coroutines.iscoroutine (see: source code). If not a coroutine, then an error is raised, otherwise the coroutine will be passed to loop.run_until_complete (see: source code).

The run_until_complete function expects a Future (see below section for what a Future is) and uses another helper function futures.isfuture to check the type provided. If not a Future, then the low-level API ensure_future is used to convert the coroutine into a Future (see source code).

ℹ️ NOTE

here is a comparison of the various methods for validating if a function is a coroutine. The results aren’t necessarily what you might expect.

In older versions of Python, if you were going to manually create your own Future and schedule it onto the event loop, then you would have used asyncio.ensure_future (now considered to be a low-level API), but with Python 3.7+ this has been superseded by asyncio.create_task.

Additionally with Python 3.7, the idea of interacting with the event loop directly (e.g. getting the event loop, creating a task with create_task and then passing it to the event loop) has been replaced with asyncio.run, which abstracts it all away for you (see ‘Running an asyncio program’ to understand what that means).

The following APIs let you see the state of the tasks running on the event loop:

asyncio.current_task
asyncio.all_tasks

ℹ️ NOTE

for other available methods on a Task object please refer to the documentation.

Futures

A Future is a low-level awaitable object that represents an eventual result of an asynchronous operation.

To use an analogy: it’s like an empty postbox. At some point in the future the postman will arrive and stick a letter into the postbox.

This API exists to enable callback-based code to be used with async/await, while loop.run_in_executor is an example of an asyncio low-level API function that returns a Future (see also some of the APIs listed in Concurrent Functions).

ℹ️ NOTE

for other available methods on a Future please refer to the documentation.

Running an asyncio program

The high-level API (as per Python 3.7+) is:

import asyncio

async def foo():
    print("Foo!")

async def hello_world():
    await foo()  # waits for `foo()` to complete
    print("Hello World!")

asyncio.run(hello_world())

The .run function always creates a new event loop and closes it at the end. If you were using the lower-level APIs, then this would be something you’d have to handle manually (as demonstrated below).

loop = asyncio.get_event_loop()
loop.run_until_complete(hello_world())
loop.close()

Running Async Code in the REPL

Prior to Python 3.8 you couldn’t execute async code within the standard Python REPL (it would have required you to use the IPython REPL instead).

To do this with the latest version of Python you would run python -m asyncio. Once the REPL has started you don’t need to use asyncio.run(), but just use the await statement directly.

asyncio REPL 3.8.0+ (heads/3.8:5f234538ab, Dec  1 2019, 11:05:25)

[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.

>>> import asyncio
>>> async def foo():
...   await asyncio.sleep(5)
...   print("done")
...
>>> await foo()
done

Notice the REPL automatically executes import asyncio when starting up so we’re able to use any asyncio functions (such as the .sleep function) without having to manually type that import statement ourselves.

Use another Event Loop

If for some reason you didn’t want to use the event loop provided by asyncio (which is a pure Python implementation), you can swap it out for another event loop such as uvloop.

uvloop is a fast, drop-in replacement of the built-in asyncio event loop. uvloop is implemented in Cython and uses libuv under the hood.

According to the authors of uvloop, it is comparible in speed to that of Go programs! I recommend reading their blog post about its initial release.

If you want to utilize uvloop then first install it with pip install uvloop, then add a call to uvloop.install() like so:

import asyncio
import uvloop

async def foo():
    print("Foo!")

async def hello_world():
    await foo()
    print("Hello World!")

uvloop.install()
asyncio.run(hello_world())

Concurrent Functions

The following functions help to co-ordinate the running of functions concurrently, and offer varying degrees of control dependant on the needs of your application.

asyncio.gather: takes a sequence of awaitables, returns an aggregate list of successfully awaited values.
asyncio.shield: prevent an awaitable object from being cancelled.
asyncio.wait: wait for a sequence of awaitables, until the given ‘condition’ is met.
asyncio.wait_for: wait for a single awaitable, until the given ‘timeout’ is reached.
asyncio.as_completed: similar to gather but returns Futures that are populated when results are ready.

ℹ️ NOTE

gather has specific options for handling errors and cancellations. For example, if return_exceptions: False then the first exception raised by one of the awaitables is returned to the caller of gather, where as if set to True then the exceptions are aggregated in the list alongside successful results. If gather() is cancelled, all submitted awaitables (that have not completed yet) are also cancelled.

Deprecated functions

@asyncio.coroutine: removed in favour of async def in Python 3.10
asyncio.sleep: the loop parameter will be removed in Python 3.10

ℹ️ NOTE

you’ll find in most of these APIs a loop argument can be provided to enable you to indicate the specific event loop you want to utilize). It seems Python has deprecated this argument in 3.8, and will remove it completely in 3.10.

Examples

`gather`

The following example demonstrates how to wait for multiple asynchronous tasks to complete.

import asyncio


async def foo(n):
    await asyncio.sleep(5)  # wait 5s before continuing
    print(f"n: {n}!")


async def main():
    tasks = [foo(1), foo(2), foo(3)]
    await asyncio.gather(*tasks)


asyncio.run(main())

`wait`

The following example uses the FIRST_COMPLETED option, meaning whichever task finishes first is what will be returned.

import asyncio
from random import randrange


async def foo(n):
    s = randrange(5)
    print(f"{n} will sleep for: {s} seconds")
    await asyncio.sleep(s)
    print(f"n: {n}!")


async def main():
    tasks = [foo(1), foo(2), foo(3)]
    result = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    print(result)


asyncio.run(main())

An example output of this program would be:

1 will sleep for: 4 seconds
2 will sleep for: 2 seconds
3 will sleep for: 1 seconds

n: 3!

(
{<Task finished coro=<foo() done, defined at await.py:5> result=None>}, 
{
<Task pending coro=<foo() running at await.py:8> 
  wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10322b468>()]>>, 
<Task pending coro=<foo() running at await.py:8> 
  wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x10322b4c8>()]>>
}
)

`wait_for`

The following example demonstrates how we can utilize a timeout to prevent waiting endlessly for an asynchronous task to finish.

import asyncio


async def foo(n):
    await asyncio.sleep(10)
    print(f"n: {n}!")


async def main():
    try:
        await asyncio.wait_for(foo(1), timeout=5)
    except asyncio.TimeoutError:
        print("timeout!")


asyncio.run(main())

ℹ️ NOTE

the asyncio.TimeoutError doesn’t provide any extra information so there’s no point in trying to use it in your output (e.g. except asyncio.TimeoutError as err: print(err)).

`as_completed`

The following example demonstrates how as_complete will yield the first task to complete, followed by the next quickest, and the next until all tasks are completed.

import asyncio
from random import randrange


async def foo(n):
    s = randrange(10)
    print(f"{n} will sleep for: {s} seconds")
    await asyncio.sleep(s)
    return f"{n}!"


async def main():
    counter = 0
    tasks = [foo("a"), foo("b"), foo("c")]

    for future in asyncio.as_completed(tasks):
        n = "quickest" if counter == 0 else "next quickest"
        counter += 1
        result = await future
        print(f"the {n} result was: {result}")


asyncio.run(main())

An example output of this program would be:

c will sleep for: 9 seconds
a will sleep for: 1 seconds
b will sleep for: 0 seconds

the quickest result was: b!
the next quickest result was: a!
the next quickest result was: c!

`create_task`

The following example demonstrates how to convert a coroutine into a Task and schedule it onto the event loop.

import asyncio


async def foo():
    await asyncio.sleep(10)
    print("Foo!")


async def hello_world():
    task = asyncio.create_task(foo())
    print(task)
    await asyncio.sleep(5)
    print("Hello World!")
    await asyncio.sleep(10)
    print(task)


asyncio.run(hello_world())

We can see from the above program that we use create_task to convert our coroutine function into a Task. This automatically schedules the Task to be run on the event loop at the next available tick.

This is in contrast to the lower-level API ensure_future (which is the preferred way of creating new Tasks). The ensure_future function has specific logic branches that make it useful for more input types than create_task which only supports scheduling a coroutine onto the event loop and wrapping it inside a Task (see: ensure_future source code).

The output of this program would be:

<Task pending coro=<foo() running at create_task.py:4>>
Hello World!
Foo!
<Task finished coro=<foo() done, defined at create_task.py:4> result=None>

Let’s review the code and compare to the above output we can see…

We convert foo() into a Task and then print the returned Task immediately after it is created. So when we print the Task we can see that its status is shown as ‘pending’ (as it hasn’t been executed yet).

Next we’ll sleep for five seconds, as this will cause the foo Task to now be run (as the current Task hello_world will be considered busy).

Within the foo Task we also sleep, but for a longer period of time than hello_world, and so the event loop will now context switch back to the hello_world Task, where upon the sleep will pass and we’ll print the output string Hello World.

Finally, we sleep again for ten seconds. This is just so we can give the foo Task enough time to complete and print its own output. If we didn’t do that then the hello_world task would finish and close down the event loop. The last line of hello_world is printing the foo Task, where we’ll see the status of the foo Task will now show as ‘finished’.

Callbacks

When dealing with a Task, which really is a Future, then you have the ability to execute a ‘callback’ function once the Future has a value set on it.

The following example demonstrates this by modifying the previous create_task example code:

import asyncio


async def foo():
    await asyncio.sleep(10)
    return "Foo!"


def got_result(future):
    print(f"got the result! {future.result()}")


async def hello_world():
    task = asyncio.create_task(foo())
    task.add_done_callback(got_result)
    print(task)
    await asyncio.sleep(5)
    print("Hello World!")
    await asyncio.sleep(10)
    print(task)


asyncio.run(hello_world())

Notice in the above program we add a new got_result function that expects to receive a Future type, and thus calls .result() on the Future.

Also notice that to get this function to be called, we pass it to .add_done_callback() which is called on the Task returned by create_task.

The output of this program is:

<Task pending coro=<foo() running at gather.py:4> cb=[got_result() at gather.py:9]>
Hello World!
got the result! Foo!
<Task finished coro=<foo() done, defined at gather.py:4> result='Foo!'>

Pools

When dealing with lots of concurrent operations it might be wise to utilize a ‘pool’ of threads (and/or subprocesses) to prevent exhausting your application’s host resources.

This is where the concurrent.futures module comes in. It provides a concept referred to as an Executor to help with this and which can be run standalone or be integrated into an existing asyncio event loop (see: Executor documentation).

Executors

There are two types of ‘executors’:

Let’s look at the first way to execute code within one of these executors, by using an asyncio event loop to schedule the running of the executor.

To do this you need to call the event loop’s .run_in_executor() function and pass in the executor type as the first argument. If None is provided, then the default executor is used (which is the ThreadPoolExecutor).

The following example is copied verbatim from the Python documentation:

import asyncio
import concurrent.futures


def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open("/dev/urandom", "rb") as f:
        return f.read(100)


def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))


async def main():
    loop = asyncio.get_running_loop()

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(None, blocking_io)
    print("default thread pool", result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(pool, blocking_io)
        print("custom thread pool", result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(pool, cpu_bound)
        print("custom process pool", result)


asyncio.run(main())

The second way to execute code within one of these executors is to send the code to be executed directly to the pool. This means we don’t have to acquire the current event loop to pass the pool into it (as the earlier example demonstrated), but it comes with a caveat which is the parent program won’t wait for the task to be completed unless you explicitly tell it to (which I’ll demonstrate next).

With that in mind, let’s take a look at this alternative approach. It involves calling the executor’s submit() method:

import concurrent.futures
import time


def slow_op(*args):
    print(f"arguments: {args}")
    time.sleep(5)
    print("slow operation complete")
    return 123


def do_something():
    with concurrent.futures.ProcessPoolExecutor() as pool:
        future = pool.submit(slow_op, "a", "b", "c")

        for fut in concurrent.futures.as_completed([future]):
            assert future.done() and not future.cancelled()
            print(f"got the result from slow_op: {fut.result()}")


if __name__ == "__main__":
    print("program started")
    do_something()
    print("program complete")

ℹ️ NOTE

be careful with a global process executor (e.g. placing something like PROCESS_POOL = concurrent.futures.ProcessPoolExecutor() within the global scope and using that reference within our do_something() function) as this means when the program is copied into a new process you’ll get an error from the Python interpreter about a leaked semaphore. This is why I create the process pool executor within a function.

One thing worth noting here is that if we hadn’t used the with statement (like we do in the above example) it would mean we’d not be shutting down the pool once it has finished its work, and so (depending on if your program continues running) you may discover resources aren’t being cleaned up.

To solve that problem you can call the .shutdown() method which is exposed to both types of executors via its parent class concurrent.futures.Executor.

Below is an example that does that, but now using the threadpool executor:

import concurrent.futures

THREAD_POOL = concurrent.futures.ThreadPoolExecutor(max_workers=5)


def slow_op(*args):
    with open("/dev/urandom", "rb") as f:
        return f.read(100000)


def do_something():
    future = THREAD_POOL.submit(slow_op, "a", "b", "c")

    THREAD_POOL.shutdown()

    assert future.done() and not future.cancelled()

    print(f"got the result from slow_op: {len(future.result())}")


if __name__ == "__main__":
    print("program started")
    do_something()
    print("program complete")

Pay attention to the placement of the call to .shutdown(). We no longer have any code to handle waiting for the task to complete. You might have expected calling .shutdown() and then immediately checking if the task is complete (e.g. assert future.done()) to cause an error to be raised as the future is unlikely to be finished.

ℹ️ NOTE

remember also if you call .done() on a future when a value has not yet been set, then you’ll see an exception such as asyncio.InvalidStateError.

But no error is raised, and the future is indeed considered ‘done’ by the time we check it. This is because the shutdown method has a single argument defined called wait and its default value is set to True, which means it would wait for all scheduled tasks to complete before shutting down the executor pool.

Thus the .shutdown() method is a synchronization call (i.e. it ensures all tasks are complete before shutting down, and thus we can guarantee all results will be available).

Now if we had passed .shutdown(wait=False) instead, then the call to future.done() would have raised an exception (as the scheduled task would still be running as the threadpool was being closed), and so in that case we’d need to ensure that we use another mechanism for acquiring the results of the scheduled tasks (such as concurrent.futures.as_completed or concurrent.futures.wait).

`asyncio.Future` vs `concurrent.futures.Future`

One final thing to mention is that a concurrent.futures.Future object is different from an asyncio.Future.

An asyncio.Future is intended to be used with the asyncio’s event loop, and is awaitable. A concurrent.futures.Future is not awaitable.

Using the .run_in_executor() method of an event loop will provide the necessary interoperability between the two future types by wrapping the concurrent.futures.Future type in a call to asyncio.wrap_future (see next section for details).

`asyncio.wrap_future`

Since Python 3.5 we can use asyncio.wrap_future to convert a concurrent.futures.Future to an asyncio.Future. An example of this can be seen below…

import asyncio
import random
from concurrent.futures import ThreadPoolExecutor
from time import sleep


def return_after_5_secs(message):
    sleep(5)
    return message


pool = ThreadPoolExecutor(3)


async def doit():
    identify = random.randint(1, 100)
    future = pool.submit(return_after_5_secs, (f"result: {identify}"))
    awaitable = asyncio.wrap_future(future)
    print(f"waiting result: {identify}")
    return await awaitable


async def app():
    # run some stuff multiple times
    tasks = [doit(), doit()]

    result = await asyncio.gather(*tasks)
    print(result)

print("waiting app")
asyncio.run(app())

The output of this program would be:

waiting app
waiting result: 62
waiting result: 83

# ...five seconds pass by...

['result: 62', 'result: 83']

Go Arrays and Slices

Sun, 17 Nov 2019 00:00:00 +0000

I found myself recently trying to recall specific details of how slices work when needing to do something that meant I wanted to not mutate the underlying array data structure of the slice I was working with.

Now the reason for why I wanted to do that isn’t important. What’s motivating this write-up is my want for a good reference document (not saying the official go blog isn’t a good reference, but I have my own things I like to focus in on in these situations).

Composite Types

Let’s recap on what we’re talking about when we say “array” and “slice”:

Array: fixed-length data structure, zero-indexed, contains same type
Slice: variable-length sequence, contains same type

Arrays

An array uses subscript notation to access elements, and is zero-indexed. Here is an example of both those concepts:

a := [3]string{"a", "b", "c"}

fmt.Println(a[1]) // "b"

If you define an array without using the ‘literal’ syntax (as demonstrated in the above example program), then the values will be initialized with the zero value of the given type, for example:

var a [3]string

fmt.Printf("%#v", a) // [3]string{"", "", ""}

The size of the array is actually part of its ‘type’ definition, for example the following two arrays have unique types:

a := [3]string{"a", "b", "c"}
b := [2]string{"a", "b"}

fmt.Printf("a: %T, b: %T", a, b) // a: [3]string, b: [2]string

If you don’t want to have to count the number of elements you’re defining, then you can use an ellipsis ... instead:

a := [...]int{1, 2, 3}

fmt.Printf("%T %#v", a, a) // [3]int [3]int{1, 2, 3}

Notice how the output of the above program inserts the calculated length of the array.

As arrays are fixed-length, it means they cannot be resized once full.

Arrays are also ‘copied’ when being passed into a function. This means if a function modifies a given array, it’s actually only modifying a copy of the array and not the original (i.e. go doesnt use ‘pass-by-reference’ semantics like some other languages).

If dealing we an array data structure, remember you’ll need to pass a pointer to it if you require a function to be able to modify the original array in memory.

Slices

A slice is a lightweight data structure that provides access to a subsequence (or ‘window’ view) of an underlying array.

The slice data structure consists of the following fields:

ptr: pointer to array.
len: length of slice (number of elements it contains).
cap: capacity of slice (number of elements in array, starting from the first element in the slice).

ℹ️ NOTE

a slice cannot grow larger than its capacity, nor can you reslice a slice to attempt to access earlier elements in the array.

Here is an example program that creates an array and then makes a slice of a subsequence of the original array:

a := [3]int{1, 2, 3}
fmt.Printf("array:  %T\n\t%#v\n\n", a, a)

s := a[1:]
fmt.Printf("slice:  %T\n\t%#v\n", s, s)

The output from the above code would be as follows (notice how the slice provides a narrower ‘view’ of the original array):

array: [3]int
       [3]int{1, 2, 3}

slice: []int
       []int{2, 3}

Notice that a slice ‘type’ looks the same as an array’s, but just omits a length (e.g. slice: []int, array: [3]int).

Slices, much like arrays, cannot dynamically grow larger at runtime. When a slice is full we must create a new slice, which requires the use of go’s builtin functions.

When modifying a slice you are infact modifying the underlying array, as demonstrated below:

a := [3]int{1, 2, 3}
s := a[1:]

s[0] = 4

fmt.Printf("array:  %T\n\t%#v\n\n", a, a)
fmt.Printf("slice:  %T\n\t%#v\n", s, s)

The above code results in the following output:

array: [3]int
       [3]int{1, 4, 3}

slice: []int
       []int{4, 3}

Notice both the slice and the underlying array have been updated.

Although a lot of people refer to this as “updating a slice” when talking about their code, I personally think it’s best not to think of these as two distinct pieces of data. The array holds the data and the slice is just a language abstraction that enables us to control how we view that data.

Additionally, it’s important to realize that because a slice contains a pointer to an underlying array, it means multiple slices can point to the same array in memory (as demonstrated below).

a := [3]int{1, 2, 3}
s1 := a[:2]
s2 := a[1:]

fmt.Printf("array:  %T\n\t%#v\n\n", a, a)
fmt.Printf("slice1: %T\n\t%#v\n\n", s1, s1)
fmt.Printf("slice2: %T\n\t%#v\n\n", s2, s2)

// make modification via the first slice
s1[1] = 4

fmt.Print("---\n\n")
fmt.Printf("array:  %T\n\t%#v\n\n", a, a)
fmt.Printf("slice1: %T\n\t%#v\n\n", s1, s1)
fmt.Printf("slice2: %T\n\t%#v\n\n", s2, s2)

The output of the above program is shown below. Notice how the two slices, s1 and s2 both point at the same underlying array and so although we make a modification via the first slice we can see that both slices will highlight the changed value:

array:  [3]int
    [3]int{1, 2, 3}

slice1: []int
    []int{1, 2}

slice2: []int
    []int{2, 3}

---

array:  [3]int
    [3]int{1, 4, 3}

slice1: []int
    []int{1, 4}

slice2: []int
    []int{4, 3}

Append

You’ll find in a lot of situations code that needs to append data to a slice. This results in code that uses the builtin append function, but interestingly will nearly always reassign the returned value back to the slice variable itself:

a := [...]int{1, 2, 3, 4, 5}
s := a[1:]

fmt.Printf("array:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n\n", a, a, len(a), cap(a))
fmt.Printf("slice:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n\n", s, s, len(s), cap(s))

s = append(s, 6)

fmt.Printf("array:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n\n", a, a, len(a), cap(a))
fmt.Printf("slice:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n\n", s, s, len(s), cap(s))

In the above program we have an array that contains five elements. We take a slice of it which is the last four elements. We then attempt to append a new value (6) to the slice (which should mean appending it to the underlying array). The output of that program is as follows:

array:  [5]int
    [5]int{1, 2, 3, 4, 5}
    len: 5
    cap: 5

slice:  []int
    []int{2, 3, 4, 5}
    len: 4
    cap: 4

array:  [5]int
    [5]int{1, 2, 3, 4, 5}
    len: 5
    cap: 5

slice:  []int
    []int{2, 3, 4, 5, 6}
    len: 5
    cap: 8

ℹ️ NOTE

the append function always returns a new slice.

Notice how s is showing an updated ‘view’ (e.g. 2, 3, 4, 5, 6). What’s interesting here is the new slice must have also resulted in a new array being allocated (and subsequently the new slice is pointing to it) because the original array (a) isn’t showing the appended value (6).

Now we can check this with some overly complicated code that ‘reflects’ into the internal go code (this will enable us to locate the slice’s pointer and to dereference that pointer to access the underlying array):

s := []int{1, 2, 3, 4}

hdr := (*reflect.SliceHeader)(unsafe.Pointer(&s))
data := *(*[4]int)(unsafe.Pointer(hdr.Data))

fmt.Printf("slice:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n", s, s, len(s), cap(s))
fmt.Printf("hdr: %#v\n", hdr)   // &reflect.SliceHeader{Data:0x40e020, Len:4, Cap:4}
fmt.Printf("data: %#v\n", data) // [4]int{1, 2, 3, 4}

s = append(s, 5)

hdr2 := (*reflect.SliceHeader)(unsafe.Pointer(&s))
data2 := *(*[8]int)(unsafe.Pointer(hdr.Data))

fmt.Printf("slice:  %T\n\t%#v\n\tlen: %d\n\tcap: %d\n", s, s, len(s), cap(s))
fmt.Printf("hdr2: %#v\n", hdr2)   // &reflect.SliceHeader{Data:0x45e020, Len:5, Cap:8}
fmt.Printf("data2: %#v\n", data2) // [8]int{1, 2, 3, 4, 5, 0, 0, 0}

ℹ️ NOTE

s := []int{1, 2, 3, 4} causes an underlying array to be created and then referened by the slice we defined.

The output of the above program is as follows:

slice:  []int
    []int{1, 2, 3, 4}
    len: 4
    cap: 4

hdr: &reflect.SliceHeader{Data:0x40e020, Len:4, Cap:4}
data: [4]int{1, 2, 3, 4}

slice:  []int
    []int{1, 2, 3, 4, 5}
    len: 5
    cap: 8

hdr: &reflect.SliceHeader{Data:0x45e020, Len:5, Cap:8}
data: [8]int{1, 2, 3, 4, 5, 0, 0, 0}

We can see the Data field on the SliceHeader is pointing to a different memory address!

We can also see that the capacity of the slice (cap) has increased to double! This demonstrates what’s happening ‘behind-the-scenes’, and is similar to how resizing an array is done in other languages (i.e. you create a new array at double the size of the old array, then you append the new values until that new array is full and the process repeats).

So what has happened is that append has returned a new slice (which is expected), but that the slice is now pointing to a new underlying array. This is the primary reason why when appending a value to a slice you’ll see the slice variable is updated to the return value of append: because we don’t know (unless we do some inspection of the returned slice) whether the underlying array has been copied to a new array in memory.

Caveat of Appending

If the underlying array had enough capacity for the appended value(s), then append would have still returned a new slice (because remember a slice cannot grow beyond its defined capacity) but the underlying array would still be the same array in memory. Let’s see an example of that below:

a := [6]int{1, 2, 3, 4, 5, 6}
s := a[1:4]
x := append(s, 0)

fmt.Printf("array:    %T\n\t  %#v\n\t  len: %d\n\t  cap: %d\n\n", a, a, len(a), cap(a))
fmt.Printf("slice1 s: %T\n\t  %#v\n\t  len: %d\n\t  cap: %d\n\n", s, s, len(s), cap(s))
fmt.Printf("slice2 x: %T\n\t  %#v\n\t  len: %d\n\t  cap: %d\n\n", x, x, len(x), cap(x))

The output of that program is as follows:

array:    [6]int
      [6]int{1, 2, 3, 4, 0, 6}
      len: 6
      cap: 6

slice1 s: []int
      []int{2, 3, 4}
      len: 3
      cap: 5

slice2 x: []int
      []int{2, 3, 4, 0}
      len: 4
      cap: 5

Notice how the first slice s has a length of 3 and a capacity of 5. That is because when we created the slice we specified explicitly we wanted only three elements (1:4) and that the slice’s capacity is derived from the number of elements in the underlying array starting from the first element referenced within the slice).

Meaning the first element in the slice is 2 and counting from there within the underlying array would result in a capacity of 5 (e.g. 2, 3, 4, 5, 6 is five elements).

Now when we call append on s we know that append will always return a new slice and so we assign that new slice to a different variable x. We can see for x the length is one element longer because of the append (e.g. we appended the value 0) but the capacity is still 5 meaning there is more room in the underlying array and so we don’t create a new array nor update the slice pointer to that new array.

What’s really interesting here is that the appending of the value 0 to the first slice has caused the underlying array to be modified in a potentially unexpected way!

By that I mean, we can see the underlying array has now replaced the element value 5 with 0 instead of having 0 appended to the end of it. You might have expected a new array to be created (because really the underlying array has a length of 6 and so an array with elements 1, 2, 3, 4, 5, 6, 0 would be a length of seven, thus not enough capacity in the array).

But because of how slices work, and its capacity is determined by the number of elements in the underlying array starting from the first element referenced by the slice, it means when we append to the underlying array, we are doing so from the perspective of the underlying array ending at the element value 4.

Yes, this is confusing.

See the go tour for more examples.

ℹ️ NOTE

there’s also a gotcha which is worth being aware of, and is related to the fact that slices point to the same underlying array, which occurs when the slice modifications don’t change the array’s capacity. See https://yourbasic.org/golang/gotcha-append/ for details.

Staying Anonymous

Wed, 04 Sep 2019 00:00:00 +0000

Privacy is becoming more and more important in our modern world. We’re constantly tracked and having our movements sold to advertisers.

Even more so our privacy becomes a necessity if we find ourselves doing things that could get us arrested (or even killed), such as being a journalist or a whistleblower.

I am lucky enough to not suffer from such necessity, but wherever possible I do like to have control over my web browsing and other online habits.

What follows are some tools that I use to try and limit my exposure to the internet. This isn’t a ‘how to be 100% anonymous’ guide, but maybe some of these tools will be of interest to you.

DNS

When you make a request for an online resource (for example this web page), your computer had to first convert the domain integralist.co.uk into an IP so that it could lookup the resource.

I’ve discussed how DNS works previously so I wont dive into the details of it here, but suffice to say the whole process of resolving the domain name to an IP address is something that happens unencrypted.

That means every resource you try to access will be known by your ISP, and anyone else for that matter who are snooping on the packets crossing over the network.

Additionally, a ‘man-in-the-middle’ (MITM) could modify DNS responses in such a way as to direct you to the wrong location (e.g. phishing).

To avoid these concerns you should utilize an encrypted connection to a DNS resolver. There are organizations, such as Cloudflare and Google, who offer this for free (that in itself is usually something to pay attention to for those who are very security minded).

There are currently two forms that this type of encryption can take:

DNS over TLS (DoT)
DNS over HTTPS (DoH)

ℹ️ NOTE

there are arguments over which is better, see this article for a breakdown of what it all means.

Here are some useful links to Cloudflare and Google if you’re interested in their services:

Cloudflare:
- DNS over TLS
- DNS over HTTPS
Google:
- DNS over TLS
- DNS over HTTPS

VPN

VPN stands for “virtual private network”, and is used to encrypt your internet traffic. This means no one, not even your ISP can see what resources you are requesting. All your ISP will see is you make a request to a VPN and from there you have established a secure ‘tunnel’ to pass data back-and-forth.

It’s important to ensure your chosen VPN doesn’t leak DNS requests, which can occur if your VPN doesn’t offer a feature such as a ‘network kill switch’ (or ‘network lock’). Typically this is handled by the VPN software you download and run on your computer.

ℹ️ NOTE

there are online tools available that help you verify your VPN isn’t leaking data.

Using a VPN also means your IP is protected and can’t be identified because resources that are requested are actually requested from the VPN server(s). IP protection can also be increased if you utilize a request pattern such as Double Hop or an extra service such as Tor.

Typically you pay for a VPN, although in some cases you might find one that offers a free tier (e.g. Windscribe). Although in the case of Windscribe that requires them to monitor your logs in order to distinguish between free tier users who are abusing their service.

ℹ️ NOTE

there are online resources for comparing different VPNs, which can be useful for helping you choose who to use.

Tracking bandwidth might be a concern to you if you’re overly security-minded, or it might be an acceptable trade-off for utilizing a free service. I personally believe that you get what you pay for, and a VPN is usually money well spent.

Paying for a VPN doesn’t have to be expensive though. I pay £6 a month which is acceptable considering I encrypt all my traffic, but I can appreciate that this might still be too much money for some people depending on circumstance.

You can find VPNs for as cheap as £1 a month (e.g. Windscribe’s custom built plans) or from as low as £2.99 with NordVPN depending on the length of the plan you select.

Tor

Tor (aka. The Onion Router) is a free service that directs your requested resource through multiple servers (much like the layers of an onion). The servers are chosen at random and increase your privacy by obfuscating the origin of the request through the use of encryption.

Tor does not replace a VPN. Tor only anonymizes your web browsing, whereas a VPN encrypts all internet traffic (e.g. locally running desktop applications, to terminal command line applications etc).

Using Tor with a VPN is possible but only really for those who require complete privacy (arguably not possible without a lot of effort).

For most typical users who want to protect their privacy and online behaviours, the additional overhead of using Tor is unnecessary when they already use a VPN.

If you feel using Tor would be a good thing, then it’s best to first connect to your VPN before opening up Tor because it means your VPN won’t be able to see what you’re doing inside the Tor network (which might be useful in the unlikely scenario where your VPN is compromised).

Equally, if there is a bug in Tor (this has happened), then the VPN will offer that extra layer of protection.

ℹ️ NOTE

there is even an entire OS that is bootable from a USB stick that routes all traffic through Tor, called Tails (see this simple rundown for a recent update, and this older review for a more general feature run through).

Web Browser Extensions

I personally use the Firefox as it offers me the best of performance and customization. Along with using a VPN I like to utilize some additional ‘extensions’ (or add-ons as they’re referred to by Firefox) to extend the behaviour of my web browser and to help reduce my ability to be tracked.

The below image shows some of the add-ons I utilize:

The ones of relevance here are:

Cookie Quick Manager: simple UI for editing/deleting cookies (useful to see what’s being set and by whom!).
Privacy Badger: blocks invisible third-party trackers.
ExpressVPN: allows for doing things like ‘double hops’, spoofing my location, blocking WebRTC and enforcing HTTPS.
NoScript: blocks JavaScript and allows me to configure how specific scripts are run and for how long they’re trusted.
uBlock: blocks ads, pop ups, and trackers.

Old add-ons I used to have installed…

Cookie AutoDelete: highly configurable way to delete cookies once you’ve finished browsing (whether it be changing domain, or tabs or browser restart).
DuckDuckGo Privacy Essentials: blocks hidden trackers, forces encryption via HTTPS, private search data, tools for grading websites based on their privacy performance.

In the case of ‘DuckDuckGo Privacy Essentials’ I removed that once I had ‘ExpressVPN’ and ‘Privacy Badger’ installed as I needed ‘ExpressVPN’ any way (and that has ‘HTTPS Everywhere’ built-in like DDG did), but also ‘Privacy Badger’ was much a more advanced tracker blocker than DDG was.

In the case of ‘Cookie AutoDelete’ I removed that once I installed ‘Privacy Badger’ and ‘NoScript’ as both those add-ons aided with the same problem.

Below is a screenshot of what the ‘NoScript’ add-on looks like:

Below is a screenshot of what the ‘uBlock’ add-on looks like when visiting google.com:

Below is a screenshot of what the ‘DuckDuckGo Privacy Essentials’ add-on looks like:

Also, each release of Firefox improves the Privacy settings available and so things like content blocking and preventing ‘browser fingerprinting’ is starting to be rolled out more generally for Firefox users:

Below is a screenshot of Firefox’s built-in “Do Not Track” setting (which is definitely worth enabling, even if respecting it is only voluntary):

Below is a screenshot of what Firefox looks like when inspecting a web page that has had content blocked:

But all that said, it’s still worth checking what your browser is doing. For example, Firefox enables specific types of data collection which may be worth you disabling:

Email

Email through free providers such as Google’s Gmail are definitely NOT private. Google has already admitted to reading their user’s emails in order to serve them advertising based on the content of their email.

If you are not paying for the service then you’re not a ‘customer’. You are in fact a ‘target’.

An alternative is to use an email provider that encrypts your messages and doesn’t have access to their content in any way. A good option (which has a free tier, but obviously you’ll use up that free tier pretty quickly) is Proton.

ℹ️ NOTE

due to how they encrypt your data, if you forget your password and try to reset it, you’ll lose all your email because they don’t have a means to decrypt those files unless you provide the relevant password.

Conclusion

Remember, staying ‘anonymous’ and ‘protecting your privacy’ are actually different things. The tools I use above will not help to keep my internet actions anonymous. If you want a good breakdown of what that would take (which is a lot of effort!) then read this article by Windscribe VPN.

Security is hard, and a never ending challenge to get right. But hopefully this post has given you some ideas about the various tools you can use to help you stay anonymous and to protect your privacy.

HTTP Caching Guide

Tue, 06 Aug 2019 00:00:00 +0000

Introduction

Caching is hard. Let’s try and understand it a little better.

ℹ️ NOTE

some sections are purposefully brief. I’m not aiming to be a specification document.

Caching at multiple layers

Caching can occur at both a ‘client’ level and a ‘cache proxy’ level.

Consider the following request flow architecture diagram…

In the above diagram, the “CDN” is a ‘caching proxy’ and so caching can (and of course does) happen there.

The box that’s labelled “proxy” in the above diagram isn’t a caching proxy, it’s just a standard ‘reverse proxy’ that’s figuring out which (of many) ‘origins’ the request should be proxied onto (meaning: caching does not happen there).

We’re able to control caching for both ‘clients’ and ‘cache proxies’, using the following two HTTP response headers:

Cache-Control: tells a ‘client’ (e.g. web browser) how caching should be handled.
Surrogate-Control: tells a ‘proxy’ (e.g. caching proxy/cdn) how caching should be handled.

In order to cache content efficiently we need to use a combination of the two headers.

ℹ️ NOTE

Surrogate-Control is typically stripped from the response, by a cache proxy, before the client receives it.

Cache-Control Directives

The Cache-Control cache response header has many directives you can configure, the following are some important ones to understand and will help you make an informed choice as to what values you set if you intend on controlling the caching of your content.

public: content can be cached by both client and proxy (implicit default).
private: content can be cached by client, but not proxy (as content is unique to the user).
max-age: determines how long (seconds) content is cached, after which requests will reach origin.
s-maxage: used by ‘shared cache’ proxies and is equivalent to Surrogate-Control: max-age=<...> (except not stripped).
must-revalidate: cached content can be served if TTL hasn’t expired, but do not serve ‘stale’ content under any circumstance.
no-cache: cached content must revalidate on all requests (i.e. serve stale is fine, but only after consulting origin).
no-store: prevents client or proxies from caching the content.
no-transform: proxies aren’t allowed to modify content (e.g. don’t send compressed content if origin didn’t).

References: MDN: Cache-Control and W3C Specification (see also: MDN: Caching).

client requests

One typically unmentioned feature of the Cache-Control HTTP header is that it can also be utilised at the client level for indicating to a cache what it will and wont accept.

This is an interesting perspective on caching that we rarely see.

In the case of Fastly CDN, they will ignore the Cache-Control header and its directives when provided as part of the client request.

Reference: RFC.

no-cache vs must-revalidate

A lot of the Cache-Control directives have subtle overlapping responsibilities, and no-cache/must-revalidate is one of the more confusing ones, so I’d like to take a moment to add some extra clarity…

With must-revalidate it’s suggesting that cached content must revalidate and not serve stale after the cached content’s max-age TTL has expired.

This is effectively saying “we have some content cached and its TTL is still valid so we’ll return that to you”, but the moment the max-age TTL expires it then changes to “sorry, we had cached content but it’s now stale and we’re not allowed to use it. we’ll go to the origin and grab fresh content for you.”

Unlike no-cache which is effectively saying “we have cached content, but before we release it to you we’re going to check with the origin that there isn’t a fresher version first”. It’s a way of enforcing a rigid ‘freshness’ plan.

Whenever using these (and other similar) directives, you need give consideration to what else needs to be in place to make them function efficiently. Specifically you need to be sure your origins are actually capable of handling revalidation (see ‘Serving Stale Content’ for more details), otherwise the behaviours might not work as intended or perform as well as you think they will.

Surrogate-Control Directives

For the Surrogate-Control cache response header you’ll mostly utilize the max-age directive along with the ‘extensions’ for serving stale content (e.g. stale-while-revalidate and stale-if-error).

There are some other directives defined in the W3C Specification, but it’s unclear how well those are supported by our primary CDN provider (Fastly) and so it’s best to avoid them unless you fully understand why you are setting them.

Fastly CDN

Fastly has some rules about the various caching response headers it respects and in what order this behaviour is applied. The following is a summary of these rules:

Surrogate-Control determines proxy caching behaviour (takes priority over Cache-Control) †.
Cache-Control determines client caching behaviour.
Cache-Control determines both client/proxy caching behaviour if no Surrogate-Control †.
Cache-Control determines both client/proxy caching behaviour if it includes both max-age and s-maxage.
Expires determines both client/proxy caching behaviour if no Cache-Control or Surrogate-Control headers.
Expires ignored if Cache-Control is also set (recommended to avoid Expires).
Pragma is a legacy cache header only recommended if you need to support older HTTP/1.0 protocol.

† except when Cache-Control contains private.

It’s worth reiterating a segment of the above priority list which is that Cache-Control can include serving stale directives such as stale-while-revalidate and stale-if-error, but they are typically utilized with Surrogate-Control more than they are with Cache-Control. If Fastly receives no Surrogate-Control but it does get Cache-Control with those directives it will presume those are defined for its benefit.

Client devices (e.g. web browsers) can respect those stale directives, but it’s not very well supported currently (see MDN compatibility table).

If you’re interested in learning more about Fastly (inc. Varnish and VCL), then read my blog post on the topic.

Default TTLs

The CDN (Fastly) has some rules about how long it will cache content for.

Below is a (oversimplified †) summary of these rules.

If your origin doesn’t set a cache response header, then cache content TTL will be 1hr.
1hr TTL is set by Fastly’s VCL boilerplate (applied by default).
1hr TTL can be overridden by your own custom VCL.

† Fastly has many factors it takes into account when deciding if an object stays in its cache (see: LRU).

Disable Caching

Depending on your requirements, when trying to disable caching it can be confusing to know which cache control directives to utilize due to the various permutations. Below is a Fastly recommended list of directives your origin can use for handling three common scenarios.

Disable Client Caching (docs): Cache-Control: no-store, must-revalidate
Disable Proxy Caching (docs): Cache-Control: private
Disable ALL Caching (docs):
- Cache-Control: no-cache, no-store, private, must-revalidate, max-age=0, max-stale=0, post-check=0, pre-check=0
- Pragma: no-cache
- Expires: 0

ℹ️ NOTE

regarding the disabling of caching at the client level, I reached out to Fastly because of their suggested use of must-revalidate with no-store (which doesn’t make sense). They have since consulted with their resident RFC expert who confirmed this was redundant, and so expect their documentation to be updated to just no-store.

It’s also worth mentioning that Fastly’s use of post-check and pre-check is also redundant as per this old Microsoft article that states setting them to zero does not actually ‘do anything’!

Serving Stale Content

As you saw at the beginning of this post, we have a ‘reverse proxy’ that’s placed in front of our origin servers. This proxy will configure (as a default) cache settings that will result in stale versions of our origin’s cached content being served when the max-age TTL for that content has expired.

There are two distinct types of serving stale logic:

stale-while-revalidate: serve stale content while asynchronously checking for fresh content.
stale-if-error: serve stale content if request to origin fails.

A key part of the stale-while-revalidate flow is for the origin to indicate when a particular resource has been ‘refreshed’ (i.e. a newer version is available). This is achieved by the origin providing either an ETag (entity tag) or a Last-Modified response header.

The ETag and Last-Modified response headers (sent from the origin) are used as values in a ‘conditional request’ made by the cache (client or proxy cache) to determine if the origin should send back either a full response or a smaller 304 Not Modified if the revalidated content hasn’t changed.

The benefit of a conditional request is that we’re able to reduce bandwidth whenever a 304 Not Modified is returned from the origin, because it will be sent with an empty response body (where as, of course, 200 OK will include the full response body).

There are many conditional headers, but the following are the most common when dealing with ETag/Last-Modified:

ETag: If-None-Match
Last-Modified: If-Modified-Since

Reference: sequence diagrams demonstrating the various request flows.

If neither ETag nor Last-Modified is sent by the origin, then the cache will not be able to update the cache object. This means the object’s TTL (i.e. max-age) will still be expired, and other properties of the cache object will also not be updated, such as its ‘age’ (reset it back to zero once the content is refreshed), nor its ‘grace’ period (how long it will be able to serve that content stale for).

ETag or Last-Modified?

The official W3C specification provides ‘rules’ for when to use ETag vs Last-Modified. In summary…

the preferred behavior for an HTTP/1.1 origin server is to send both a strong entity tag and a Last-Modified value.

A good additional reference is MDN’s article on Cache Validation.

Revalidation TTL

One aspect of serving stale content that normally confuses people is how to determine the TTL for stale-while-revalidate. It would seem that determining a TTL for stale-if-error is fairly straight forward, in that you’ll just pick an arbitrarily long time period to serve stale, while the stale-while-revalidate directive isn’t as simple.

Consider the following diagram which highlights a typical request flow when using (for example) an ETag to handle the revalidation step:

ℹ️ NOTE

this diagram presumes the use of a CDN like Fastly which has specific behaviours, such as ‘request collapsing’ built-in.

In this request flow we can see that although we’re successfully serving stale content when a cached object’s TTL has expired, this is still potentially going to result in multiple requests to the origin (rather than acting as a cache HIT) if we have an influx of requests for the same resource.

Now with that said, the ability to reach the origin (if we’re using Fastly at least, other CDN providers may differ) is only going to be on a datacenter by datacenter basis. The reason being, each cache node will perform request collapsing which will mitigate the damage of having to allow a request through to your origin.

With this in mind, having a large stale-while-revalidate TTL might not necessarily be a good idea because ultimately for that time period, if there is no updated version of the content, new client requests are going to be able to reach the origin.

Yes, it’s not going to be millions of requests (even if you’re a globally distributed brand), and Yes, having an empty response sent back from origin with the 304 Not Modified is better as far as bandwidth consumption is concerned, but the origin still has to spend time and resources constructing the response.

So depending on your origin and potentially the costs related to these types of requests it might be better to have a shorter stale-while-revalidate TTL so that it would expire more quickly.

This would then mean the request would go back to the origin sooner, in order to get a full response back to be re-cached, which would then result in future client requests actually getting a cache HIT and saving the origin from having to handle that extra unnecessary load.

On the ‘flip side’, I think the actual question needed to be asked is: “how important is it that you get fresh content as quickly as possible”?

The answer to that will firstly depend on whether we were using a CDN where purging content dynamically was not possible (specifically whenever fresh content was published by an origin). Fastly enables this dynamic purging by providing support for setting a Surrogate-Key HTTP response header.

So if we were in that situation where we couldn’t dynamically purge our CDN cache, and the freshness of our content was important then I would imagine we would set a longer stale-while-revalidate TTL in order to force the caching proxy to attempt to revalidate more often.

Otherwise, again if we were in that situation where we couldn’t dynamically purge our CDN cache and we had ended up setting a short stale-while-revalidate TTL, then that would mean we’d go back to origin and get a new max-age TTL set (which could be set to a very long value) and so we could end up with users getting a cache HIT for content which for all extensive purposes could very well be stale anyway but you would lose the opportunity to try and acquire fresh content via revalidation now.

If you’re using a CDN such as Fastly, you can utilize Surrogate-Key to purge your content dynamically whenever fresher versions have been published. Meaning, you could have a short revalidation TTL and if there was no fresh content within that time period you wouldn’t actually have to worry about going to origin and getting the same content back but now with a long max-age TTL, because you know you could dynamically trigger a cache MISS whenever your fresh content was published anyway.

Open Question: do you think stale-while-revalidate should contain a long or short TTL (and why)?

Strong and Weak Validators

As far as the actual setting of an ETag is concerned, this isn’t necessarily as straight forward as you might first imagine. For example, a typical approach is to use a hash function to generate a digest of the response body to verify the content has changed.

Of course there is the potential for the hash function to not be robust enough to avoid hash conflicts, but additionally there is a concept referred to as “validation” which needs to be considered (e.g. should the ETag be marked as being a “strong” validator or a “weak” validator).

These are things that you’ll need to consider when generating an ETag for a resource, and it’s recommended you read documentation on what constitutes a “strong” or “weak” validator.

ℹ️ NOTE

it’s important to realize that generating an ETag and figuring out the Last-Modified date of a resource is outside the responsibility of a proxy, hence the proxy sat in front of our origins doesn’t set these headers even when they aren’t set by the origin.

In essence a strong ETag indicates that the resource’s content is the same with regards to both the response body and the response headers, whereas a weak ETag indicates that the two representations are semantically equivalent. It compares only the response body. Weak ETags are prefixed with W\ and thus can easily be distinguished between weak and strong.

Cache Headers Example

Let’s wrap up by considering a real-world example of cache headers and what they look like, and why.

The request flow architecture that I have at work takes the form of:

Client > CDN > Load Balancer(LB) > Proxy > LB > Proxy > LB > Origin

There are reasons for the multiple proxy layers in front of our origin servers, but I’m not going to dig into that here.

The proxy nearest the front of that list is acting as a gateway of sorts, and so if our origins fail to set any caching instructions, that proxy will add some defaults on their behalf.

This is all documented as service contracts, and so these origins are choosing to opt into those defaults, but they have full control over how their content is cached if they so choose to.

This proxy layer will by default tell the client to not cache your content, while indicating to any ‘caching proxies’ (e.g. Fastly CDN) that they should cache your content:

Cache-Control: no-store
Surrogate-Control: max-age=86400, stale-while-revalidate=60, stale-if-error=31536000

ℹ️ NOTE

values are in seconds, so 86400 = 1 day, 60 = 1 minute, 31536000 = 1 year.

The reason for choosing to only cache content at the CDN rather than the client is because we have very granular control over our CDN cached content (thanks to our CDN provider, Fastly) and so its preferable, for our situation, to have complete control over the caching of our content rather than let a client’s browser determine what happens.

For example, if we didn’t do this and we allowed the client to cache our content in their own private caches, then it would mean if we had a request from our legal department to take down a piece of content, we wouldn’t be able to remove it from the client’s cache as that’s outside of our control.

Things can get a bit murky if there are proxies placed in front of our CDN (e.g. ISP proxies or maybe client is inside a local corporate network that runs caching proxies), but we set no-store which should tell clients/proxies to not cache. The reason this works with Fastly is because it doesn’t respect no-store (it sees that as a client cache directive), it instead respects private (which is more appropriate), and so that’s another way of us hopefully catching any unknown proxies in front of our CDN.

All of this leads us to the fact that when we purge our cache, we feel confident that we’re capable of successfully purging it from the internet as a whole.

Conclusion

Yes, caching is hard but it’s also not rocket science either. Take some time to read over the Cache-Control directives. Try to get comfortable with them.

It’s our responsibility as engineers building HTTP services to understand the platform we’re building services upon.

❤️

New Laptop Configuration

Wed, 10 Apr 2019 00:00:00 +0000

Introduction

I’m an engineer with a new laptop, which requires setting up with various development tools and configuration. This post is my attempt to capture and document my process for getting a new dev environment set-up.

I used to try and automate a lot of this with bash scripts, but realised over time that things go out of date quite quickly (e.g. OS configurations can change substantially, as well as my preferred ways of working).

I also find that if an error occurs with an automated script (unless you’ve coded things defensively enough) you can end up with your machine in a weirdly broken state.

Given a straight forward set of instructions, doing things manually doesn’t take long at all, and you can modify things at that point in time without just blindly installing various things you no longer need.

Defaults

It’s good to begin by surveying your current system and understanding what you have already installed. For me this looked something like:

OS: macOS Mojave (10.14.4)
Curl: /usr/bin/curl (7.54.0)
Bash: /bin/bash (3.2.57)
Python: /usr/bin/python (2.7.10)
Ruby: /usr/bin/ruby (2.3.7p456)
Git: /usr/bin/git (2.20.1)
$PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

What’s worth me noting additionally here is that I primarily use two programming languages: Go and Python. The reason I mention this is because Python has an interesting history with the name of its binaries.

The binary name python generally refers to Python version 2.x. Where as Python 3.x has traditionally been named python3 to help distinguish the two. So looking above we can see which python reveals the location as /usr/bin/python and without checking the version (e.g. python --version) I was fairly certain it would be a 2.x version (based on the naming history).

This has been the generally accepted rule for a while, except! when dealing with tools that handle virtual environments.

For example, pipenv is a tool that helps you to manage not only different Python versions but also the dependencies installed for different projects (referred to as virtual environments). A tool like pipenv will proxy a command such as python through a shim script (e.g. /Users/integralist/.pyenv/shims/python) and that shim script will then determine which Python interpreter/binary to execute.

A shim script typically identifies the virtual environment you’re working under and will then figure out the most appropriate Python interpreter to invoke. So within that virtual env if you call python, then you may not necessarily get the Python2 interpreter, as your virtual env might be configured such that the expectation is to proxy your invocation to a Python3 interpreter.

This is why, when setting up a new laptop, getting a good development environment setup is essential because it can get quite confusing untangling a mess of default Python’s vs brew install ... versions of Python 3 and then subsequently using multiple environment tools like pipenv which confuse things further by hiding the actual versions behind the generically named python command.

The situation reminds me a lot of XKCD’s classic comic strip…

Package Manager

Let’s begin our journey by first installing a ‘package manager’. This software will enable us to search and install various pieces of software. The macOS provides its own GUI implementation referred to as the ‘App Store’, but it’s heavily moderated by Apple and an app can only be found there if it abides by Apple’s own set of rules and criteria for what they consider to be ‘safe’ software.

ℹ️ NOTE

there are many apps that aren’t available in the App Store because Apple can be a bit anti-competition (see Spotify’s “time to play fair” campaign).

So we have to download our own package manager, and the defacto standard for macOS is a program called Homebrew (which is a terminal based tool, so no GUI). In fact, I’m not actually sure what alternatives to Homebrew exist (other than MacPorts, which if you want to understand the differences between it and Homebrew then read this)? On Linux you have tools such as yum or apt but for macOS you either use the built-in App Store or find your own alternative (so in this case, we’ll use Homebrew).

To install Homebrew, execute the following command in your terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

ℹ️ NOTE

notice that installation command uses the default installation of Ruby which Homebrew presumes is available (and for the most part is a safe presumption to make as Ruby as been provided by macOS for the longest time).

If you need to update Homebrew you can execute a brew update command, but the installation will install the latest version any way, so that won’t be necessary.

Essential Packages

OK, so I start with what I refer to as a ‘essential packages’, and specifically these are packages that do not require any configuration on my part. Meaning, I can install them and consider the job done, where as with other packages I install I’ll have to make some additional tweaks to (which we’ll see as we move on past the ‘essential’ segments of this post).

To install a package via Homebrew, execute the following command in your terminal:

brew install <package_name>

Here is a list of the packages I’ll install:

ag: a grep like tool (aka. the_silver_searcher)
gnu-sed: it’s the gnu version of sed (gsed) used for filtering/transforming text
jq: tool for parsing/inspecting json output
docker: useful for running containerized programs
hugo: static site generator (used to make this website)
node: server-side js programming language (used to compile a static search feature for my website)
pwgen: generates random passwords
reattach-to-user-namespace: used by tmux for clipboard storage
shellcheck: bash linter
transmission: torrent client † (alt. npm install -g t-get)
tree: displays directory heirarchy structures as a tree
watch: executes given command every N seconds

† see transmission user guide

Here’s a handy one-liner:

brew install ag gnu-sed jq docker hugo node pwgen reattach-to-user-namespace shellcheck tree watch

Essential Apps

Homebrew now allows you to also install GUI applications, not just command line tools, but to do this you’ll need to configure Homebrew to use Cask:

brew tap homebrew/cask

Once that’s done you can install an app via Homebrew using the command:

brew cask install <app_name>

Here is a list of the apps I’ll install:

alfred: like Apple’s Spotlight search, but better
caffeine: stops the Mac from going to sleep
dash: offline documentation
docker: this is the counter-part to the ‘package’ installed earlier †
google-backup-and-sync: syncs files between computer and Google Drive
google-chrome: web browser
lepton: GitHub Gist UI
slack: communication tool
spotify: music streaming service
vlc: video player with support of lots of codecs

† if you installed the docker ‘package’, then you need the docker ‘app’ as well for it to work. You can’t have one without the other (this is because the app sets up the interface for macOS to interact with the underlying docker client/server implementation).

Here’s a handy one-liner:

brew cask install alfred caffeine dash docker google-backup-and-sync google-chrome lepton slack spotify vlc

The Dash app will ask you what documentation you would like to download so it’s available offline. I use the following docsets (I used to have lots more but realised I never really used them, so this is my ‘essential’ docs list):

boto3
Go
HTTP Header Fields
HTTP Status Codes
NGINX
Python2
Python3
Regular Expressions
tmux
Tornado
vim

A couple of apps probably worth mentioning are:

Curl

I like to use a more modern version of curl (e.g. supports HTTP/2, and other features):

brew install curl

But in order to use this version of curl you’ll need to modify your $PATH:

export PATH="/usr/local/opt/curl/bin:$PATH"

Git

Similarly to curl, I like to have the most recent version of git installed:

brew install git

Once that’s installed I configure it like so:

curl -LSso ~/.git-prompt.sh https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh
curl -LSso ~/.gitignore-global https://raw.githubusercontent.com/Integralist/dotfiles/master/.gitignore-global
curl -LSso ~/.gitconfig https://raw.githubusercontent.com/Integralist/dotfiles/master/.gitconfig

ℹ️ NOTE

it’s always worth checking ~/.gitignore-global is up to date (i.e. not referencing file types I no longer work with).

Shell

To install and configure latest version of the Bash shell, execute the following commands:

brew install bash
echo /usr/local/bin/bash | sudo tee -a /etc/shells
chsh -s /usr/local/bin/bash

Also make sure to install auto-complete for bash:

brew install bash-completion

Finally, we’ll configure Bash like so:

curl -LSso ~/.bash-preexec.sh https://raw.githubusercontent.com/rcaloras/bash-preexec/master/bash-preexec.sh
curl -LSso ~/.bashrc https://raw.githubusercontent.com/Integralist/dotfiles/master/.bashrc
curl -LSso ~/.bash_profile https://raw.githubusercontent.com/Integralist/dotfiles/master/.bash_profile

ℹ️ NOTE

~/.bashrc references ~/.fzf.bash which is needed, and comes from installing the FZF vim plugin (which we’ll sort out shortly).

Terminal

To install my custom terminal theme:

curl -LSso /tmp/Integralist.terminal \
https://raw.githubusercontent.com/Integralist/mac-os-terminal-theme-integralist/master/Integralist.terminal
open /tmp/Integralist.terminal
rm /tmp/Integralist.terminal

ℹ️ NOTE

don’t forget to change the terminal font to menlo (if not already set) and also set Integralist theme as your default. I used to do this via defaults write com.apple.Terminal "Default Window Settings" Integralist and defaults write com.apple.Terminal "Startup Window Settings" Integralist but those have changed now in the latest macOS (see defaults read).

UPDATE 2020.09.08

I’ve reverted to using the “Basic” theme provided by macOS, and just modifying the font to be “Menlo Regular 16 pt.”

GitHub

Let’s now set-up a new SSH key for GitHub access:

mkdir ~/.ssh
cd ~/.ssh && ssh-keygen -t rsa -b 4096 -C 'foobar@example.com'
eval "$(ssh-agent -s)"
ssh-add -K ~/.ssh/github_rsa

Don’t forget to pbcopy < ~/.ssh/github_rsa.pub and paste your public key into the GitHub UI. Once that’s done you can execute the following command to test your SSH key is set-up correctly and working:

ssh -T git@github.com

ℹ️ NOTE

there is a slight catch-22 here which is if your password for GitHub is in your Password Store (see next section), then that makes things trickier. For me I also have a copy of the encrypted store on my phone and so I can utilise that to access the password. But failing that, you can just ‘reset your password’ in GitHub UI’s and follow the email instructions to gain access and thus login and add your new SSH key.

Password Store

I use the open-source password store for handling secrets and passwords. This tool provides the pass command, and that requires the use of gpg, so let’s start by installing GPG:

brew install gpg

Now you have gpg, make sure you pull your private key from wherever you have it stored (e.g. external USB stick), then execute:

gpg --import <private-key>
gpg --export <key-id> # public key by default

ℹ️ NOTE

don’t forget you can sign your git commits:
git config --global user.signingkey <key-id>

Next, install pass and pass otp commands:

brew install pass pass-otp zbar

You can now pass a QR code into pass otp and use the terminal for generating one-time pass codes for 2FA/MFA authentication:

zbarimg -q --raw /tmp/qr.png | pass otp insert Work/Acme/totp/foo`  

pass otp -c Work/Acme/totp/foo

ℹ️ NOTE

installing zbar provides the zbarimg command

Lastly, we need to setup a new Password Store, and to do that we need to provide our GPG key id:

# <ref> needs to be your email, or part of the key's description
keyid=$(gpg --list-keys <ref> | head -n 2 | tail -n 1 | cut -d ' ' -f 7)

pass init $keyid

Now we can pull our Password Store from a private repository:

pass git init
pass git remote add origin git@github.com:Foo/Bar.git
pass git pull
git branch --set-upstream-to=origin/master master

ℹ️ NOTE

I also like to ensure my encrypted datastore is sync’ed up to other online providers, and symlinked to ~/.password-store as well so changes are backed up automatically in multiple places.

Mobile Password Store

There is also a mobile app for Android that you can download from the Google Play store (and other places) that allows you to access the Password Store if it has been pushed to a distributed version-control system such as GitHub (better still if the repository is private – “out of sight, out of mind”).

To get set-up, go through the following steps:

Install Password Store app.
Select “Clone from Server” option.
Add in github credentials (e.g. git@github.com:Foo/Bar.git)
Create new SSH key via Password Store app (give it a password).
Encrypt your SSH key with symmetrical encryption (e.g. gpg --symmetric)
Email SSH key to self.
Decrypt SSH key and copy it into GitHub UI.
Password Store app will ask for SSH key password, then it’ll clone the repo.

Before you can access the content of the Password Store (remember all the content is individual GPG keys) you’ll need the GPG private key in order to decrypt files that would have been encrypted using your public key.

Export your private key as ASCII (via laptop: gpg --armor --output passkey.txt --export-secret-keys <key_id>).
Encrypt your exported private key with symmetrical encryption (gpg --symmetric passkey.txt)
Email your private key to yourself.
Download private key to your phone.
Install OpenKeychain app (via Google Play)
Locate the downloaded encrypted private key and open with OpenKeychain app.
Enter password used to encrypt the private key.
Then click into the extracted passkey.txt file and then click “Import”.
In Password Store app set the options:
- Select “OpenPGP Provider” (choose: OpenKeychain)
- Select “OpenPGP Key id” (choose: the imported public key)

Go

We’ll install the latest version of go (as far as Homebrew is concerned):

brew install go

This is required because in order to handle different versions of go we’ll want to manually compile go, and that ironically requires a version of the go compiler.

Finally, make sure the default Go directory is set in your $PATH so that any installed binaries will be available:

export PATH="$HOME/go/bin:$PATH"

Python

The macOS comes only with Python 2.x and although the specific version should (according to the Python docs) have the pip command available, that’s not the case. So we have to install pip for Python2 manually using the very old (but built-in) easy_install command:

sudo easy_install pip

Now when running pip --version we should see:

pip 19.0.3 from /Library/Python/2.7/site-packages/pip-19.0.3-py2.7.egg/pip (python 2.7)

At this point I’m going to ask you to read Python Management and Project Dependencies which is separate/dedicated post I wrote about installing multiple Python versions and how to utilize virtual environments.

Python Packages

Here are some packages I like to install as a general rule…

black: formatter (like golang’s gofmt)
flake8: linter
flake8-import-order: validates imports
mypy: static analysis
ipython: REPL
pytest: testing framework
structlog: structured logging
tornado: async web framework
boto3: AWS CLI tool
tox: generic virtualenv management and testing tool

ℹ️ NOTE

for an example of how to configure Flake8 and its plugins, see this gist.

I would also strongly recommend installing the following tools:

isort
autopep8
unimport

Here’s a one liner to install some of these packages that I’m guaranteed to use in all projects…

python3 -m pip install isort autopep8 unimport tox mypy flake8 flake8-import-order

I then reference these in my .vimrc:

" Execute Python isort
autocmd BufWritePost *.py :execute '!isort %' | edit

" Execute Python autopep8
autocmd BufWritePost *.py :execute \
'!autopep8 --experimental --verbose --aggressive --aggressive --recursive --in-place %' | edit

" Execute Python unimport
autocmd BufWritePost *.py :execute '!unimport --remove %' | edit

If you’re using pipx (a tool that helps to install packages as self isolated binaries) just be sure the pipx ensurepath call doesn’t update the shell PATH by appending the /Users/integralist/.local/bin but by prepending it instead. This might require you to manually update your ~/.bash_profile like so:

export PATH="/Users/integralist/.local/bin:$PATH"

ℹ️ NOTE

if you install pipx via Homebrew then it’ll be attached to that Python version. Meaning if you upgrade your Python version, then pipx could break (e.g. none of the installed packages will work). The solution is to run pipx reinstall-all --python python3.

Vim

You can either install more recent version of vim via Homebrew:

brew install vim

Or you can manually compile vim yourself:

ℹ️ NOTE

I manually compile vim as I need Python3 support baked in, which Homebrew’s version no longer does (it used to, but not any more). Python3 support means my Python linting tools will work as expected.

So let’s start at the beginning. To manually compile Vim you would think to do something like the following…

git clone https://github.com/vim/vim.git

cd vim

./configure --with-features=huge \
            --enable-multibyte \
            --enable-rubyinterp=yes \
            --enable-python3interp=yes \  # relies on `brew install python3`
            --enable-perlinterp=yes \
            --enable-luainterp=yes \
            --enable-gui=gtk2 \
            --enable-cscope \
            --prefix=/usr/local
            
make && make install

The above code will copy the compiled vim binary into /usr/local/bin so which vim will show /usr/local/bin/vim

This works but I’ve found that it only works when the python3 interpreter is the same version as what Vim is itself internally expecting to be available.

What I mean by that is I recently upgraded my Python version to 3.7.7 and Vim suddenly broke. I tried the above compilation and it didn’t work. I could see from the errors being printed that Vim was looking around my system for a Python version 3.7.4 which didn’t exist (hence Python3 support wasn’t compiled into vim).

The solution was to firstly to tell Vim what version of Python3 to use, and secondly (and just as important) to ignore any previously cached aspects of a compilation (e.g. if I tell you to use Python 3.7.7 don’t then go and try to be helpful and use a cached run where I was using 3.7.4 – which really confused me for a long time!):

ℹ️ NOTE

now I should say, that if you can use the above example compilation code, but just run make clean distclean first, then do that! I suspect I could have done that and Vim would have been compiled with Python 3.7.7 just by using the --enable-python3interp flag set. I didn’t think about that at the time though, hence the following still worked.

make clean distclean

./configure --with-features=huge \
  --enable-multibyte \
  --enable-rubyinterp=yes \
  --enable-python3interp=yes \
  --with-python3-command=\
  /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/bin/python3.7 \
  --with-python3-config-dir=\
  /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/ \
  --enable-perlinterp=yes \
  --enable-luainterp=yes \
  --enable-gui=gtk2 \
  --enable-cscope \
  --prefix=/usr/local

make && make install

The key flags are…

--enable-python3interp: tell the compilation you want Python3 support
--with-python3-command: give it a path to a Python3 interpreter/binary (†)
--with-python3-config-dir: a configuration directory used by the version of Python3 you want to use.

† e.g. if I run that full path in my terminal shell it’ll actually run the Python3 REPL so I know it’s a valid path to provide.

Things get even more confusing when you are using a Python version manager like pyenv as that overrides the Python interpreter version. So although Vim might report it’s using Python 3.7.7 (as shown in the vim Ex command below), if you shell out to a command like isort (e.g. !isort %) you’ll find that the shell will complain no such command exists.

This is because the command doesn’t exist. Not for the version of Python that’s running in the shell! The shell is running whatever version of Python pyenv has activated. So you need to make sure when you start vim that you activate a virtual environment that has these tools available.

Here is the Ex command to see what version of Python vim is compiled with:

:py3 import sys; print(sys.version)

3.7.7 (default, Mar 10 2020, 15:43:03) 
[Clang 11.0.0 (clang-1100.0.33.17)]

So as I mentioned, the approach I take with Vim is to activate a specific virtual environment when in a project repo.

This could be a pyenv virtual environment but actually it can just be a standard Homebrew Python virtual environment:

# in a new shell where pyenv has no affect on the python interpreter

python3 -m venv venv/vim
source venv/vim/bin/activate
python3 -m pip install isort autopep8 unimport tox mypy flake8 flake8-import-order

Now I know that if I start up a new shell and cd to my project repo, and even if pyenv has set the python interpreter I can activate the Homebrew Python virtual environment I created (which is going via the Homebrew installed version of Python) and Vim will know about the packages installed in that virtual environment.

ℹ️ NOTE

even if I do python3 --version the shell will now report the Homebrew version of Python (so it overrides the pyenv version that might have been set via a .python-version file)!

Next, I configure vim with vim-plug plugin manager:

curl -fLo ~/.vim/autoload/plug.vim \
  --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
curl -LSso ~/.vimrc https://raw.githubusercontent.com/Integralist/dotfiles/master/.vimrc

Ensure Vim is configured with spell checking options:

vim -E -s <<EOF
:set spell
:quit
EOF

Install plugins by opening vim and executing:

:PlugInstall

ℹ️ NOTE

fzf doesn’t need a brew install when installed via vim. See my .vimrc configuration file for more details, but in essence it contains: Plug 'junegunn/fzf', { 'dir': '~/.fzf', 'do': './install --all' }

Also ensure the Golang environment has what it needs by executing:

:GoInstallBinaries

Tmux

Install tmux:

brew install tmux

Configure tmux and expose tmuxy command (defined in my ~/.bashrc for quickly spinning up a new working environment):

curl -LSso ~/.tmux.conf https://raw.githubusercontent.com/Integralist/dotfiles/master/.tmux.conf
curl -LSso ~/tmux.sh https://raw.githubusercontent.com/Integralist/dotfiles/master/tmux.sh

ℹ️ NOTE

check $PATH to make sure tmux isn’t double setting values in your PATH as it starts up. If it does you can check an older version of my ~/.bash_profile for a work-around.

Miscellaneous

Not every app can be installed via Homebrew. Monosnap is one such example.

If you want an easy way to hide menu bar items, then try the hidden-bar app (github)

Also, if you’re into torrents, then the transmission server/client (or alternatively npm install -g t-get) might be of interest to you.

macOS

It can be cool to configure the macOS via the terminal, things like mouse cursor speed or keyboard repeat key performance. But unfortunately that all changed with macOS Mojave and I couldn’t be bothered to figure out the correct way to do it via the terminal when doing the setup via the GUI is just as quick (and I know the few things I like to tweak off-by-heart).

For example, you used to be able to do things like:

defaults write NSGlobalDomain ApplePressAndHoldEnabled -bool false

But since macOS Mojave those settings and namespaces seem to have changed. If you’re interested in figuring it out, then I’d recommend starting with:

defaults read

The above command will display all the current macOS settings for you. From there you can drill down into individual namespaces like so:

defaults read "Apple Global Domain" com.apple.mouse.tapBehavior

ℹ️ NOTE

here are the settings I used to configure via the terminal.

One thing I like to do is to make sure macOS’ “Spaces” feature doesn’t rearrange spaces based on their recent usage, and to do that you need to open up the ‘Mission Control’ settings panel and disable the option:

Automatically rearrange Spaces based on most recent use

Multiple Branches in Git

Fri, 22 Mar 2019 00:00:00 +0000

Introduction

There are times where you might be working from a particular git branch and need to quickly jump over to a different branch to do some urgent work.

Typically you would need to first git stash anything you were working on (as it’s unlikely to be in a state where it can be committed), and then you’d have to leave your current branch to create a new branch from master and thus begin working on your new urgent task.

This is a fairly straightforward workflow, but there is a mild annoyance which is that I happen to git stash a lot and I find when jumping over to a new branch to do some urgent work that I might end up git stash‘ing a few more times along the way.

Ultimately, when I’m done with my urgent task and ready to go back to my other branch, I then have to sift through my stash to find the relevant one I want to pop. OK so not that tragic considering git stash list will indicate the branch on which the stash was taken (which helps), but I do then need to Google what the syntax is for popping a specific stash (e.g. it’s git stash apply stash@{n} where n is the index you want to apply.)

ℹ️ NOTE

for the life of me I wish I could remember the syntax but it just eludes me every time.

Oh, and then you have to think about whether you actually want to use apply, which leaves the stashed changes in the stack, or if you meant to actually pop the stashed content (git stash pop stash@{n}) so it’s properly removed from the stack.

This is where I was recently introduced to a concept in git referred to as a ‘worktree’ (thanks Kiran).

Worktree

Git offers a feature referred to as a worktree, and what it does is allow you to have multiple branches running at the same time.

It does this by creating a new directory for you with a copy of your git repository that is synced between the two directories where they are stored.

This is different to manually creating a new directory and git cloning your repo down, because with the worktree model the two sub directories are aware of each other.

ℹ️ NOTE

as you’ll see, although this workflow is pretty cool, you could argue that git stash is just plain simpler and easier for a human mind to reason about. I’ll leave that up to the reader to decide.

Example

In the following example I’m going to create a new git repo. I’ll make a change in master, then create a new branch for doing some work. We’ll then imagine that I have been given an urgent task that I must complete now and yet my current non-master branch is in such a state that I want to avoid just stashing everything.

ℹ️ NOTE

I use tmux to split my terminal into multiple windows, and this demonstration will require two windows (or two separate terminal instances if you’re not using a screen multiplexer) for the sake of demonstration.

Create a new repo

mkdir foo_project
cd foo_project
touch foo
git add foo
git commit -m "created foo file"

Create a new branch

git checkout -b foo_contents
echo 123 > foo
git add -u
git commit -m "added content to foo"

Now I’ll create a new file and stage it for committing, but I won’t commit it (this is where we pretend my branch is in some hideously complex state).

Create new worktree branch

git worktree add ../foo_hotfix

ℹ️ NOTE

you’ll want to create the new worktree in a directory outside of your current repo’s directory (just so there’s a clear distinction).

At this point you’ll find your current terminal is still in the same foo_contents, but there is now a new directory called foo_hotfix outside your current repo’s directory.

Make changes in new worktree branch

Open up a new terminal (or split window) and run through the following steps:

cd ./foo_hotfix (or cd ../foo_hotfix if your new terminal is currently set to your main git repo directory)
git log

OK, so if you do a git log you’ll find that the worktree has a branch automatically created and named after the worktree (so the branch is called foo_hotfix in my case).

The important thing to realize is that git worktree add is a bit like git branch in that it creates the new worktree from the current branch you’re in. Meaning that my foo_hotfix branch has the “added content to foo” commit from the foo_contents branch as that’s where I ran the git worktree add command from.

This is what git log looks like for me in this new worktree:

* d374dcb (Integralist) - (HEAD -> foo_hotfix, foo_contents) added content to foo (2 minutes ago)
* 9ae3a7f (Integralist) - (master) created foo file (3 minutes ago)

I don’t want the commit d374dcb in there as it’s coming from a branch (foo_contents) that’s still in progress, and so I’ll need to rebase out that commit:

git rebase -i 9ae3a7f

ℹ️ NOTE

the rebase editor opens and I change pick to drop to get rid of the commit.

Now at this point I have a new working directory that I can work in:

echo hotfix > baz
git add baz
git commit -m "some hotfix"

Merge my hotfix back into master

I’m going to change into my master branch, but remember I’m still in the foo_hotfix directory, so my main repo directory foo_project (open in another terminal window) is still in the foo_contents branch).

git checkout master
git merge foo_hotfix

Removing the worktree

OK, so at this point we’ve merged our hotfix into master. I want to go back to my original repo directory and make sure I have the latest master rebased in before continuing on with my foo_contents work.

To remove the worktree you can either remove it using the git interface (e.g. git worktree remove foo_hotfix) or manually remove it (e.g. cd ../ && rm ./foo_hotfix), where git will, at some point in the future, internally run a prune and remove any references to this orphaned branch/working tree (you could also manually trigger that prune using git worktree prune).

ℹ️ NOTE

if I do git worktree remove foo_hotfix while currently residing inside the foo_hotfix directory, I’ll find that the .git repository is removed from the directory.

Continuing working on my feature branch

Presuming I’m still in the foo_hotfix directory and that’s where I ran git worktree remove foo_hotfix:

cd ../foo_project
git rebase master < whoops! I need to stash my changes first †
git stash pop

† why yes, this does seem a bit strange considering that’s what I was trying to avoid in the first place, but in this case it’s a single ‘stash’ and so a simple git stash pop will suffice to get me back to where I need to be.

I can now continue working on my foo_contents branch.

Conclusion

Well, this was fun heh! 😉

Do you think you have any uses for git’s worktree feature?

Let me know on twitter.

Algorithms in Python

Wed, 13 Mar 2019 00:00:00 +0000

In this post I’ll be demonstrating a few common algorithms using the Python language. I’m only covering a very small subset of popular algorithms because otherwise this would become a long and diluted list.

Instead I’m going to focus specifically on algorithms that I find useful and are important to know and understand:

Sorting
- Merge Sort
- Quick Sort
Searching

Three out of the four ‘search’ algorithms listed above will be implemented around a graph data struture. Graphs appear everywhere in life. For example, your Facebook list of friends, mutual friends, and extended friends (i.e. friends of your friends who you don’t know) is a perfect example of a ‘graph’.

Graphs can also be ‘weighted’, so they can indicate that a relationship between two nodes within the graph are possibly stronger than another connection, and is typically used in road maps for determining the quickest path to a particular node (we’ll come back to this later when reviewing Dijkstra’s Algorithm).

Merge Sort

Given an unsorted list of integers, the merge sort algorithm enables us to sort the list.

Best: Ω(n log(n))
Average: O(n log(n))
Worst: O(n log(n))

The algorithm can be broken down into the following pseudo-steps:

Recursively split given list into two partitions.
Divide the list until reaching its smallest partition.
Recursively merge each pair of partitions (repeat till reconstructed list).

That short list actually defines two distinct processes: a ‘break up into partitions’ step followed by a ‘merge’ step. The merge pseudo-steps could be further broken down to look something like the following:

To merge two partitions we iterate over each of them.
Compare elements from both partitions (left and right).
Append smaller element to new ‘results’ (i.e. sorted) list.
After comparing the elements, increment index for winning partition.

The above merge steps process is carried out against each recursively sorted set of partitions, and is done until we reach a final ‘sorted’ list.

Below is an implementation of this sorting algorithm:

def merge(left, right):
    left_index, right_index = 0, 0
    result = []

    while left_index < len(left) and right_index < len(right):
        if left[left_index] < right[right_index]:
            result.append(left[left_index])
            left_index += 1
        else:
            result.append(right[right_index])
            right_index += 1

    result += left[left_index:]
    result += right[right_index:]

    return result


def merge_sort(collection):
    if len(collection) <= 1:
        print('collection is <= 1\n')
        return collection

    middle = len(collection) // 2
    left = collection[:middle]
    right = collection[middle:]

    left = merge_sort(left)
    right = merge_sort(right)

    return merge(left, right)

collection = [10, 5, 2, 3, 7, 0, 9, 12]

result = merge_sort(collection)

print(f'merge sort of {collection} result: {result}\n\n')

Quick Sort

Given an unsorted list of integers, the merge sort algorithm enables us to sort the list.

Best: Ω(n log(n))
Average: O(n log(n))
Worst: O(n^2)

The algorithm can be broken down into the following pseudo-steps:

Pick a ‘pivot’ (e.g. random index from the list).
Iterate the collection twice, creating two new lists.
Create a list consisting of elements less than pivot.
Create a list consisting of elements greater than pivot.
function return value is fn(less) + pivot + fn(greater).

ℹ️ NOTE

notice the ‘less’ and ‘greater’ lists are passed recursively back through the quick sort function.

Below is an implementation of this sorting algorithm:

from random import randrange

collection = [10, 5, 2, 3, 7, 0, 9, 12]

def quicksort(collection):
    if len(collection) < 2:
        return collection
    else:
        random = randrange(0, len(collection))
        pivot = collection.pop(random)

        less = [i for i in collection if i <= pivot]
        greater = [i for i in collection if i > pivot]

        return quicksort(less) + [pivot] + quicksort(greater)

cloned_collection = collection.copy()  # avoid mutation

result = quicksort(cloned_collection)

print(f'quick sort of {collection} result: {result}\n\n')

Difference between Merge and Quick sort

Both merge sort and quick sort are ‘divide and conquer’ algorithms, e.g. they divide a problem up into two and then process the individual partitions, and will keep dividing up the problem for as a long as it can.

So when considering which you should use (i.e. which is better?) you’ll find merge sort is more performant because its particular implementation is more efficient with the operations it carries out.

Specifically, the difference between merge sort and quick sort is that with quick sort you’ll loop over the collection twice in order to create two partitions, but remember that doing so doesn’t mean the partitions are sorted.

These quick sort partitions are not sorted until the recursive calls end up with a small enough partition (length of 1) where the left and right partitions can be analyzed, joined and therefore considered ‘sorted’.

Compare that to merge sort which recursively creates multiple partitions (all the way down to the smallest possible partition) before it then recursively rolls back up the execution stack and attempts to sort each of the partitions along the way.

Additionally, with a merge sort it’s possible to parallelize the data over multiple processes, while quick sort requires data to be processed within a single process.

This means that quick sort has potentially more operations to be carried out than merge sort, and therefore has a greater time complexity. Although quick sort can offer better ‘space’ complexity (worst case: O(log(n))) compared to merge sort (worst case: O(n)).

All that said, the implementation of the algorithms can be tweaked as needed to produce better or worst performance. For example, quick sort could be modified to use another algorithm called intro sort which is a mix of quick sort, insertion sort, and heapsort, that’s worst-case O(n log(n)) but retains the speed of quick sort in most cases.

Radix Search?

Another searching algorithm that I see crop up in discussions every now and then is ‘radix search’. The way it works is to loop over your data structure twice and (if we’re dealing with integers) for the first iteration you’ll ‘bucket’ the elements by their 1’s digit, followed by another iteration to bucket the elements by their 10’s digit.

So with a collection like 25, 17, 85, 94, 32, 79, after the first iteration we would have created ‘numbered’ buckets that looked something like the following:

1: <empty>
2: 32
3: <empty>
4: 94
5: 25, 85
6: <empty>
7: 17
8: <empty>
9: 79

If we remove the empty buckets it means we’ll end up with a partially sorted list of 32, 94, 25, 85, 17, 79. Now for the second iteration we re-bucket by the 10’s, so this means we end up with:

1: 17
2: 25
3: 32
4: <empty>
5: <empty>
6: <empty>
7: 79
8: 85
9: 94

Again if we remove the empty buckets we’ll find we now have a fully sorted list: 17, 25, 32, 79, 85, 94.

It’s an interesting algorithm but ultimately is quite limited and so other sorting algorithms like merge/quick sort are generally preferred. Some constraints to be aware of are is that it’s generally less efficient than other comparison sorting algorithms.

Radix sort is also targeted at integers, fixed size strings, floating points and <, > or lexicographic order comparison predicates.

Binary Search

The most popular algorithm (by far) for searching a value in a sorted list is ‘binary search’. The reason for its popularity is that it provides logarithmic performance (on average) for access, search, insertion and deletion operations.

Average: O(log(n))
Worst: O(n)

The algorithm can be broken down into the following pseudo-steps:

define ‘start’ and ‘end’ positions (usually length of list).
locate middle of list.
stop searching if correct value found.
if value is greater: change end position to the middle index.
if value is smaller: change start position to the middle index.
repeat above steps until value is found.

What this ultimately achieves is shortening the search ‘window’ of items by half each time (that’s the logarithmic part). So if you have a list of 1000 elements, then we can say it’ll take a maximum of ten operations to find the number you’re looking for (that’s: log 2(10) == 2^10 == 1024).

That’s outstanding.

Below is an implementation of this popular search algorithm:

def binary_search(collection, item):
    start = 0
    stop = len(collection) - 1

    while start <= stop:
        middle = round((start + stop) / 2)
        guess = collection[middle]

        if guess == item:
            return middle
        if guess > item:
            stop = middle - 1
        else:
            start = middle + 1

    return None

collection = [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 20, 21]

result = binary_search(collection, 9)

print(f'the value was found at index {result}\n\n')  # index 4

ℹ️ NOTE

you could swap round((start + stop) / 2) for (start + stop) // 2 which uses Python’s // floor division operator, but I typically opt for the clarity of using the explicit round function.

Breadth First Search

A BFS (breadth first search) is an algorithm that searches a data structure from either a root node or some arbitrary starting point. It does this by exploring all the neighbouring nodes, before moving onto other connected nodes.

The following graph represents a group of people. Some people know each other (e.g. both Alice and Bob know each other), where as other people don’t (e.g. Dave knows Ethan, but Ethan knows no one else in this group).

We’ll use the BFS (breadth first search) algorithm to locate ‘Ethan’.

The time complexity of this algorithm will be, at worst, O(V+E).

ℹ️ NOTE

in the case of dealing with a graph, V = vertex (a node in the graph) and E = edge (the line between nodes), the worst case scenario will mean we have to explore every edge and node.

The algorithm can be broken down into the following pseudo-steps:

Pick a starting point in the data structure.
Track nodes to process (e.g. a queue).
While the queue has content:
- Take a node from the queue.
- If the node has been searched already, then skip it.
- If not searched, check if it’s a match, otherwise update queue.
- Queue should be updated with that node’s adjacent nodes.
- If match is found, then exit the queue loop.

Below is our example implementation of the BFS algorithm:

import random
from collections import deque

graph = {'alice': ['bob', 'charlie', 'dave'],
         'bob': ['alice', 'charlie'],
         'charlie': ['alice', 'bob'],
         'dave': ['alice', 'ethan']}

def search(starting_point, name):
    print(f'starting point: {starting_point}')

    queue = deque()
    queue += graph[starting_point]  # add starting_point's neighbours
    print(f'queue: {queue}')

    searched = []

    while queue:
        person = queue.popleft()
        print(f'person: {person}')

        if person not in searched:
            if person == name:
                print(f'found a match: {person}')
                return True
            else:
                queue += graph[person]  # add this item's neighbours
                print(f'queue updated: {queue}')
                searched.append(person)
        else:
            print(f'skipping {person} as they have already been searched')

    return False

starting_point = random.choice(list(graph.keys()))
search(starting_point, 'ethan')

ℹ️ NOTE

an alternative implementation might use Python’s set data structure to avoid having to filter already searched people.

Let’s take a look at the first run of this program:

starting point: dave
queue: deque(['alice', 'ethan'])
person: alice
queue updated: deque(['ethan', 'bob', 'charlie', 'dave'])
person: ethan
found a match: ethan

From that output we can see that we started very conveniently at the ‘Dave’ node. Dave’s adjacent nodes are Alice and Ethan. Due to the order of the nodes in the queue we attempt to process Alice next. Then we check the next node (Ethan) and find what we’re looking for.

Now consider a second run which has a different starting point (notice the difference in the number of operations):

starting point: charlie
queue: deque(['alice', 'bob'])
person: alice
queue updated: deque(['bob', 'bob', 'charlie', 'dave'])
person: bob
queue updated: deque(['bob', 'charlie', 'dave', 'alice', 'charlie'])
person: bob
person: charlie
queue updated: deque(['dave', 'alice', 'charlie', 'alice', 'bob'])
person: dave
queue updated: deque(['alice', 'charlie', 'alice', 'bob', 'alice', 'ethan'])
person: alice
skipping alice as they have already been searched
person: charlie
skipping charlie as they have already been searched
person: alice
skipping alice as they have already been searched
person: bob
skipping bob as they have already been searched
person: alice
skipping alice as they have already been searched
person: ethan
found a match: ethan

ℹ️ NOTE

I’ve used a dict to represent a graph, which helped to make the code simpler to understand. A different data structure (e.g. a directed tree) would require a different implementation of the algorithm. Remember, the basic premise is to search a graph node’s adjacent fields, and then their adjacent nodes.

Depth First Search

The following image shows a tree structure that represents various people…

A DFS (depth first search) is an algorithm that searches a data structure (such as a tree, or a graph) from its root node. It searches downwards through each child node until there are no more children.

We’ll use the DFS (depth first search) algorithm to locate the node ‘Ethan’.

Using our example tree structure (above) we would start with Alice, then check the first child Bob. Bob has no children so we would move onto Charlie. Charlie has a single child Fred so we would check him next. Fred has no children so we start back up at Dave. Finally, we check the child of Dave which is Ethan.

Notice how if don’t find a match for what we’re looking for, then we backtrack up to the top of the tree and start again at the root node’s next child node.

The time complexity of this algorithm will be, at worst, O(V+E) (as noted with Breath First Search, this means we could end up hitting every single node and edge in the data struture).

ℹ️ NOTE

this example uses a traditional tree data structure instead of a graph to represent the underlying data to be searched.

The algorithm can be broken down into the following pseudo-steps:

Start at the root node.
Check first child to see if it’s a match.
If not a match, check that child’s children.
Keep checking the children until a match is found.
If no children are a match, then start from next highest node.

Below is our example implementation of the DFS algorithm:

class Tree(object):
    def __init__(self, name='root', children=None):
        self.name = name
        self.children = []

        if children is not None:
            for child in children:
                self.add_child(child)

    def __repr__(self):
        return self.name

    def add_child(self, node):
        assert isinstance(node, Tree)
        self.children.append(node)


tree = Tree('Alice', [Tree('Bob'),
                      Tree('Charlie', [Tree('Fred')]),
                      Tree('Dave', [Tree('Ethan')])])


def search_tree(tree, node):
    print(f'current tree: {tree}')

    if tree.name == node:
        print(f'found node: {node} in {tree}')
        return tree

    for child in tree.children:
        print(f'current child: {child}')

        if child.name == node:
            print(f'found node: {node} in {child}')
            return child

        if child.children:
            print(f'attempt searching {child} for {node}')
            match = search_tree(child, node)

            if match:
                print(f'returning the match: {match}')
                return match

result = search_tree(tree, 'Ethan')

print(f'result: {result}')

ℹ️ NOTE

in our example we use a simple tree data structure to represent the data to be searched. Our implementation is designed to work with that structure. So, for example, if we had multiple children per node (left and right child nodes), then we would need to account for the backtrack to the relevant child right node near the top of the tree.

Let’s take a look at the output of this program:

current tree: Alice
current child: Bob
current child: Charlie
attempt searching Charlie for Ethan
current tree: Charlie
current child: Fred
current child: Dave
attempt searching Dave for Ethan
current tree: Dave
current child: Ethan
found node: Ethan in Ethan
returning the match: Ethan
result: Ethan

So from this output we can see we started at Alice, and first checked Bob but because Bob has no children we moved back up to Charlie. From Charlie we go down to Fred but as there’s no more children we move back up to Dave. Finally we check Dave’s children to find Ethan.

When to choose BFS vs DFS?

Generally BFS is better when dealing with relationships across fields, where as DFS is better suited to tree hierarchies.

That said, below is a short list of things to consider when opting for either a BFS or DFS:

ℹ️ NOTE

the data strutures used (including the implementation of your algorithm) can also contribute to your decision.

If you know the result is not far from the root: BFS.
If result(s) are located deep in the structure then: DFS.
If the depth of the structure is very deep, then in some cases: BFS.
If the width of the structure is very wide, then memory consumption could mean: DFS.

Ultimately, it all depends.

Searching an unsorted collection?

If you have an unsorted collection, then your only option for searching is a linear time complexity O(n). To improve that performance we would need to first sort the collection so we could use a binary search on it.

Another potential option is to run a linear search across multiple CPU cores so you’re effectively parallelizing the processing. It’s still O(n) linear time complexity, but the perceived time would be shorter (depending on the restructuring of the collection split chunks).

Dijkstra’s Algorithm

The Dijkstra algorithm tells you the quickest path from A to B within a weighted graph.

In the above graph we have a few options:

Start > A > End (Cost: 7)
Start > B > End (Cost: 7)
Start > B > A > End (Cost: 6)

ℹ️ NOTE

as you can see, the route that visually looks longer is actually quicker when considering the weighted nature of the graph.

The time complexity of this algorithm will be, at worst, O(V+E) (as noted with Breath First Search and Depth First Search, this means we could end up hitting every single node and edge in the data struture).

The algorithm can be broken down into the following pseudo-steps (it’s important to note that in this implementation we calculate the route in reverse):

Identify the lowest cost node in our graph.
Acquire the adjacent nodes.
Update costs for each node while accounting for surrounding nodes.
Track the processed nodes.
Check for new lowest cost node.

These steps are very specific to our graph, so if your data structure is different, then the implementation of the algorithm will need to change to reflect those differences. Regardless this should be a nice introduction to the fundamental properties of the algorithm.

Without further ado, below is an example implementation of the Dijkstra’s algorithm:

graph = {
    'start': {
        'a': 6,
        'b': 2
    },
    'a': {
        'end': 1
    },
    'b': {
        'a': 3,
        'end': 5
    },
    'end': {}
}

costs = {
    'a': 6,
    'b': 2,
    'end': float('inf')  # set to infinity until we know the cost to reach
}

parents = {
    'a': 'start',
    'b': 'start',
    'end': None  # doesn't have one yet until we choose either 'a' or 'b'
}

processed = []
route = []

def find_lowest_cost_node(costs):
    lowest_cost = float('inf')
    lowest_cost_node = None

    for node in costs:
        cost = costs[node]

        if cost < lowest_cost and node not in processed:
            lowest_cost = cost
            lowest_cost_node = node

    return lowest_cost_node

def find_fastest_path():
    node = find_lowest_cost_node(costs)

    while node is not None:
        cost = costs[node]
        neighbours = graph[node]

        for n in neighbours.keys():
            new_cost = cost + neighbours[n]

            if costs[n] > new_cost:
                costs[n] = new_cost
                parents[n] = node
        processed.append(node)
        node = find_lowest_cost_node(costs)

def display_route(node=None):
    if not node:
        route.append('end')
        display_route(parents['end'])
    elif node == 'start':
        route.append(node)
        reverse_route = list(reversed(route))
        print('Fastest Route: ' + ' -> '.join(reverse_route))
    else:
        route.append(node)
        display_route(parents[node])

find_fastest_path()  # mutates global 'costs' & 'parents' arrays
display_route()  # Fastest Route: start -> b -> a -> end

The output of this program is (as expected):
Fastest Route: start -> b -> a -> end.

Remote Working

Sat, 09 Mar 2019 00:00:00 +0000

Introduction

I was asked on twitter recently about how I am able to be successful and impactful at an organization as a remote worker. What follows is a run down of how I work remotely, and how it has changed over the last seven years.

This is not a “rules for how you should do remote working” but more so a single perspective on the problem of remote working and my own simplified set of guidelines that I feel have worked for me.

Why Remote?

Let’s start off by talking about why I am a remote worker. I typically work for organizations who have an office in Central London, while my home is out on the east coast of England.

My commute into work in the morning is just over two hours, and my commute home is either the same or worse depending on the state of the trains and the nightmare that is the home time commuting hours of 4pm-7pm. That’s almost five hours a day sitting (or more likely standing) on a train.

Also, because of my locality I have to be up before 6am in order to get to work by a reasonable hour. It also means, to avoid the commuter rush in the evening (including trying to acquire a seat for that long trip home), I typically have to leave the office by approximately 4.30pm latest.

I don’t know about you but I consider that…

stressful
tiring
inefficient

Before I had a family, that would have been time back to myself to do things such as: having a more relaxed breakfast, having more time to get ready for the day ahead and maybe even going to the gym before work.

Now that I have a family, that’s critical time I get to spend with them before I start work, as well as to do some house chores and help my wife with getting our son ready in the morning.

If you don’t have children, then it’s probably best that I explicitly state what otherwise might be considered ‘stating the obvious’ (and apologies if so), but just in case: getting children ready in the morning is a lot harder, time consuming and stressful than you’ll probably realise. So it’s important I help my wife with this otherwise it’s a massive burden on her shoulders.

ℹ️ NOTE

this can often be something as simple (but meaningful) as looking after our son for a mere ten minutes just so she can get washed and dressed!

Not to mention things like getting deliveries to your home, or getting a plumber round to fix your sink, or even time to take your kids to the doctors.

All of these things that people have to deal with in their day-to-day lives can be made easier if we were able to work from home.

Problems with Remote Working

OK, so not everything is roses. Here is a short list of things that can be a concern when working from home (and I’ll dig into each afterwards):

When do I start/finish?
How do I get noticed/feel included?
How can I be impactful?
How should I communicate?
How do I prove I’m getting stuff done?
How do I avoid procrastinating?
How can I stay healthy?

ℹ️ NOTE

this is a small list from a much larger set of questions people often have about remote working, but hopefully these are enough items for me to discuss to help get across my thoughts.

When do I start/finish?

This is important. You need a routine. Doesn’t matter what it is, but you need one. If you think remote working is waking up whenever you like every day and wandering around the house in your pyjamas and whimsically selecting the code tasks you’re going to hack on, then you’re in for a rude awakening.

Remote working is WORK. So you get up at a consistent time in the morning, you get washed and dressed, you have breakfast and a coffee/tea, and then you think about the day ahead. Start WORK.

This is likely going to sound no different to maybe how you’re working at the moment as a ‘non-remote’ worker, right! Exactly.

I’m up at 7am and I start work by 9am (sometimes 10am if there’s something else I need to do in the morning). I do a full day’s work: so if I start at 9am I’ll be finished by 5pm. If I’m starting at 10am I’ll work till 6pm.

Sometimes you’ll find a natural wrap up of your day is earlier than when you should ‘officially’ finish. For example, you’re planning to finish at 5pm but it’s 4:30 and you don’t want to pick up another large task today.

In those instances, I would normally just jump over to read through any late email I’ve received. I’d then jump onto Slack and check-in with my team, most likely I’ll be dropping an ‘end of day summary’ (which can be very helpful for my team mates who are based in different timezones to me).

How do I get noticed/feel included?

This is actually a subtle question with many different layers to it. In most companies I work for I start as a ‘partial’ remote worker (so maybe two days a week in the office). That can be really helpful as far as building up relationships with my colleagues. You need good relationships.

But if you’re starting off as a completely 100% remote worker, don’t worry, building relationships is easy enough to do. It just takes a bit of effort on your part (which I’ll explain next). I can say it’s easy to do because I have built up many great working relationships with my colleagues based in the United States (both in the eastern and western timezones).

ℹ️ NOTE

although reaching the western timezone is tricker than eastern as it’s a solid eight hours behind. It’s possible to do still, but again will take just a bit more effort if that’s what you want.

Ultimately it comes down to communication (and good communication at that). For communication to be considered ‘good’ it needs to happen at the right time and in the right medium (see How should I communicate? for more details on this subject).

You’ll find that I will, every morning (hopefully around the same time) drop a message into Slack (for the UK engineering team) saying something along the lines of…

Morning all 👋

<funny gif>

_^^ some explanation of the gif_

I typically put an explanation below the gif if it is based on the current weather outside, or maybe there’s a big event happening, or maybe it’s just a funny caption to go with the gif, whatever.

The reason I do this is because that’s me. That’s the type of person I am and I want other people to recognise that this is my personality coming through. I’m giving a bit of myself to these people. I also then do the same thing for my team (who currently are all based outside of the UK) in our own team Slack channel.

Of course I don’t just do that 😉 I’m also very open about my personal life and what I’m doing. For example, I’ll often post pictures of my daily lunch time walks with my son.

Again, the reason I do this isn’t to gloat (although sometimes it is when the weather is sunny) but a nice coastal scenery shot with me and my son in it, reminds people that I’m a human being and not some anonymous entity behind a keyboard and screen.

You should try to learn more about your colleagues, and appreciate what sorts of things they’re into. Maybe they love photography or are into games or books. Maybe they like riding their bike on the weekend or going rock climbing. Having freeform non-work related conversations is great for this (so find a space to have those conversations).

So for example I know a colleague of mine who works in Argentina loves cats and has two of her own. I also have cats and we chat regularly in DM’s about not only work, but we share pictures of our cats. Simple things like that help build stronger relationships and makes working together more meaningful.

We also follow each other on Instagram, and I have many work friends (current and old) who follow me (and vice-versa) on twitter and facebook. We spend a large majority of our life at work, so making friends and really caring about who you work with will make a massive positive impact on your enjoyment while at work.

Don’t limit yourself to reaching out to just your team, or to people in your locality. Make an effort to take part in various channels of communication (including those that might be happening across different timezones). This might be as simple as joining various Slack channels where people talk about certain languages or tech, but it can include joining various ‘working groups’.

I’ve been part of, and led, many working groups (architecture, code design, documentation, communication). You can spin up a working group about anything you’re interested in and by sharing it with the organization you’ll be surprised to discovered that there are likely lots of like-minded individuals interested in the same things and willing to discuss and to want to help improve the quality of the ‘thing’ happening within the organization.

Once you start doing the groundwork of understanding your colleagues and proactively sharing a piece of your true self at work, you’ll also find that people will start reaching out to you, for both your opinion and to include you in conversations (this could be because they recognize you have specific skills that will benefit their project or maybe it’s something more informal – someone cracked a joke and cc’s you as part of it).

You have (for lack of a better word) ‘exposed’ yourself to the organization.

How should I communicate?

Communication takes many forms and is one of the biggest contributing factors to misunderstanding requirements, and to personal unhappiness. It’s one of the most critical skills to learn and to utilize (regardless of whether you’re a remote worker or not).

For example, don’t just Slack people all the time. You should notice when conversations are getting out of hand (i.e. too much back and forth overlapping textual messages with no one really reading the other person’s message, you’re both firing off in the hope that your point is heard first) and offer instead to jump over to a ‘face-to-face’ video call to clarify.

With face-to-face people generally are more easily able to listen because the convention in our society isn’t to shout interweavingly across each other. Instead, someone talks and (if you’re doing what you’re supposed to be) you’ll be listening and really taking in what the other person is saying. You’ll then adapt your response based on what the other person has just said †

† this is much different to what a lot of people mistakenly do which is: stay quiet until the other person has stopped talking and then reply with their pre-canned response (i.e. again they’re only interested in getting across their own points).

One thing that I like to do after taking a long Slack conversation into a face-to-face video call is to go back to Slack and to summarize what was just disussed. There are a few reasons for doing this:

it’s a documented record of what was discussed.
it confirms that everyone has understood the outcome correctly.
it shows you’re conscientious of people in other timezones (who might have been following the conversation).

Another important characteristic of good communication: you should also aim to be ‘clear’ and, if the situation calls for it, concise (this is something I’m guilty of not being good at, but I’m aware of that problem and try hard to be better at).

In my blog posts I’m definitley not concise, but then I’m usually ‘telling a story’ and that’s a different situation to writing an email or a Slack message (but even still I’ve a long way to go to improve – no one’s perfect).

For those of you who are interested I would highly recommend reading the following material:

A couple of other important notes to make:

Make sure you block out time in your calendar for important events, such as a lunch hour or family time etc. This becomes especially useful when working within a distributed company, for example, in my calendar I block out 5pm to 10pm for ‘family time’ so people in New York and LA can’t accidentally try to book a meeting with me.

Schedule monthly 1:1 meetings with people of differing levels within the company, such as interns, seniors, managers, designers etc. This is useful because it gives you a regular ‘check-in’ with what’s happening elsewhere in the organisation and it also helps others to get to know you.

How can I be impactful?

To have ‘impact’ can take many different forms. Being impactful isn’t just about getting a job done. It could be mentoring colleagues, or it could be noticing a problem space (either in the tools your organization uses or the services it provides) and producing an RFC documentation that tackles some possible solutions.

I think ultimately to be impactful requires you to care. You need to care about the organization and its success. You need to care about your colleagues and their success.

If you’re just turning up at work to ‘go through the motions’ and to just ‘get paid’, then you’ll likely always fall short of having true life affirming impact.

How do I prove I’m getting stuff done?

This is a classic concern for both people wanting to be a remote worker, as well as organizations who are unsure remote working is something they want to offer to their workers.

Any half decent organization is going to have ‘process’ and that process should be effective regardless of whether you’re remote or not. You should have some form of project/task tracker (e.g. Jira), you should have some form of methodology of working (e.g. Kanban, Scrum, Agile etc).

With this process you should have enough to demonstrate you’re ‘getting things done’. For example…

Daily team standups: this is your chance to say what you worked on yesterday, what you worked on today, and whether anything is blocking you. This is a great time to express things concerning you and for your manager/team to get an insight into that.
Task tracking: if you use Jira then you’ll have a set of tasks on a backlog that are prioritized and are assigned to specific individuals. Management can track and review what you’re doing and see if things aren’t getting done.
One to one meetings: you have an opportunity at 1-1 meetings with your line manager to express your concerns, and it can be a good time to highlight things you’re doing (not project/task specific as that’s not the point of 1-1’s, but maybe you’ve created a working group or are working on an important RFC).

How do I avoid procrastinating?

I personally feel that people are capable of procrastinating regardless of whether they’re remote or not. To avoid procrastinating means being focused, getting rid of distractions, settings yourself deadlines and generally being organized.

None of these are traits that should be limited to remote workers, as this is just what it means to be an attentive and considerate worker.

I would say that if you’re going to be a remote worker though, you’ll want to have yourself a space where you can work. Yes it’s nice to be able to work from any where in your home, but to have a dedicated ‘office space’ is important.

I have an office space which I use the majority of the time because it’s quiet and it’s away from the family noise. But I also find that when it’s sunny I like to work from the lounge in the morning (because the sun hits that room first, and also means I can be around my son while he’s enjoying his morning playtime).

I might then move to working from the kitchen so I can look out at my garden and again enjoy some sunshine and scenery as the sun moves around during the day. But again, I do mainly work from my office space, especially when I have meetings or I feel I really need to concentrate.

If you listen to music while working then that can actually be a cause of distraction. For me, if I need to focus (or am working on something I’m unsure about) I tend to find music with lyrics generally distracts me so I’ll listen to genres like jazz, classical or downbeat instrumental (trip hop’ish stuff like DJ Crush’s “Code 4109”). But if I’m working on a system I know very well and I’m just churning out stuff like a batch of unit tests, then I can listen to anything.

Ultimately you need to get used to recognizing when you’re distracted and to combat that however it makes sense for you.

How can I stay healthy?

This is a very important topic as remote workers are notorious from suffering burnout from over working. You need to have clear boundaries, so you need to say: I start at N time in the morning and I’ll finish at N time in the evening. I’ll also have lunch at N time during the day (and have a full lunch).

For me personally I have a desk that transitions from a sitting desk to a standing desk. I find in the morning I like to stand for an hour or two and then I’ll sit for an hour or two, and then I’ll go for a walk along the seafront (sometimes with my son or sometimes by myself if I need some space) during my lunch hour.

I make sure that my lunch is something I cook. I avoid ‘ready meals’ or snacky foods like crisps and chocolates (I generally only ever have a nibble of dark chocolate in the evenings if I have a sugar craving).

As an example, a general day of eating will look something like…

Breakfast: bowl of mixed nuts, raisins, blackberries, banana, pumpkin seeds and oat milk.
Lunch: salmon and salad.
Dinner: meat, veg and a bit of carb.

I also don’t drink coffee or ‘normal’ tea. I’m primarily a drinker of water, peppermint tea, and if I need caffeine I’ll drink a pot of Guayusa green tea (which has more caffeine than coffee but is much healthier for you).

ℹ️ NOTE

if you’re able to, try and do some exercise as well! A sedentary lifestyle is not good for us. If you can’t get to a gym or aren’t motivated to do exercise then don’t kick yourself about it. But do get up and out of your chair for a walk every once in a while. Fresh air really does make a world of difference.

The key thing is to be ‘active’ and to have space to think about other things that aren’t work related as this will actually help your subconscious to really kick-in behind the scenes and process conversations and other things going on at work.

It’s also important to realize that you’re remote so you don’t have to work from ‘home’. You can work anywhere there’s an internet connection (if your work provides you with a mobile wifi then that’s even better because you can just go to the beach and work from there if you like).

I’ve never done it but I could imagine going to my local library and working from there to be quite nice. Maybe going to a bustling coffee shop with free wifi? Either way, get out and away from your home as often as you can so you don’t start building up a form of ‘cabin fever’.

What doesn’t work well?

OK, there are some things that still don’t work that great when being a remote. Below is a small selection:

Whiteboard discussions: architecture design meetings using a whiteboard can be difficult. there is technology to help with this but it’s not great. One thing we’re looking to explore is for all employees (remote and in-office) to have access to a drawing tablet for their computer so they can draw with an electronic pen into some shared screen software. This would be much more flexible and easier to work with than trying to draw with a mouse.
Remote meetings: specifically I’m referring to people who aren’t familiar with doing remote meetings and so they’ll do silly things like treating the remotes as if they aren’t there. So they’ll sit with their back to the camera, or they’ll ask for opinions from people in the room but not take the time to say: “for those here who are attending remotely, does anyone have anything to add?” (because jumping into a conversation remotely can be much harder than if you’re in a physical office/room).
Unplanned conversations: sometimes conversations are sparked in the office between colleagues. They don’t always happen on Slack, and so it’s important that those people in the office are thinking about their remote colleagues and include them when necessary.

That said, there’s not much that’s so problematic that remote working couldn’t be a consideration for most organizations.

Conclusion

Hopefully you’ve found this rundown of remote working (from someone who has been remote working for many many years now with great success and much learnings under his belt). If you have any additional questions then do please reach out to me on twitter.

Mocking in Python

Fri, 08 Mar 2019 00:00:00 +0000

Introduction

Mocking resources when writing tests in Python can be confusing if you’re unfamiliar with doing such things. In this post I am going to cover various aspects of mocking code, which will hopefully be a useful resource for those who are a bit stuck.

ℹ️ NOTE

in the code examples I’m using pytest, but for the most part that shouldn’t matter.

unittest.mock or mock

In order to ‘mock’ a resource we’ll first need the mock module, and this is our first stumbling block: which version do we need? i.e. there’s two and they both look to be official (mock and unittest.mock).

The mock module is a backwards compatible library you can download from PyPy, where as unittest.mock is the same thing but only compatible with the version of Python you’re using.

So in almost all cases you’ll want to import it like so:

import unittest.mock as mock

For more examples, see this reference guide

Decorator

The most common way to mock resources is to use a Python decorator around your test function:

@mock.patch("thing")
def test_stuff(mock_thing):
    mock_thing.return_value = 123

In this case, what we’re patching (thing) can be a variable or a function.

When you do this you’ll need to pass an argument to your function (you can name it whatever you want †) which will be a MagicMock.

This means if you don’t do anything else, then calls to thing will (in the example above at least) result in the value 123 being returned.

† convention is to name the variable mock_<noun>.

If you’re mocking multiple things then you’ll stack the mock decorators ontop of each other, and pass them along in order to the test function:

@mock.patch("third")
@mock.patch("second")
@mock.patch("first")
def test_stuff(mock_first, mock_second, mock_third):
    ...

Resource location

It’s important to know that when mocking you should specify the location of the resource to be mocked, relevant to where it’s imported. This is best explained by way of example…

Imagine I have a module app.foo and within that module I import another dependency like so:

from app.bar import thing

You might think that when you call mock.patch that you pass it a reference to the resource like app.bar.thing. That would only be relevant if the resource was being called with that full path within the app.foo module (e.g. if app.foo called app.bar.thing(...)).

If the full namespace path isn’t referenced, which it isn’t in the above example (notice we import just the thing resource). It means we need to specify the reference namespace to mock as where it’s imported:

@mock.patch('app.foo.thing')

So even though thing exists within app.bar we specify app.foo.thing as app.foo is where we’ve imported it for use. This catches people out all the time.

Mock return_value vs side_effect

If your function has a try/except around it, then you can use side_effect to cause the calling of the function to trigger an Exception as the returned value:

@mock.patch('app.aws.sdk.confirm_sign_up', side_effect=Exception('whoops'))

ℹ️ NOTE

if you had used return_value=Exception('whoops') then the mock would return the string representation of the Exception rather than raising an exception like side_effect does.

Otherwise if you just need a static value returned, so it’s evaluated at the time it’s defined (not when it’s called), then you can use return_value instead:

@mock.patch('app.security.secret_hash', return_value='###')

Mock Nested Calls

Calling a property on a mock returns another mock, so in order to mock very specific properties you’ll need to nest your return_value or side_effect:

m = mock.MagicMock()
m.return_value.get.side_effect = [1, 2]
m.return_value.post.return_value = 'foo'

x = m()

x.get()   # 1
x.post()  # foo
x.get()   # 2

Things can get a little more confusing when you want to verify a specific nested method on a mocked object was called:

import unittest.mock as mock

from tornado.ioloop import IOLoop

@mock.patch("__main__.IOLoop")
def foo(mock_ioloop):
    IOLoop.current()
    mock_ioloop.current.assert_called()  # will fail if assertion isn't True

    IOLoop.current().start()
    mock_ioloop.current().start.assert_called()  # will fail if assertion isn't True

The reason this can get more complicated is due to how a mock will return a new mock when accessing a property on a mock:

import unittest.mock as mock

from tornado.httpserver import HTTPServer

@mock.patch("__main__.HTTPServer")
def bar(mock_httpserver):
    server = HTTPServer()

    server.listen(8080)
    HTTPServer().listen(123)

    mock_httpserver.assert_called()
    mock_httpserver().listen.assert_called_with(8080)

The above code will error:

AssertionError: expected call not found.
Expected: listen(8080)
Actual: listen(123)

You’ll need to make sure you assert the mock at the right time:

import unittest.mock as mock

from tornado.httpserver import HTTPServer

@mock.patch("__main__.HTTPServer")
def bar(mock_httpserver):
    server = HTTPServer()
    mock_httpserver.assert_called()
    
    server.listen(8080)
    mock_httpserver().listen.assert_called_with(8080)
    
    HTTPServer().listen(123)
    mock_httpserver().listen.assert_called_with(123)

Verify Exceptions

If we want to verify that some piece of code throws an Exception type when we need it to we can mock specific resources to throw an exception and then use pytest.raises as a context manager around the caller of our code to be verified.

In the following example our code (in app.account.confirm(...)) catches a generic Exception and re-raises it as exceptions.CognitoException.

We can catch and make assertions against this expected behaviour by first mocking the resource we want to throw an exception and get it to throw our own fake exception using the side_effect parameter.

Next we specify the exact exception type we’re expecting to be raised using pytest.raises(T):

@mock.patch('app.aws.sdk.confirm_sign_up', side_effect=Exception('whoops'))
def test_account_confirm_failure(mock_signup):
    with pytest.raises(exceptions.CognitoException) as exc_info:
        app.account.confirm(123, 'foo')
        assert True is True  # this will never be executed!
        
    assert exc_info.typename == 'CognitoException'
    assert str(exc_info.value) == 'SIGNUP_CONFIRMATION_FAILED'

ℹ️ NOTE

don’t make the mistake of putting any assertions within the with context manager. Once the Exception is raised by the function being called within the with context manager, all code after it inside the block is skipped.

Clearing lru_cache

If a function you wish to test has the functools.lru_cache decorator applied, then you’ll need to be mindful of mocking the response of that function as it’ll be cached in one test and the cached result will be returned when calling the function again to test some other behaviour (and might likely confuse you when you see the unexpected response).

To fix this issue is very easy because lru_cache provides additional functions when decoratoring your functions, it provides:

cache_info
cache_clear

The latter (cache_clear) is what you would need to call. This is demonstrated below:

@lru_cache(5)
def foo():
    print('Executing foo...')
    
foo()  # Executing foo...
foo()  # <nothing printed as None response was cached and returned>
foo.cache_info()  # CacheInfo(hits=1, misses=1, maxsize=5, currsize=1)
foo.cache_clear()
foo()  # Executing foo... (notice the 'side effect of print is executed again)

ℹ️ NOTE

debugging this isn’t always obvious. Later on I demonstrate how to mock the builtin open function, and in that scenario I stumbled across this issue, because although I wasn’t mocking the top level function itself (I was mocking the call to open within), the contents of the file being opened was what was returned and being cached.

Mock Module Level/Global Variables

With a module variable you can can either set the value directly or use mock.patch.

In the following example we have the variable client_id which is a global variable inside the app.aws module which we import to reference elsewhere in our code:

import app.aws


def test_account_confirm_successful():
    app.aws.client_id = 456  # used internally by `confirm()`
    ...
    
@mock.patch('app.aws.client_id', 456)
def test_account_confirm_successful():
    ...

In the mock.patch example, there are two key things to notice:

we don’t use return_value.
there is no mock instance passed to the test function.

This is because we’re modifying a variable and not a direct function or ‘callable’ so there’s no need to pass a mock into the test function (if you want to change the value a few times within the test itself then you would mock the variable but not immediately assign a value in the decorator).

Mock Instance Method

There are multiple ways to achieve mocking of an instance method. One common approach is to use mock.patch.object, like so:

from unittest import mock

def test_foo():
    with mock.patch.object(FooClass, 'method_of_class', return_value=None) as mock_method:
        instance = SomeClass()
        instance.method_of_class('arg')
        mock_method.assert_called_with('arg')

Another approach is to mock the method like you would a normal function, but you reference the method via the classname:

def test_bar():
    r = mock.Mock()
    r.content = b'{"success": true}'

    with mock.patch('requests.get', return_value=r) as get:  # Avoid doing actual GET request
        some_function()  # internally calls requests.get
        get.assert_called_once()

Another (although more heavy handed) approach for mocking a class instance method is to take advantage of the fact that a Mock will return a new mock instance when called:

@mock.patch("foo.bar.SomeClass")
def test_stuff(mock_class):
    mock_class.return_value.made_up_function.return_value = "123"

ℹ️ NOTE

in the above example we mock the entire class, which might not be what you want. If not, then use the previous mock.patch.object example instead.

The reason the above example works is because we’re setting return_value on our mock. Because this is a MagicMock every attribute referenced returns a new mock instance (a function or property you call on a mock doesn’t have to exist) and so we call made_up_function on the returned mock, and on that newly created mock we set the final return_value to 123.

But as mentioned in the above note, this approach might be a little too blunt depending on what your needs are (whether you care whether you have a some what functioning class or not).

Mock Class Method

To mock a class method is a similar approach to mocking an instance method.

One approach might be that you mock the entire class (but now you have one less return_value to assign to):

mock_class.ClassMethodName.return_value = "123"

Or better yet you should mock it like you would any normal function, but just reference the method via the class:

@mock.patch('myapp.Foo.class_method_name')
def test_classmethod(self, mock_class_method):
    mock_class_method.return_value = "foobar"

Mock Entire Class

To mock an entire class you’ll need to set the return_value to be a new instance of the class.

@mock.patch('myapp.app.Car')
def test_class(self, mock_car):

    class NewCar(object):
        def get_make(self):
            return 'Audi'

        @property
        def wheels(self):
            return 6

    mock_car.return_value = NewCar()
    ...

See other class related mocking tips here

Mock Async Calls

Mocking asynchronous code is probably the most confusing aspect of mocking. My ‘go to’ solution I’ll explain first, but after that I’ll share some alternative methods I’ve seen and tried in the past.

First consider this asynchronous code inside of a app.foo module:

import app.stuff

async def do_thing(x):
  return await app.stuff.some_concurrent_function(x)

If we need to mock the coroutine app.stuff.some_concurrent_function, then we can solve this by creating a function that acts as a coroutine and allow it to be configurable for different types of responses:

ℹ️ NOTE

the example uses tornado for running an asynchronous test.

def make_coroutine(response):
    """You could pass response as a mock or a raw data structure, doesn't matter."""
    
    async def coroutine(*args, **kwargs):
        """*args will be whatever is passed to the original async function.
        
        Meaning you could have a conditional check that let's us change 
        the response to be anything we need.
        """

        return response

    return coroutine

class TestThing(tornado.testing.AsyncTestCase):
    @mock.patch('app.stuff.some_concurrent_function')
    @tornado.testing.gen_test
    def test_async_func(self, mock_thing):
        mock_thing.side_effect = make_coroutine('some response')
        result = yield app.foo.do_thing('xyz')
        assert result == 'some response'

If you do include an if statement within make_coroutine, you could pass in a MagicMock as a simple way of having a single input give you multiple different values back…

def make_coroutine(response):
    async def coroutine(*args, **kwargs):
        if args[0] == 'x':
            return response.x
        elif args[0] == 'y':
            return response.y
        else:
            return response.default

    return coroutine

m = mock.MagicMock(x=1, y=2, default=3)
coro = make_coroutine(m)

When dealing with side_effects that need to sometimes trigger an Exception and other times suceed you could use a slightly modified mock implementation that checks if the given response object is callable or not…

count = 0

def make_side_effect_coroutine(side_effect):
    """Side effect friendly mock coroutine.

    In some tests we need to have a mocked coroutine return a different value
    when it's called multiple times, but a mock side_effect can't trigger a
    raised exception when given an iterator, and so we have to construct that
    behaviour ourselves.
    """

    async def coroutine(*args, **kwargs):
        return side_effect(*args, **kwargs) if callable(side_effect) else side_effect
    return coroutine
    
@mock.patch('app.thing')
def test_confirm_email_change_failure(self, mock_thing):

    def side_effects(*args, **kwargs):
        """Use global var to control mock side effects."""

        global count

        if count > 0:
            raise Exception('whoops')

        count += 1
        return  # don't raise an exception the first time around

    mock_thing.side_effect = make_side_effect_coroutine(side_effects)

If the above approach doesn’t work for you, here are some alternatives…

AsyncMock

ℹ️ NOTE

this utilizes the package pytest-asyncio to help with testing asyncio code

Let’s start with the code to be mocked…

import asyncio

async def sum(x, y):
    await asyncio.sleep(1)
    return x + y

Now here’s how we’d mock it…

import pytest
import asyncio

# create a new pytest fixture called mock_sum
#
@pytest.fixture()
def mock_sum(mocker):
    async_mock = AsyncMock()
    mocker.patch('app.sum', side_effect=async_mock)
    return async_mock

    # Python <3.8 would have used
    #
    # future = asyncio.Future()
    # mocker.patch('app.sum', return_value=future)
    # return future

@pytest.mark.asyncio
async def test_sum(mock_sum):
    mock_sum.return_value = 4

    # Python <3.8 would have used
    #
    # mock_sum.set_result(4)

    result = await sum(1, 2)
    assert result == 4

Monkey Patch

# allow mock to be used as an await expression...

async def async_response():
    return namedtuple('_', ['body'])('{"state": "success"}')


def mock_async_expression(our_mock):
    return async_response().__await__()


mock.MagicMock.__await__ = mock_async_expression

MagicMock Subclass

class AsyncMock(MagicMock):
    async def __call__(self, *args, **kwargs):
        return super(AsyncMock, self).__call__(*args, **kwargs)
        
class TestHandlers(testing.AsyncTestCase):
    @mock.patch('app.handlers.trigger_soft_cdn_purge', new_callable=AsyncMock)
    @mock.patch('app.handlers.api')
    @testing.gen_test
    async def test_update_cache(self, api_mock, trigger_soft_cdn_purge):
        response = mock.MagicMock()
        response.code = 200
        api_mock.buzz = AsyncMock(return_value=response)

Async Inline Function

@mock.patch('app.buzz_api.api_gateway')
@testing.gen_test
async def test_buzz_api(self, client_mock):
    async def get(url, **kwargs):
        return
        
    client_mock.get.side_effect = get

Mock Instance Types

When mocking an object you’ll find that the mock replaces the entire object and so it can cause tests to pass (or fail) in unexpected ways.

Meaning, if you need to make a mock more like the concrete interface, then there are two ways to do that:

spec
wrap

We can use mock’s spec feature to mimic all methods/attributes of the object being mocked. This ensures your mocks have the same api as the objects they are replacing.

ℹ️ NOTE

there is a stricter spec_set that will raise an AttributeError.

This is best demonstrated with an example:

import unittest.mock as mock
import tornado.simple_httpclient

from tornado.httpclient import AsyncHTTPClient


http_client = AsyncHTTPClient()
type(http_client)  # tornado.simple_httpclient.SimpleAsyncHTTPClient

isinstance(http_client, tornado.simple_httpclient.SimpleAsyncHTTPClient)  # True

isinstance(mock.MagicMock(), tornado.simple_httpclient.SimpleAsyncHTTPClient)  # False

m = mock.MagicMock(spec=tornado.simple_httpclient.SimpleAsyncHTTPClient)
isinstance(m, tornado.simple_httpclient.SimpleAsyncHTTPClient)  # True

The wrap parameter on the other hand allows you to ‘spy’ on the implementation, as well as affect its behaviour. In the following example I want to spy on the builtin datetime implementation:

@pytest.mark.parametrize("input_date, input_url, valid", [
    ("2017-06-17T00:00:00.000000Z", "foo", True),
    ("2017-06-18T00:00:00.000000Z", "bar", False),
])
@mock.patch("app.handlers.data.datetime", wraps=datetime)
def test_valid_video(mock_datetime, input_date, input_url, valid):
    mock_datetime.now.return_value = datetime(2017, 6, 18, 00, 00, 00, 000000)
    assert valid_video(input_date, input_url) is valid

Another way to use wrap is like so:

from unittest import mock


class Foo(object):
    def bar(self, x, y):
        return x + y + 1

def test_bar():
    foo = Foo()
    with mock.patch.object(foo, 'bar', wraps=foo.bar) as wrapped_foo:
        foo.bar(1, 3)
        wrapped_foo.assert_called_with(1, 2)

test_bar()

Another simplified version of the above that mocks the whole object, not just a single method:

from unittest import mock

class Foo():
    def bar(self, msg):
        print(msg)

f = Foo()
spy = mock.MagicMock(wraps=f)

spy.bar('baz')
spy.bar.assert_called_with('beep')  # raises AssertionError

Mock builtin `open` function

Python’s mock library provides an abstraction for mocking the builtin open function a lot simpler…

def test_load_ui_messages_successful():
    """Verify ui message YAML file can be read properly."""

    file_content = 'foo: bar'

    with mock.patch('bf_auth.utility.open', mock.mock_open(read_data=file_content), create=True) as mock_builtin_open:
        assert utils.load_ui_messages('./path/to/non/existing/file.yaml') == {'foo': 'bar'}

The create=True param set on mock.patch means that the mock.MagicMock returned will automagically create any attributes that are called on the mock (this is because the open function will attempt to access lots of different things and it’s easier for mock to mock out all of that for you).

Conclusion

There we’ll end. Hopefully this list of mocking techniques will be able to see you through even the most complex of code needing to be tested. Let me know what you think on twitter.

Calculating Big-O

Thu, 28 Feb 2019 00:00:00 +0000

Introduction

This post includes a condensed version of the information gleened from the excellent interactivepython.org section on algorithm analysis. I strongly recommend you read that if you require more detailed information.

The purpose of this post is simply to restructure and simplify the principles offered so I can use it as a quick reference for future practice.

Algorithm

def sumOfN(n):
   theSum = 0
   for i in range(1,n+1):
       theSum = theSum + i

   return theSum

print(sumOfN(10))  # 55

Analysis Steps

You want to quantify the number of operations (or ‘steps’) in the algorithm.

To do this:

Identify the basic unit of computation.
Track any operational constants (e.g. theSum = 0).
Track repeatable operations (e.g. theSum = theSum + i).
Identify the most ‘dominant’ portion.

Explanation

Think about our algorithm sumOfN: we have n number of items and we’re concerned about the time complexity.

If we decide the basic unit of computation is variable assignment, then the formula we would use to express this is T(1+n).

If T(1+n) is the ‘size of the problem’, then using sumOfN as our algorithm, we can evaluate it to mean…

T(1+n) == 1+n steps

The 1 is a constant (i.e. theSum = 0 happens only once), and n is the number of iterations we carry out where a single assignment is made (i.e. theSum = theSum + i) within the sumOfN function.

A critical part of understanding time complexity is that as the problem gets larger, a portion of T(1+n) is likely to overpower the rest and this is where we use the syntax f(...) to represent that portion of the algorithm.

Instead of T(1+n) we could say the dominant part is f(n), and is also referred to as being ‘the order of magnitude’ (which is what the ‘O’ in Big-O stands for).

ℹ️ NOTE

‘order of magnitude’ describes the part of T(n) that increases the fastest as n increases.

We can represent the order of magnitude in ‘Big-O’ syntax like so:

O(f(n))

Where:

f(n) == dominant part of T(n)

Typically we’ll not include the f(...) part of the syntax when using Big-O though. So instead of O(f(n)) we’ll just say O(n).

Significant or Insignificant?

As n gets larger, continuing to use T(n) = 1+n as our example, the ‘constant’ 1 (i.e. the computation that happened once: theSum = 0) becomes less and less significant.

Meaning we can drop 1 from our syntax, resulting in just O(n) instead of O(1+n), and our approximation is just as accurate without it.

To clarify this further, I’m going to paste verbatim the interactivepython description, as I feel they explain this very well…

As another example, suppose that for some algorithm, the exact number of steps is T(n) = 5n2 + 27n + 1005.

When n is small, say 1 or 2, the constant 1005 seems to be the dominant part of the function.

However, as n gets larger, the n2 term becomes the most important.

In fact, when n is really large, the other two terms become insignificant in the role that they play in determining the final result.

Again, to approximate T(n) as n gets large, we can ignore the other terms and focus on 5n2.

In addition, the coefficient 5 becomes insignificant as n gets large.

We would say then that the function T(n) has an order of magnitude f(n) = n2, or simply that it is O(n2).

Example Analysis

a = 5
b = 6
c = 10

for i in range(n):
   for j in range(n):
      x = i * i
      y = j * j
      z = i * j

for k in range(n):
   w = a*k + 45
   v = b*b

d = 33

The above code can be calculated as:

T(n) == 3 + 3n2 + 2n + 1

Which can be condensed slightly, by combining the singular constants, to:

T(n) == 3n2 + 2n + 4

The constants 3 and 1 are the top level variable assignments: a=5, b=6, c=10 and d=33.

The 3n2 is because there are three constant variable assignments (x, y and z, hence the 3 in 3n2) that are occurring within the first set of for statements. The top level for statement iterates over n items, and then does so again hence the n2 portion of 3n2.

The 2n is because there are two constant assignments (w and v) and they happen n times due to the last for statement iterating over n items.

With this in mind we can say the code is O(n2) because when we look at the exponents of each segment of the time analysis (i.e. the condensed version: 3n2 + 2n + 4) we can see that as n grows, the n2 portion is the most significant.

Think about it: looping over n items and making two assignments within each iteration (which is the 2n) is definitely less complexity than looping over n items twice and within each iteration making three assignments (which is the 3n2).

Remember: although we write ‘Big-O’ as O(...) the underlying principle is O(f(...)), where f(...) is the dominant part of T(...) and when focusing in on the dominant part of the time complexity we drop the constants – also known as the coefficient – (e.g. 3n2 thus becomes n2). This is because the constants become insignificant as n grows.

When is Big-O not relevant?

I recently asked in a computer science forum for help in understanding what the Big-O time complexity would be for a ‘web crawler’. Specifically I was asking in relation to the following crawler implementation I had built: go-web-crawler.

The architecture of the program looks something like this:

I wasn’t sure how to calculate the Big-O for this program because there didn’t seem to be any one unit of computation that made sense to use as the foundation of the algorithmic analysis. In the earlier examples it was the variable assignment, but in a web crawler there are so many different moving pieces that make up the whole program.

Also, the implementation between web crawlers will determine different outcomes. So based on my implementation, the running time to handle a list of size n containing nested sublists of size x appears to be O(nx). I had not seen this type of Big-O analysis before, and is an indication of maybe I’m analysing the wrong things.

ℹ️ NOTE

see my previous post for common examples of Big-O.

So how did we come to O(nx)? Here’s the breakdown:

I’m iterating over each list (there are n of them), and each contains x items, so you iterate over nx items in all.
The amount of work per item appears to be constant, i.e., O(1) (it doesn’t appear to depend on n or x).
Multiplying those together, we see that the total time to handle that list is O(nx).

The feedback I received was that Big-O might not be useful for analysing a ‘system’ such as mine because Big-O analysis ignores the constant factors, where as for systems code, we often care about the constant factors a lot.

In other words: I was looking at the system from too high a view. I was looking at the whole rather than picking a specific sub implementation of a particular algorithm.

Another issue is that not all operations are the same, and yet Big-O treats them as such. For example, a variable assignment is not as intensive (computationally or time based) as a network request that can suffer latency and require context switching etc. So in that case Big-O analysis isn’t useful for understanding the performance of the system in practice.

So “no”, it doesn’t make sense to use Big-O all the time. It’ll only make sense from a smaller algorithmic perspective.

These are all useful things to keep in mind when thinking about Big-O time complexity analysis.

Algorithmic Complexity in Python

Sat, 02 Feb 2019 00:00:00 +0000

Introduction

In this post we’re going to review some different algorithmic time complexities. Let me begin by clarifying, when I say ‘algorithm’ I mean: ‘logic written in code’ and when I say ‘operation’ I mean: ‘a unit of code was evaluated’, and that operation could be something as simple as x + y.

Asymptotic Analysis

Asymptotic analysis is the computing of an algorithm’s running time, and there are actually a few different variations that allow us to measure different aspects of that running time:

Big Omega: represents the lower bound.
Big Theta: represents both the lower and upper bounds.
Big O: represents the upper bound.

We’re interested in the last notation in that list (Big O notation) and what the various algorithmic complexity symbols mean when applied to simplified implementations of specific algorithms written in Python.

The reason for selecting Big O over other notations is that it’s the most relevant for performance analysis, as it helps us to understand the worst case behaviour.

ℹ️ NOTE

a fantastic ‘quick’ reference for Big O notation is bigocheatsheet.com, but also Python documents the time complexity associated with its various builtin functions (which is super useful).

Measuring Algorithmic Performance

When measuring the performance of an algorithm we’re interested in how the increase in input size will affect the growth rate of operations required, and this attribute is typically referred to as the algorithm’s ‘time complexity’.

There are two types of complexity we might be interested in (both dependant upon the length of the input to be processed):

time: quantifies the amount of time taken by the algorithm.
space: quantifies the amount of memory used by the algorithm.

ℹ️ NOTE

the Big O notation used to describe these complexities is telling us the ‘growth rate’ of a function, which is typically referred to as the ‘order of the function’ (hence the ‘O’ in Big O).

Orders of Complexity

Good:
- O(1)
- O(log n)
- O(√n)
OK:
- O(n)
Bad:
- O(n log n)
Awful:
- O(n^2)
- O(2^n)
- O(n!)

ℹ️ NOTE

when writing a logarithm you are expected to specify the base: log 2(n) but with Big O notation you typically omit the two, resulting in: O(log n).

Growth Types

O(1): constant time
O(Log n): logarithmic time
O(√n): square root time
O(n): linear time
O(n Log n): linearithmic time
O(n*n): quadratic time (squared)
O(n^2): polynomial time
O(2^n): exponential time
O(n!): factorial time

Constant Time

An algorithm has ‘constant time’ when the number of operations doesn’t change as the number of elements increase.

l = []
l.append(1)
len(l)  # 1

l = list(range(1000))
len(l)  # 1000
l.append(1)
len(l)  # 1001

In the above example code it doesn’t matter how many elements are contained with the list (l). Regardless of whether we append a new element to a list consisting of one element or a thousand elements, the time complexity is O(1) constant time.

Similarly for acquiring the length of a list len(l), regardless of whether the list has one element or a thousand, the time complexity stays constant time.

Logarithmic Time

An algorithm is ‘logarithmic’ when the number of operations decreases by a specific factor with each step.

Consider an algorithm for searching a specific element from a given input:

def binary_search(l, item):
    first = 0
    last = len(l)-1
    found = False

    while first<=last and not found:
        midpoint = round((first + last)/2)
        if l[midpoint] == item:
            found = True
        else:
            if item < l[midpoint]:
                last = midpoint-1
            else:
                first = midpoint+1

    return found

input = [0, 1, 2, 8, 13, 17, 19, 32, 42,]

print(binary_search(input, 3))   # found: False
print(binary_search(input, 13))  # fount: True

The algorithm used here is known as a ‘binary search’ and only works when the given input is sorted/ordered. It works on the principle of dividing the range of elements to be searched by two.

This algorithm fits the requirements of logarithmic time complexity because instead of iterating over the entire input list, we actually shorten the input by half on each ‘step’ of the algorithm.

ℹ️ NOTE

I discuss binary search in more detail in an older post.

Square Root Time

An algorithm has ‘sqrt’ (or √) time complexity when the number of operations increases dependant on the number of primes under the square root of the given number.

Consider an algorithm for checking if a number is a prime. This would have square root time complexity:

def is_prime_number(x):
    if x >= 2:
        for y in range(2,x):
            # if x divides with zero remainder (i.e. equal to bool False)
            if not (x % y):
                return False
    else:
        return False
    return True

Linear Time

An algorithm is ‘linear’ when the number of operations increases linearly with the number of elements.

Consider an algorithm for searching a specific element from a given input:

def search(x, input):
    for i in input:
        print(i)
        if i == x:
            print('found element')
            return

search(5, range(10))

This example search function will loop over every element until it finds the number 5, resulting in it having O(n) linear time complexity. Meaning: if the input range changes from 10 to 1000, then the number of operations (i.e. loop iterations) increases linearly with it.

The worst case scenario is if x happens to be a number that doesn’t exist in the given input. We would have to iterate over the entire input before we realized the number didn’t exist.

ℹ️ NOTE

a much better algorithm to use (if the input was guaranteed to be ordered/sorted) would be a binary search.

What’s interesting to keep in mind is that there are algorithms that are slower still than the example we’ve given above but are still considered to be ‘linear time’ (e.g. O(n)). Consider a collection of ten items that you loop over twice! That’s twice as many operations as our initial example but it’s still O(n) and not something like O(n*2).

Why is that? Well, it stems from how you calculate algorithmic complexity (see my blog post on the subject). But in essence the calculation starts off very explicit until you identify which portion of the algorithm is the ‘dominant’ (e.g. as the size of the input grows, which part of the algorithm gets worse). In the case of O(n*2) the ‘constant’ value is 2, but if the collection size changed from 10 to 1000 then (that being the n part) the n becomes the ‘dominant’ portion of the algorithm and so we can simplify big-o to just O(n).

Linearithmic Time

An algorithm is ‘linearithmic’ when the number of operations increases by the number of elements (i.e. linear time) times the result of log n (i.e. logarithmic time).

Consider the ‘quick sort’ algorithm whose implementation (below) selects a random element as the ‘pivot’ and then loops the entire input list (minus the pivot) in order to identify elements that are less than the pivot and elements that are greater than the pivot (this is the ‘reduce by half’ logarithmic principle). The function recursively calls itself passing in smaller and smaller subsets of input which are iterated over:

from random import randrange

input = [10, 5, 2, 3, 7, 0, 9, 12]

def quicksort(arr):
    if len(arr) < 2:
        return arr
    else:
        rand = randrange(0, len(arr))  # grab a random index
        pivot = arr.pop(rand)
        less = [i for i in arr if i <= pivot]
        greater = [i for i in arr if i > pivot]
        return quicksort(less) + [pivot] + quicksort(greater)

print("sorted:  ", quicksort(input))

ℹ️ NOTE

I discuss quick sort in more detail in an older post.

Quadratic Time

An algorithm is ‘quadratic’ when the number of operations become the square of the number of elements.

Consider an algorithm whose implementation is looping over some input and then nesting another loop of that same input:

def search(input):
    for i in input:
        for j in input:
            print(f'i: {i}, j: {j}')

search(range(10))

This example search function will loop over every element in input and then for each item iterated over it’ll loop through the same top-level input collection again, resulting in it having O(n*n) quadratic time complexity.

In our example code our collection is a list of ten items. Item 1 is the initial list value 0, and for that we’ll loop again ten times. We move to Item 2 (which is the value 1) and again we’ll end up looping ten times …and so on.

This means if the input range changed from 10 to 1000, then the number of operations (i.e. total loop iterations) increases as a square of the number of elements.

Polynomial Time

A polynomial time complexity is effectively a ‘quadratic’ algorithm in the sense that with quadratic O(n*n) (where n is 10) we have a number of operations equal to 100, and if we compare that to polynomial O(n^2) (where n is 10) then again we have the number of operations equal to 100.

Now the difference comes when the exponent (2) in the polynomial equation is increased (where as with quadratic it will always be a squared number). For example, O(n^3) is also referred to as being polynominal.

Exponential Time

An algorithm is ‘exponential’ when the number of operations grows exponentially with the number of elements (i.e. growth whose rate becomes ever more rapid in proportion to the growing total number or size).

ℹ️ NOTE

exponential time complexity O(2^n) is worse than polynomial O(n^2) because maths tells us that over time exponential will quickly overtake polynomial.

Consider an algorithm whose implementation is calculating fibonacci numbers (which grows exponentially relative to the calculated output).

The fibonacci numbers are calculated as the addition of the last element with the current element. The sequence itself starts at zero, so the first seven numbers in the sequence would be:

0, 1, 1, 2, 3, 5, 8

So we can see 0 + 1 = 1, then 1 + 1 = 2, then 1 + 2 = 3 and so on.

Below is a possible implementation:

def fibonacci(num):
    if (num <= 1):
       return num
    return fibonacci(num - 2) + fibonacci(num - 1)

Meaning, if we wanted the sixth number in the sequence (8) we would call our function with fibonacci(6) (as the numbers are zero indexed).

Recursive functions can be difficult to build a mental model from, so let’s attempt to pick it apart with some pseudo-logic:

# f is our fibonacci function
# the logic below is the result of calling f(3)
# the third sequence number is 2
# remembering the sequence is zero indexed
# [0] = 0, [1] = 1, [2] = 1, [3] = 2, [4] = 3, [5] = 5, [6] = 8

f(3):
  3 - 2 = 1
  3 - 1 = 2
  f(1) + f(2)

  f(1) = 1
  f(2) =
     2 - 2 = 0
     2 - 1 = 1
     f(0) = 0
     f(1) = 1

  1 + 1 = 2

Factorial Time

An algorithm is ‘factorial’ when the number of operations increases in line with the number of permutations associated with the number of elements.

Consider an algorithm whose implementation is calculating the factorial, i.e. the number of permutations, for a given number (this is classically known as ‘the travelling salesman’ problem):

def factorial(n):
    for i in range(n):
        print(f'iteration [{i}]: {n}')
        factorial(n-1)

The factorial function recursively calls itself, and the output of this function should be the various permutations of the given number:

There are six permutations for a number up to three (123, 132, 213, 231, 312, 321), but imagine if you wanted the various permutations of a number up to ten?

The calculation for that would yield a disastrously large number:

10*9*8*7*6*5*4*3*2*1 = 3,628,800

Hence this is the worst performing time complexity of the lot.

ℹ️ NOTE

I discuss factorials in more detail in an older post.

Data Types and Data Structures

Wed, 30 Jan 2019 00:00:00 +0000

In this post we will be looking briefly at, and at a high-level, the various data types and data structures used in designing software systems, and from which specific types of algorithms can subsequently be built upon and optimized for.

There are many data structures, and even the ones that are covered here have many nuances that make it impossible to cover every possible detail. But my hope is that this will give you an interest to research them further.

Data Types

A data type is an attribute of data which tells the compiler (or interpreter) how the programmer intends to use the data.

Scalar: basic building block (boolean, integer, float, char etc.)
Composite: any data type (struct, array, string etc.) composed of scalars or composite types (also referred to as a ‘compound’ type).
Abstract: data type that is defined by its behaviour (tuple, set, stack, queue, graph etc).

ℹ️ NOTE

You might also have heard of a ‘primitive’ type, which is sometimes confused with the ‘scalar’ type. A primitive is typically used to represent a ‘value type’ (e.g. pass-by-value semantics) and this contrasts with ‘reference types’ (e.g. pass-by-reference semantics).

If we consider a composite type, such as a ‘string’, it describes a data structure which contains a sequence of char scalars (characters), and as such is referred to as being a ‘composite’ type. Whereas the underlying implementation of the string composite type is typically implemented using an array data structure (we’ll cover data structures shortly).

ℹ️ NOTE

in a language like C the length of the string’s underlying array will be the number of characters in the string followed by a ‘null terminator’.

An abstract data type (ADT) describes the expected behaviour associated with a concrete data structure. For example, a ‘list’ is an abstract data type which represents a countable number of ordered values, but again the implementation of such a data type could be implemented using a variety of different data structures, one being a ‘linked list’.

ℹ️ NOTE

an ADT describes behaviour from the perspective of a consumer of that type (e.g. it describes certain operations that can be performed on the data itself). For example, a list data type can be considered a sequence of values and so one available operation/behaviour would be that it must be iterable.

Data Structures

A data structure is a collection of data type ‘values’ which are stored and organized in such a way that it allows for efficient access and modification. In some cases a data structure can become the underlying implementation for a particular data type.

For example, composite data types are data structures that are composed of scalar data types and/or other composite types, whereas an abstract data type will define a set of behaviours (almost like an ‘interface’ in a sense) for which a particular data structure can be used as the concrete implementation for that data type.

When we think of data structures, there are generally four forms:

Linear: arrays, lists
Tree: binary, heaps, space partitioning etc.
Hash: distributed hash table, hash tree etc.
Graphs: decision, directed, acyclic etc.

ℹ️ NOTE

for a more complete reference,
please see this Wikipedia article.

Let’s now take a look at the properties that make up a few of the more well known data structures.

Array

An array is a finite group of data, which is allocated contiguous (i.e. sharing a common border) memory locations, and each element within the array is accessed via an index key (typically numerical, and zero based).

The name assigned to an array is typically a pointer to the first item in the array. Meaning that given an array identifier of arr which was assigned the value ["a", "b", "c"], in order to access the "b" element you would use the index 1 to lookup the value: arr[1].

Arrays are traditionally ‘finite’ in size, meaning you define their length/size (i.e. memory capacity) up front, but there is a concept known as ‘dynamic arrays’ (and of which you’re likely more familiar with when dealing with certain high-level programmings languages) which supports the growing (or resizing) of an array to allow for more elements to be added to it.

In order to resize an array you first need to allocate a new slot of memory (in order to copy the original array element values over to), and because this type of operation is quite ‘expensive’ (in terms of computation and performance) you need to be sure you increase the memory capacity just the right amount (typically double the original size) to allow for more elements to be added at a later time without causing the CPU to have to resize the array over and over again unnecessarily.

One consideration that needs to be given is that you don’t want the resized memory space to be too large, otherwise finding an appropriate slot of memory becomes more tricky.

When dealing with modifying arrays you also need to be careful because this requires significant overhead due to the way arrays are allocated memory slots.

So if you imagine you have an array and you want to remove an element from the middle of the array, try to think about that in terms of memory allocation: an array needs its indexes to be contiguous, and so we have to re-allocate a new chunk of memory and copy over the elements that were placed around the deleted element.

These types of operations, when done at scale, are the foundation behind why it’s important to have an understanding of how data structures are implemented. The reason being, when you’re writing an algorithm you will hopefully be able to recognize when you’re about to do something (let’s say modify an array many times within a loop construct) that could ultimately end up being quite a memory intensive set of operations.

ℹ️ NOTE

interestingly I’ve discovered that in some languages an array (as in the composite data type) has been implemented using a variety of different data structures such as hash table, linked list, and even a search tree.

Linked List

A linked list is different to an array in that the order of the elements within the list are not determined by a contiguous memory allocation. Instead the elements of the linked list can be sporadically placed in memory due to its design, which is that each element of the list (also referred to as a ‘node’) consists of two parts:

the data
a pointer

The data is what you’ve assigned to that element/node, whereas the pointer is a memory address reference to the next node in the list.

Also unlike an array, there is no index access. So in order to locate a specific piece of data you’ll need to traverse the entire list until you find the data you’re looking for.

This is one of the key performance characteristics of a linked list, and is why (for most implementations of this data structure) you’re not able to append data to the list (because if you think about the performance of such an operation it would require you to traverse the entire list to find the end/last node). Instead linked lists generally will only allow prepending to a list as it’s much quicker. The newly added node will then have its pointer set to the original ‘head’ of the list.

There is also a modified version of this data structure referred to as a ‘doubly linked list’ which is essentially the same concept but with the exception of a third attribute for each node: a pointer to the previous node (whereas a normal linked list would only have a pointer to the following node).

ℹ️ NOTE

again, performance considerations need to be given for the types of operations being made with a doubly linked list, such as the addition or removal of nodes in the list, because you now have not only the pointers to the following node that need to be updated, but also the pointers back to a previous node that now also need to be updated.

Tree

The concept of a ‘tree’ in its simplest terms is to represent a hierarchical tree structure, with a root value and subtrees of children (with a parent node), represented as a set of linked nodes.

A tree contains “nodes” (a node has a value associated with it) and each node is connected by a line called an “edge”. These lines represent the relationship between the nodes.

The top level node is known as the “root” and a node with no children is a “leaf”. If a node is connected to other nodes, then the preceeding node is referred to as the “parent”, and nodes following it are “child” nodes.

There are various incarnations of the basic tree structure, each with their own unique characteristics and performance considerations:

Binary Tree
Binary Search Tree
Red-Black Tree
B-tree
Weight-balanced Tree
Heap
Abstract Syntax Tree

Binary Tree

A binary tree is a ‘rooted tree’ and consists of nodes which have, at most, two children. This is as the name suggests (i.e. ‘binary’: 0 or 1), so two potential values/directions.

Rooted trees suggest a notion of distance (i.e. distance from the ‘root’ node)

ℹ️ NOTE

in some cases you might refer to a binary tree as an ‘undirected’ graph (we’ll look at graphs shortly) if talking in the context of graph theory or mathematics.

Binary trees are the building blocks of other tree data structures (see also: this reference for more details), and so when it comes to the performance of certain operations (insertion, deletion etc) consideration needs to be given to the number of ‘hops’ that need to be made as well as the re-balancing of the tree (much the same way as the pointers for a linked list need to be updated).

Binary Search Tree

A binary search tree is a ‘sorted’ tree, and is named as such because it helps to support the use of a ‘binary search’ algorithm for searching more efficiently for a particular node (more on that later).

To understand the idea of the nodes being ‘sorted’ (or ‘ordered’) we need to compare the left node with the right node. The left node should always be a lesser number than the right node, and the parent node should be the decider as to whether a child node is placed to the left or the right.

Consider the example image above, where we can see the root node is 8. Let’s imagine we’re going to construct this tree.

We start with 8 as the root node and then we’re given the number 3 to insert into the tree. At this point the underlying logic for constructing the tree will know that the number 3 is less than 8 and so it’ll first check to see if there is already a left node (there isn’t), so in this scenario the logic will determine that the tree should have a new left node under 8 and assign it the value of 3.

Now if we give the number 6 to be inserted, the logic will find that again it is less than 8 and so it’ll check for a left node. There is a left node (it has a value of 3) and so the value 6 is greater than 3. This means the logic will now check to see if there is a right node (there isn’t) and subsequently creates a new right node and assigns it the value 6.

This process continues on and on until the tree has been provided all of the relevant numbers to be sorted.

In essence what this sorted tree design facilitates is the means for an operation (such as lookup, insertion, deletion) to only take, on average, time proportional to the logarithm of the number of items stored in the tree.

So if there were 1000 nodes in the tree, and we wanted to find a specific node, then the average case number of comparisons (i.e. comparing left/right nodes) would be 10.

By using the logarithm to calculate this we get: log 2(10) = 1024 which is the inverse of the exponentiation 2^10 (“2 raised to the power of 10”), so this says we’ll execute 10 comparisons before finding the node we were after.

To break that down a bit further: the exponentiation calculation is 1024 = 2 × 2 × 2 x 2 x 2 x 2 × 2 × 2 x 2 x 2 = 2^10, so the “logarithm to base 2” of 10 is 1024.

The logarithm (i.e. the inverse function of exponentiation) of 1000 to base 2, in this case abstracted to n, is denoted as log 2 (n), but typically the base 2 is omitted to just log(n).

When determining the ‘time complexity’ for operations on this type of data structure we typically use ‘Big O’ notation and thus the Big O complexity would be defined as O(log n) for the average search case (which is good), but the worst case for searching would still be O(n) linear time (which is bad – and I’ll explain why in the next section on red-black trees).

ℹ️ NOTE

I’ve covered the basics of logarithm and binary search in a much older post about Big O notation, and so I’ll refer you to that for more details.

Similarly when considering complexity for a particular algorithm, we should take into account both ‘time’ and ‘space’ complexity. The latter is the amount of memory necessary for the algorithm to execute and is similar to time complexity in that we’re interested in how that resource (time vs space) will change and affect the performance depending on the size of the input.

Red-Black Tree

The performance of a binary search tree is dependant on the height of the tree. Meaning we should aim to keep the tree as ‘balanced’ as possible, otherwise the logarithm performance is lost in favor of linear time.

To understand why that is, consider the following data stored in an array:

[1, 2, 3, 4]

If we construct a binary search tree from this data, what we would ultimately end up with is a very ‘unbalanced’ tree in the sense that all the nodes would be to the right, and none to the left.

When we search this type of tree (which for all purposes is effectively a linked list) we would, worst case, end up with linear time complexity: O(n). To resolve that problem we need a way to balance the nodes in the tree.

This is where the concept of a red-black tree comes in to help us. With a red-black tree (due to it being consistently balanced) we get O(log n) for search/insert/delete operations (which is great).

Let’s consider the properties of a red-black tree:

Each node is either red or black.
The root node is always black.
All leaves are ‘NIL’ and should also be black.
All red nodes should have two black child nodes.
All paths from given node to NIL must have same num of black nodes.
New nodes should be red by default (we’ll clarify below).

ℹ️ NOTE

when counting nodes we don’t include the root node, and we count each black node up to (and including) the NIL node.

The height of the tree is referred to as its ‘black-height’, which is the number of black nodes (not including the root) to the furthest leaf, and should be no longer than twice as long as the length of the shortest path (the nearest NIL).

These properties are what enable the red-black tree to provide the performance characteristics it has (i.e. O(log n)), and so whenever changes are made to the tree we want to aim to keep the tree height as short as possible.

On every node insertion, or deletion, we need to ensure we have not violated the red-black properties. If we do, then there are two possible steps that we have to consider in order to keep the tree appropriately balanced (which we’ll check in this order):

Recolour the node
in the case of a red node no longer having two black child nodes.
Make a rotation (left/right)
in the case where recolouring then requires a structural change.

The goal of a rotation is to decrease the height of the tree. The way we do this is by moving larger subtrees up the tree, and smaller subtrees down the tree. We rotate in the direction of the smaller subtree, so if the smaller side is the right side we’ll do a right rotation.

ℹ️ NOTE

there is an inconsistency between what node/subtree is affected by a rotation. Does the subtree being moved into the parent position indicate the direction or does the target node affected by the newly moved subtree indicate the direction (I’ve opted for the latter, as we’ll see below, but be aware of this when reading research material).

In essence there are three steps that need to be applied to the target node (T) being rotated, and this is the same for either a left rotation or a right rotation. Let’s quickly look at both of these rotation movements:

Left Rotation:
1. T’s right node (R) is unset & becomes T’s parent †
2. R’s original left node L is now orphaned.
3. T’s right node is now set to L.

† we now find R’s left pointer has to be set to T (in order for it to become the parent node), meaning R’s original left pointer is orphaned.

Right Rotation:
1. T’s left node (L) is unset & becomes T’s parent †
2. L’s original right node R is now orphaned.
3. T’s left node is now set to R.

† we now find L’s right pointer has to be set to T (in order for it to become the parent node), meaning L’s original right pointer is orphaned.

Let’s now visualize the movements for both rotations:

Left Rotation

Right Rotation

ℹ️ NOTE

rotations are confusing, so I recommend watching this short video for some examples and pseudo-code.

B-tree

A B-tree is a sorted tree that is very similar in essence to a red-black tree in that it is self-balancing and as such can guarantee logarithmic time for search/insert/delete operations.

A B-tree is useful for large read/writes of data and is commonly used in the design of databases and file systems, but it’s important to note that a B-tree is not a binary search tree because it allows more than two child nodes.

The reasoning for allowing multiple children for a node is to ensure the height of the tree is kept as small as possible. The rationale is that B-trees are designed for handling huge amounts of data which itself cannot exist in-memory, and so that data is pulled (in chunks) from external sources.

This type of I/O is expensive and so keeping the tree ‘fat’ (i.e. to have a very short height instead of lots of node subtrees creating extra length) helps to reduce the amount of disk access.

The design of a B-tree means that all nodes allow a set range for its children but not all nodes will need the full range, meaning that there is a potential for wasted space.

ℹ️ NOTE

there are also variants of the B-tree, such as B+ trees and B* trees (which we’ll leave as a research exercise for the reader).

Weight-balanced Tree

A weight-balanced tree is a form of binary search tree and is similar in spirit to a weighted graph, in that individual nodes are ‘weighted’ to indicate the more likely successful route with regards to searching for a particular value.

The search performance is the driving motivation for using this data structure, and typically used for implementing sets and dynamic dictionaries.

Binary Heap

A binary heap tree is a binary tree, not a binary search tree, and so it’s not a sorted tree. It has some additional properties that we’ll look at in a moment, but in essence the purpose of this data structure is primarily to be used as the underlying implementation for a priority queue.

The additional properties associated with a binary heap are:

heap property: the node value is either greater (or lesser depending on the direction of the heap) or equal to the value of its parent.
shape property: if the last level of the tree is incomplete, the missing nodes are filled.

The insertion and deletion operations yield a time complexity of O(log n).

Below are some examples of a max and min binary heap tree structure.

Max Heap:

Min Heap:

Hash Table

A hash table is a data structure which is capable of maping ‘keys’ to ‘values’, and you’ll typically find this is abstracted and enhanced with additional behaviours by many high-level programming languages such that they behave like an ‘associative array’ abstract data type.

In Python it’s called a ‘dictionary’ and has the following structure (on top of which are functions such as del, get and pop etc that can manipulate the underlying data):

table = {'name': 'foobar',
         'number': 123}

The keys for the hash table are determined by way of a hash function but implementors need to be mindful of hash ‘collisions’ which can occur if the hash function isn’t able to create a distinct or unique key for the table.

The better the hash generation, the more distributed the keys will be, and thus less likely to collide. Also the size of the underlying array data structure needs to accommodate the type of hash function used for the key generation.

For example, if using modular arithmetic you might find the array needs to be sized to a prime number.

There are many techniques for resolving hashing collisions, but here are two that I’ve encountered:

Separate Chaining
Linear Probing

Separate Chaining

With this option our keys will contain a nested data structure, and we’ll use a technique for storing our conflicting values into this nested structure, allowing us to store the same hashed value key in the top level of the array.

Linear Probing

With this option when a collision is found, the hash table will check to see if the next available index is empty, and if so it’ll place the data into that next index.

The rationale behind this technique is that because the hash table keys are typically quite distributed (e.g. they’re rarely sequential 0, 1, 2, 3, 4), then it’s likely that you’ll have many empty empty elements and you can use that empty space to store your colliding data.

ℹ️ NOTE

Linear Probing is suggested over Separate Chaining if your data structure is expected to be quite large.

Personally I don’t like the idea of the Linear Probing technique as it feels like it’ll introduce more complexity and bugs. Also, there is a problem with this technique which is that it relies on the top level data structure being an array. Which is fine if the key we’re constructing is numerical, but if you want to have strings for your keys then that wont work very well and so you’ll need to be clever with how you implement this.

Graph

A graph is an abstract data type intended to guide the implementation of a data structure following the principles of graph theory.

The data struture itself is non-linear and it consists of:

nodes: points on the graph (also known as ‘vertices’).
edges: lines connecting each node.

The following image demonstrates a ‘directed’ graph (notice the edges have arrows indicating the direction and flow):

ℹ️ NOTE

an ‘undirected’ graph simply has no arrow heads, so the flow between nodes can go in either direction.

Some graphs are ‘weighted’ which means each ‘edge’ has a numerical attribute assigned to them. These weights can indicate a stronger preference for a particular flow of direction.

Graphs are used for representing networks (both real and electronic), such as streets on a map or friends on Facebook.

When it comes to searching a graph, there are two methods:

Breadth First Search: look at siblings.
Depth First Search: look at children.

Which approach you choose depends on the type of values you’re searching for. For example, relationship across fields would lend itself to BFS, whereas hierarchical tree searches would be better suited to DFS.

Conclusion

The discussion of data structures (and the various high-level data types) in computing is a massive topic, and so we cannot hope to try and cover every aspect and detail due to the sheer broad scope.

That said, I hope you found some useful insights and are now able to look at your programming language of choice with a slightly clearer understanding of the possible structures that sit beneath the abstraction surface.

Python Code Design and Dependency Management

Wed, 02 Jan 2019 00:00:00 +0000

Introduction

This post is going to cover a few tools and features I plan on using when writing Python code in 2019.

† read my post “Python Management and Project Dependencies”.

Type Hints and Static Analysis

Some languages are weakly typed (JavaScript), some are strongly typed (Python) and some are statically typed (Go, Rust). Being strongly typed means you can’t perform operations inappropriate to the type. For example, in Python you can’t add a number typed variable with a string typed variable.

In Python 3.5 we get ‘type hints’ which are a way of annotating Python code with type information in a bid to allow for external tools to provide safety similar to what you might see with a statically typed language.

Python actually ignores these annotations so type hints won’t break your code if you specify one expected type but provide a different one at runtime.

Consider the following code snippet, which uses type hint annotations to indicate the types for both a function’s parameters as well as the function’s return value:

def foo(n: int) -> str:
    print(f'integer: {n}')
    return n

foo('not an integer')  # prints 'integer: not an integer'

So we can see that the code is stating we expect our foo function to receive a parameter of type integer but when the function is called it’s actually passed a string. You’ll find this incorrect argument type isn’t going to break your program.

Our example code also expects a string to be returned by the foo function, but we return an integer value. So this also doesn’t break our program.

If you did nothing else at this point, your code would at the very least be very descriptive of the expectations for its use, but what these type hints afford us is the ability to use external tools for handling static analysis.

The most popular tool currently is called ‘mypy’. You can run mypy via the command line, and it also provides integrated support for most code editors. This means you can catch yourself breaking the expectations of your program when writing/editing code.

Type hints by themselves are quite basic and don’t offer much additional contextual information, so Python added a new typing module to allow for more contextual annotations. One nice feature provided by this typing module is the ability to alias types. The following example is copied verbatim from the Python documentation:

from typing import Dict, Tuple, List

ConnectionOptions = Dict[str, str]
Address = Tuple[str, int]
Server = Tuple[Address, ConnectionOptions]

def broadcast_message(message: str, servers: List[Server]) -> None:
    ...

# The static type checker will treat the previous type signature as
# being exactly equivalent to this one.
def broadcast_message(
        message: str,
        servers: List[Tuple[Tuple[str, int], Dict[str, str]]]) -> None:
    ...

I intend to use type hints a lot more in 2019 to help me both identify potential issues in my code during development, as well as being a means of code clarity as to what input/output types are expected.

For more examples of using the typing module, please refer to either the official documentation or this useful tutsplus.com article.

Interfaces, Protocols and Abstract Methods

In the field of programming there are two concepts that can be a bit confusing to understand:

interfaces
abstract classes/methods

We’ll cover both of these in the following sections, along with Python’s own ‘protocols’ and ‘abstract base classes’.

In summary: an interface is a contract that defines ‘behaviour’, but has no implementation. An abstract class is an actual class that can define common behaviour (including its implementation) along with abstract methods that have no implementation. The implementation of the abstract methods will be defined by the subclass.

Interfaces

An interface is useful for when you don’t necessarily care for a specific concrete implementation of some functionality and will happily accept any object as long as it abides by the behavioural contract your receiver requires.

For the specific use case of interfaces Python has traditionally relied on ‘duck typing’, which is where a caller provides an object and the receiver will attempt to call the appropriate method on the object (thus trusting the object has a corresponding method available).

ℹ️ NOTE

this is where the term duck typing comes from “If it walks like a duck and it quacks like a duck, then it must be a duck”. If the provided object has the expected method exposed, then we presume it’s of a suitable type.

In some cases, such as a reusable shared library, it might be preferable to write code defensibly. Meaning you verify the provided object has the interface the receiver is expecting. This is demonstrated in the following example which expects a client object to be provided, and that object needs to be a HTTP client and so must have both a get and a post method.

We could trust the caller has read the documentation and provided an appropriate object, but we don’t want to rely on that, so we manually defend against that failure scenario in our code:

async def execute_fetch(client, endpoint):
    """Make asynchronous requests via given http client."""

    if invalid_client_interface(client):
        raise tornado.web.HTTPError(500, reason="Invalid HTTP Client")

    ...

def invalid_client_interface(client):
    """Ensure http client has supported interface."""

    if hasattr(client, 'post') and hasattr(client, 'get'):
        return False
    return True

Python not being a statically typed language means it has no support for traditional ‘interfaces’ but the above example ‘defensive’ code is a way to manually mimic it at runtime (as we have no means to validate this at any other time as there is no compilation step with Python, being it’s a dynamic language).

Protocols

Python provides ‘protocols’ as part of their ‘collections’ module, which are also homed alongside another concept in Python called ‘abstract base classes’ (this is something we’ll look at shortly as ABC’s are designed to play nicely with protocols).

Protocols are similar in spirit to interfaces in other languages, but in practice act more like guidelines (this is because Python is a dynamic language and so strictly speaking it isn’t able to validate code correctness because there’s no compilation step with Python).

In essence a protocol is an interface (it defines expected behaviours), while Python’s ‘Abstract Base Classes’ provide a way to offer a form of runtime safety for the interface.

But to understand ‘protocols’ and how we can utilize either MyPy or Abstract Base Classes with them, you’ll need to know a bit about ‘magic methods’ in Python…

Magic Methods:
If one of your custom defined objects implements specific ‘magic’ methods (e.g. __len__, __del__ etc), then you’ll find a selection of builtin Python functions become available to use on those objects that otherwise those builtin functions wouldn’t necessarily support.

For example, if we implement the __len__ magic method, then our object will be able to utilise the builtin len function.

If we utilize protocols, then we can use mypy along with type hinting to implement a development time interface check.

Consider the following code snippet:

class Team:
    def __init__(self, members):
        self.members = members

t = Team(['foo', 'bar', 'baz'])

t.members  # ['foo', 'bar', 'baz']

len(t)  # TypeError: object of type 'Team' has no len()

This code doesn’t work because the len function provided by the Python standard library doesn’t work on custom classes unless the class defines a __len__ magic method.

If we add a __len__ method to the above example (see below), then we would find the Team class now supports the collections.abc.Sized protocol and so the len function will be able to work when given an instance of Team:

class Team:
    def __init__(self, members):
        self.members = members

    def __len__(self):
        return len(self.members)

t = Team(['foo', 'bar', 'baz'])

t.members  # ['foo', 'bar', 'baz']

len(t)  # 3

Now if we want to utilise mypy to help verifying our code at development time, let’s say we have a function that we want to accept any argument type that supports the len function (i.e. anything that supports the collections.abc.Sized protocol), then we can do so using the typing.Sized type (see below example which adds such a function called print_size):

import typing


class Team:
    def __init__(self, members):
        self.members = members

    def __len__(self):
        return len(self.members)


t = Team(['foo', 'bar', 'baz'])


def print_size(s: typing.Sized):
    print(len(s))


print_size(t)

Notice that in the above example we state that the first argument to print_size should be a type of the typing.Sized, which is actually a mapping to the collections.abc.Sized protocol.

If we use the mypy static analysis tool as part of our application testing process (e.g. we only deploy the code if mypy is happy), then we can feel confident our code will be safe.

This is because if we were ever to change the code in a way where we were passing something to print_size that didn’t support calling len() on it, then the mypy analysis would fail.

Custom Protocols

The Python typing module also let’s you define your own protocols using typing.NewType.

Let’s look at a simple example first to understand the use of NewType:

from typing import NewType

I = NewType('I', int)

def foo() -> I:
   return I(123)

Notice a few things in the above example:

the first argument to typing.NewType needs to match the name of the variable it is assigned to.
we can’t just return an integer, it needs to be casted to the new type I first.

Now we have a basic understanding of NewType let’s consider the following example where we create a new custom protocol called CustomProtocol:

import typing


class Team:
    def __init__(self, members):
        self.members = members

    def __len__(self):
        return len(self.members)


t = Team(['foo', 'bar', 'baz'])


def print_size(s: typing.Sized):
    print(len(s))


print_size(t)  # prints '3'

CustomProtocol = typing.NewType('CustomProtocol', Team)

cp = CustomProtocol(Team(['beep', 'boop']))  # <class '__main__.Team'>

print_size(cp)  # prints '2'

ℹ️ NOTE

when we create an instance of CustomProtocol the underlying ‘type’ is Team.

The mypy static analysis tool can subsequently be used to verify code for both native protocols and custom protocols, like so (see the type hint annotation added to the print_size function, which mypy is happy with):

class Team:
    def __init__(self, members):
        self.members = members

    def __len__(self):
        return len(self.members)


CustomProtocol = typing.NewType('CustomProtocol', Team)


def print_size(s: CustomProtocol):  # we could also set type to `Team` 
    print(len(s))  # prints '2'


cp = CustomProtocol(Team(['beep', 'boop']))

print_size(cp)

ℹ️ NOTE

the argument type passed to print_size is CustomProtocol which doesn’t make mypy complain because the underlying type for CustomProtocol is actually the Team class, and the underlying Team class is supporting the typing.Sized interface (which maps to the collections.abc.Sized protocol).

If you want more information on mypy’s support of protocols, I suggest reading their specific documentation here.

Abstract Classes/Methods

An abstract class allows you to define common behaviour as well as ‘abstract methods’ that have no implementation, in which the subclass will be required to provide the implementation.

We can mimic that concept in Python using standard classes along with the classic ‘template method’ pattern as shown in the following example:

class MyAbstractClass:
    def common(self):
        print('common behaviour')

    def MyAbstractMethod(self):
        raise NotImplementedError

class Foo(MyAbstractClass):
    def MyAbstractMethod(self):
        print('do something')

class Bar(MyAbstractClass):
    pass

f = Foo()
f.common()  # prints 'common behaviour'
f.MyAbstractMethod()  # prints 'do something'

b = Bar()
b.common()  # prints 'common behaviour'
b.MyAbstractMethod()  # raises NotImplementedError

o = MyAbstractClass()  # not possible in other languages (see note below)
o.common()  # prints 'common behaviour'
o.MyAbstractMethod()  # raises NotImplementedError

ℹ️ NOTE

in other languages that support proper abstract classes, you would not be able to instantiate the abstract class directly (like we have done in our example).

Luckily Python does also provide us with what it refers to as ‘Abstract Base Classes’ (here in referred to as ABC’s) which are a form of traditional abstract class, so there’s no need to necessarily mimic the behaviour like in our earlier example. See the following example that demonstrates this feature:

import abc

class Foo(abc.ABC):
    @abc.abstractmethod
    def bar(self):
        pass
        
class Thing(Foo):
    pass
    
t = Thing()  # TypeError: Can't instantiate abstract class Thing with abstract methods bar

To make the above example code work correctly we need our class Thing to actually implement the exepected behaviour (i.e. a bar method). If Thing doesn’t provide the expected behaviour then we can’t instantiate a subclass of Foo.

This also means that if we’re using a static analysis tool such as mypy, we could have a receiver state it expects a type of Thing and know more confidently that Thing will definitely provide the behaviour we need.

It’s important to understand that the use of an abstract class is subtly different to the use of traditional interfaces in that an interface doesn’t rely on a concrete implemention.

For example, our Thing class is a concrete implementation, and so we can’t provide the receiver with a different class (even if the other class also happened to inherit from Foo) as it won’t be equivalent to a Thing type.

ℹ️ NOTE

the mypy docs have a good detailed breakdown of how to indicate a dependency of a specific class type.

Dependency Management (with pipenv)

UPDATE 2019.12.20: I no longer use Pipenv (as per below). I’ve written an updated version of how best to handle dependencies here.

Python has historically utilised a requirements.txt file for defining the dependencies required of your program, but there are various annoying complications that go along with the traditional model of handling dependencies which has meant we have a few new players in the field to help us.

One such concern is the setting up of multiple virtual environments for the various projects we need to work on:

XKCD: right as always

So here are the various alternatives we have to play with in 2019:

I’ll be showing you the last tool in the list: Pipenv.

Although another alternative approach to the specific problem of virtual environments is to utilise docker containers for doing your development, but you’ll need to be comfortable using a terminal editor like Vim (unless you want to jump through some X11 hoops). Using containers also doesn’t eliminate the other issues with determining the right dependencies, so keep reading anyway.

ℹ️ NOTE

if using Docker with a terminal editor like Vim to solve this problem sounds like a good approach for you, then review an older post of mine that explains how to do that.

Here are the commands necessary to install Pipenv on macOS:

brew install pyenv
pip install pipenv

ℹ️ NOTE

you’ll need Homebrew to install the pyenv command (a sub dependency) using brew, and macOS should have Python 2.7.x installed by default so you should have the pip command available already.

Here are my quick steps for setting up a new project with Pipenv:

mkdir foobar && cd foobar
pipenv --python 3.7

ℹ️ NOTE

use pyenv install --list to find out what Python versions are available to install.

Now when working on a Pipenv project:

cd foobar
pipenv shell or pipenv run python ./app.py

ℹ️ NOTE

use the shell subcommand to have your current terminal permanently use the chosen Python version (e.g. python ./app.py will work as if the current Python version is what you’ve defined), otherwise use the run subcommand to execute the given command (e.g. python ./app.py) within the chosen Python version temporarily.

You can now install dependencies specifically for the project’s specific Python environment:

pipenv install tornado==5.0.2
pipenv install --dev mypy tox flake8

ℹ️ NOTE

if you have an existing requirements.txt file, then you can generate a Pipfile from that using pipenv install -r requirements.txt, alternatively if you need to do the reverse (generate a requirements from a Pipfile): pipenv lock --requirements

Now none of these new tools are perfect, and if you want a good run down of one engineer’s perspective on them, read here.

Conclusion

That’s it. We’ve looked at how to handle dependencies with Pipenv, and how to utilise static analysis tool mypy along with type hinting to give us more confidence in our code (as well as having the code become clearer intent).

Lastly we looked at how to utilise interfaces and abstract classes to help improve the structure and safety of our code.

Along with new additions to the asyncio module (a simpler api for a start) and cleaner abstractions such as the new data classes features, the future of Python hasn’t looked brighter.

Modern JavaScript: Webpack and Babel

Sat, 29 Sep 2018 00:00:00 +0000

Introduction

This post will explain how to set-up and configure the various tooling necessary in order to be able to write cross-compatible modern (ES2015+) JavaScript code.

ℹ️ NOTE

if you’re unsure of what ‘modern’ JavaScript looks like, then I’ll refer you to these compatibility tables.

The tools we’ll be using:

Babel: transpiler of modern JS into ES5 compatible code.
Webpack: a js module bundler.

ℹ️ NOTE

webpack is actually capable of transforming, bundling, packaging just about anything (as we’ll see shortly).

To clarify, you don’t need ‘webpack’, as babel handles transpiling modern JS code into ES5 compatible code, but it makes sense to use webpack still as a way to help with the performance of your client-side services.

With that in mind the configuration I’ll be describing will be using webpack primarily, and under that I’ll be using webpack ‘loaders’ to utilise the babel transpiler.

If you weren’t using webpack, the configuration would be different as you’d be configuring babel directly instead of webpack.

Example Project

We’re going to create a very basic project. It’s so basic, it doesn’t really do anything. It’s the bare minimum required in order to demonstrate the setup and configuration of webpack and babel (I purposely did this because learning new tech can be confusing enough without needing to understand a real-world application at the same time).

One thing you’ll need upfront though, is Node and NPM installed, as we’ll be installing webpack and babel from existing NPM packages.

Let’s begin by creating our project directory:

mkdir modern-js && cd modern-js

ℹ️ NOTE

I will be working exclusively from the terminal.

Now create an empty package.json file:

npm init -y

Next, we’ll install all the relevant packages we’ll be needing…

npm install --save-dev webpack \
                       webpack-cli \
                       webpack-dev-server \
                       @babel/core \
                       @babel/preset-env \
                       "babel-loader@^8.0.0-beta" \
                       style-loader \
                       css-loader \
                       sass-loader \
                       node-sass \
                       eslint@4.x babel-eslint@8

npm install --save @babel/polyfill

ℹ️ NOTE

the dev dependencies need to be installed in the order they’re specified above, otherwise npm will complain/fail.

Your package.json should now have the following content:

{
  "name": "modern-js",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@babel/core": "^7.1.2",
    "@babel/preset-env": "^7.1.0",
    "babel-eslint": "^8.2.6",
    "babel-loader": "^8.0.4",
    "css-loader": "^1.0.0",
    "eslint": "^4.19.1",
    "node-sass": "^4.9.3",
    "sass-loader": "^7.1.0",
    "style-loader": "^0.23.0",
    "webpack": "^4.20.2",
    "webpack-cli": "^3.1.2",
    "webpack-dev-server": "^3.1.9"
  },
  "dependencies": {
    "@babel/polyfill": "^7.0.0"
  }
}

There are two important Babel related packages to pay attention to…

@babel/polyfill
@babel/present-env

The @babel/polyfill dependency will help emulate a full ES2015+ environment, and this means you can use new built-ins like Promise or WeakMap, static methods like Array.from or Object.assign, instance methods like Array.prototype.includes (amongst others, see the documentation for more information).

The @babel/preset-env helps manage the browser environment for you, so it’ll handle determining what additional scripts/polyfills you need. By default it’ll setup everything to support ES5 environments, but you can configure it for very specific browsers if you don’t need to worry about older browsers.

ℹ️ NOTE

@babel/polyfill also provides a useBuiltIns flag which allows Babel to selectively polyfill built-in features that were introduced as part of ES6+. Because it filters polyfills to include only the ones required by the environment, we mitigate the cost of shipping with babel-polyfill in its entirety.

Now let’s create all the files we need to build out our example application:

mkdir src dist
touch .eslintrc src/index.js src/component.js src/styles.scss dist/index.html

This should result in the following tree structure…

tree -I node_modules

.
├── .eslintrc
├── dist
│   └── index.html
├── src
│   ├── component.js
│   ├── index.js
│   └── styles.scss

We can see we have one Sass file and two JavaScript files, as well as a single HTML page that will load our scripts/css.

Let’s now look at the contents of each of these files…

.eslintrc

{
  "parser": "babel-eslint",
  "globals": {
    "console": true,
    "document": true,
    "window": true
  },
  "rules": {
    'brace-style': [2, '1tbs', {'allowSingleLine': true}],
    'camelcase': 2,
    'comma-spacing': 2,
    'comma-style': 2,
    'curly': 2,
    'eol-last': 2,
    'indent': [2, 2],
    'key-spacing': 2,
    'new-cap': 2,
    'new-parens': 2,
    'no-lonely-if': 2,
    'no-multi-spaces': 2,
    'no-multiple-empty-lines': [2, {'max': 2}],
    'func-call-spacing': 2,
    'no-trailing-spaces': 2,
    'quotes': [2, 'single', {'allowTemplateLiterals': true}],
    'semi': 2,
    'semi-spacing': 2,
    'space-before-blocks': 2,
    'space-in-parens': 2,
    'space-infix-ops': 2,
    'space-unary-ops': 2,
    'array-callback-return': 2,
    'block-scoped-var': 2,
    'consistent-return': 2,
    'eqeqeq': 2,
    'guard-for-in': 2,
    'no-array-constructor': 2,
    'no-caller': 2,
    'no-cond-assign': 2,
    'no-const-assign': 2,
    'no-control-regex': 2,
    'no-delete-var': 2,
    'no-dupe-args': 2,
    'no-dupe-class-members': 2,
    'no-dupe-keys': 2,
    'no-duplicate-case': 2,
    'no-empty-character-class': 2,
    'no-empty-pattern': 2,
    'no-eval': 2,
    'no-ex-assign': 2,
    'no-extend-native': 2,
    'no-extra-bind': 2,
    'no-fallthrough': 2,
    'no-func-assign': 2,
    'no-implied-eval': 2,
    'no-invalid-regexp': 2,
    'no-iterator': 2,
    'no-lone-blocks': 2,
    'no-loop-func': 2,
    'no-mixed-operators': [2, {
      groups: [
        ['&', '|', '^', '~', '<<', '>>', '>>>'],
        ['==', '!=', '===', '!==', '>', '>=', '<', '<='],
        ['&&', '||'],
        ['in', 'instanceof']
      ],
      allowSamePrecedence: false
    }],
    'no-multi-str': 2,
    'no-native-reassign': 2,
    'no-unneeded-ternary': 2,
    'no-unsafe-negation': 2,
    'no-new-func': 2,
    'no-new-object': 2,
    'no-new-symbol': 2,
    'no-new-wrappers': 2,
    'no-obj-calls': 2,
    'no-octal': 2,
    'no-octal-escape': 2,
    'no-redeclare': 2,
    'no-regex-spaces': 2,
    'no-script-url': 2,
    'no-self-assign': 2,
    'no-self-compare': 2,
    'no-sequences': 2,
    'no-shadow-restricted-names': 2,
    'no-shadow': 2,
    'no-sparse-arrays': 2,
    'no-template-curly-in-string': 2,
    'no-this-before-super': 2,
    'no-throw-literal': 2,
    'no-undef': 2,
    'no-unexpected-multiline': 2,
    'no-unreachable': 2,
    'no-unused-expressions': [2, {
      'allowShortCircuit': true,
      'allowTernary': true
    }],
    'no-unused-vars': 2,
    'no-use-before-define': [2, 'nofunc'],
    'no-useless-computed-key': 2,
    'no-useless-concat': 2,
    'no-useless-constructor': 2,
    'no-useless-escape': 2,
    'no-useless-rename': 2,
    'no-with': 2,
    'radix': 2,
    'require-yield': 2,
    'use-isnan': 2,
    'valid-typeof': 2,
    'wrap-iife': [2, 'any']
  }
}

You don’t have to worry too much about the contents of this file, as it’s just the configuration I’m using to define what is ‘good’ JavaScript syntax.

In other words, if your code editor is configured properly, then any JS code you write that violates any of these ES linter values, will be flagged as an error.

dist/index.html

<!doctype html>
<html>
  <head>
    <title>Hello Webpack</title>
  </head>
  <body>
    <script src="bundle.js"></script>
  </body>
</html>

This is simple enough, a HTML page that loads a bundle.js script (which currently doesn’t exist, and will be generated by webpack, via babel).

src/component.js

const c = ['x', 'y', 'z'];

export default c;

A simple script that defines an Array of items and then exports them (using modern JS module syntax, familiar to people who may have written Node applications before).

src/index.js

/*eslint no-undef: "error"*/
/*eslint-env browser*/

import '@babel/polyfill';
import './styles.scss';
import c from './component.js';

console.log(c);

let a = 1;
let b = 2;
[a, b] = [b, a];
console.log(a);
console.log(b);

const root = document.createElement('div');
root.innerHTML = `<p>Hello Webpack!</p>`;
document.body.appendChild(root);

A simple script that imports from our component.js and then logs it (to prove the import code works), it then demonstrates let variables and another modern JS feature known as ‘destructuring’ before finally creating a HTML <div> element and populating it with some text and inserting it into the DOM of the HTML page.

At the top of the script you’ll notice some code comments. These are used to tell our ES linter package what context this script is running in, and so global references such as document or console won’t trigger a linting error as we’ve told the linter that the context the script will run is the browser environment, where those globals are expected to exist. The code comment for eslint-env could be replaced by adding individual references into the .eslintrc file (and I have done that, take a look at the globals field in the file contents shown earlier), but I prefer the code comment as it can be a much clearer indicator of the expectations of the file’s scope.

You’ll also notice that we import the @babel/polyfill at the top of the file. This module must always be the first import in the file.

Lastly, you’ll notice we also import a Sass file, which admittedly is a bit strange considering we’re dealing with a JS file, but we’ll dig into this a little bit more later on and why we do that.

src/styles.scss

$color: blue;

body {
  background-color: $color;
}

This is a simple Sass file that demonstrates how to use a ‘variable’ to generate dynamic CSS output (in this case, the body element should have a blue background).

Webpack Configuration

OK, at this point we have a set of JS files that can’t be used in some browsers due to the fact that they use features not supported by most web browsers. So we want to compile this code down into something that is understandable to most browsers (i.e. ES5 standardized code).

To do that we’ll be using Babel (as our transpiler) and Webpack as our module bundler. Webpack will take the separate JS files and combine them into a single bundle.js file, which our HTML will attempt to load.

So let’s create a webpack.config.js file:

touch webpack.config.js

We’ll then add the following contents to that file:

/*eslint-env node*/

const path = require('path');

module.exports = {
  entry: './src/index.js',
  output: {
    filename: 'bundle.js',
    path: path.resolve(__dirname, 'dist'),
    publicPath: '/dist'
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules|bower_components)/,
        use: {
          loader: 'babel-loader',
        }
      },
      {
        test: /\.scss$/,
        use: [
          {loader: 'style-loader'},
          {loader: 'css-loader'},
          {loader: 'sass-loader'}
        ]
      }
    ]
  }
};

You’ll notice a similar code comment at the top of the file, which again is for the benefit of our ES linter. In this case it won’t complain when it sees certain global references such as require or module (as we’ve told the linter that the context the script will be run in is the Node environment, where those globals are expected to exist).

We see the entry field of the configuration is telling webpack that the main application JavaScript file is going to be found at ./src/index.js. Webpack will look at this file and then traverse back up its dependency tree looking for imports and it will combine each of those separate files into one single file.

The file that is generated (and where) is determined by the output field. We can see we want the file to be called bundle.js and we want the file to be saved to the ./dist directory (this is indicated by the path field).

The publicPath field is a bit different, in that it tells the webpack-dev-server package where to find the bundle.js file that the HTML is attempting to load. What won’t be clear yet is the fact that when running our code locally (in dev) we’ll be using webpack-dev-server because it allows us to utilise a web server for running our code as well as ‘hot reloading’ (which means, if we’re dealing with a complex single-page application with lots of nested state, that a change in code doesn’t cause us to lose the state the page is in).

For production we’ll statically generate our final bundle file using the standard webpack command, and so you’ll see shortly that we need to update our package.json to include two npm run ... commands that let us use either webpack-dev-server or webpack depending on where our code needs to run.

The module field tells webpack, that for every file it finds, before adding it to the final bundle.js, to run it through a ‘loader’ script for additional processing. In this case, all .js files are passed through babel and so their modern JS code is transpiled into ES5 code first before being added to bundle.js.

Finally, we look for any Sass .scss files and tell webpack to pass those files through multiple ‘loaders’:

sass-loader: transforms Sass into CSS.
css-loader: parses the CSS into JavaScript and resolves any dependencies.
style-loader: outputs our CSS into a <style> tag in the document.

Loaders are executed in a nested fashion, meaning the above order of loaders would evaluate to something like:

styleLoader(cssLoader(sassLoader("source")))

Where the source file is passed into the Sass loader, and so transforming the Sass into CSS. That CSS is then passed into the CSS loader, which allows it to be parsed by JavaScript. Finally, the Style loader places our CSS into a <style> tag within our HTML page.

Package.json Update

As mentioned earlier, we want to modify the package.json so that we have two npm run ... commands for letting us run our code in development mode (i.e. locally) or compile our code ready for use in production.

The following snippet shows the changes needed to be made:

{
  ...

  "scripts": {
    "build": "webpack --mode=production",
    "dev": "webpack-dev-server --mode=development --config webpack.config.js",

    ...
  },
  
  ...
}

Now we can run either npm run dev (for local dev) or npm run build (to compile our bundle for production).

You’ll notice that we pass a specific --mode flag to both webpack and webpack-dev-server, and this indicates to both tools what to do to the files it’s configured to interact with.

In the case of --mode=production the webpack tool will make additional modifications that means the bundle.js output is as efficient as possible (such as minifiying and obfuscating the code).

Whereas --mode=development will allow for the generation of webpack source map files (to aid you in debugging).

Conclusion

This should be the basics covered of how to use babel with webpack, and hopefully is enough to help you kickstart your exploration of new JavaScript features.

Engineer to Manager

Sat, 25 Aug 2018 00:00:00 +0000

You’re a software engineer with many years of technical experience, and you’ve decided you want to kickstart the process of working towards becoming an ‘Engineering Manager’.

So what do you need to know? Let’s discuss…

Roles

As a software engineer you have a few different paths available to you for career progression:

Senior Software Engineer
Staff Software Engineer
Principal Software Engineer
Architect
Tech Lead
Engineering Manager

ℹ️ NOTE

the role hierarchy can change per organisation

You may notice some of these roles are not like the others 😉 It’s worth recognizing that there is a slight separation between the last two roles (Tech Lead, Engineering Manager) and the rest of the roles, in that the last two are more closely related to management than the other roles.

I’ve grouped these roles together like this because they tend to be the progression path for a lot of engineers. They exhaust all possibilities as far as ‘engineering’ as an IC (individual contributor) before finally residing themselves to a management role.

This doesn’t have to be the case though. You might decide to make that shift to a more management role before progressing through the more senior engineering roles, but you may find some organisations will expect a certain level of engineering experience.

When moving up the career ladder you’ll typically find that between roles there is a fair amount of cross-over responsibilities, but for the purposes of this post let’s focus in on the last role (‘Engineering Manager’) and what that entails.

Responsibilities

Below I have listed a few different areas of responsibility (as I see them) associated with an Engineering Manager role, and of course these responsibilities can change depending on the culture of where you work.

Meaning it’s a good idea, when interviewing for an Engineering Manager role, to ask clarifying questions in order to help you understand the expectations of that role (in the specific context of that organisation).

ℹ️ NOTE

to learn more about what to talk about during an interview, read my post ‘Interview Topics’

For example, some companies have dedicated ‘Project Managers’; meaning an Engineering Manager doesn’t have to worry too much with that aspect of the job, and can instead focus on technical leadership responsibilities.

Another example is the role of a ‘Tech Lead’; some organisations use this name and ‘Engineering Manager’ interchangeably, while others treat them as distinct roles (and in some cases a less management/more code oriented role than an Engineering Manager).

ℹ️ NOTE

see also my post ‘Project Management in Five Minutes’ which discusses Lara Hogan’s venn diagram of various leadership roles, as well as a project management checklist.

Motivation

A good engineering manager must be committed to building great software, great teams, creativity, careers, and general welfare of the team members.

With that in mind, the following list feels like (to me) a reasonable breakdown of the various jobs you (as an Engineering Manager) will be expected to look after…

Team Leader
- 1:1’s (read my post on better 1:1 practices)
- Mentoring
- Support
- Career development
Technical Lead
- Define and communicate with non-technical stake holders
- Guide the architecture and technical design
- Resolve conflicts around technical discussions
- Write code and work with the engineering team
Project Management
- Define technical requirements
- Work with non-technical stake holders to define user stories
- Breakdown large tasks into smaller tasks
- Prioritise work and revise priorities when necessary
- Establish schedules, estimates and milestone deliveries
- Run and guide the technical aspect of the project
Communicator
- Up: team/project status and health to upper management
- Down: updates/decision making to your team and direct reports
- Across: identify, engage and collaborate with other teams
Recruitment
- Work with recruiters to define required engineering roles
- Screen resumes
- Interview candidates
- Assist onboarding new recruits

Communication

The most important skill you’ll need, if you are to be effective as a manager of people (e.g. your direct reports, your managers, your stake holders: both technical and non-technical), is communication.

You need to be able to build strong and trusted relationships with these people. You need to be able to challenge them and push them to do better, for their own success, not the business.

To elaborate further: helping people to reach their goals, to be happy and to help them find doing what they enjoy doing (while still challenging them) will ultimately yield better value from the work they do, and as a side-effect the business will benefit more greatly than if you push someone down another path designed with the business in mind.

This means caring personally about the people you are interacting with and doing what you can to bring out their best. It also means recognising when someone isn’t a good fit for your team, and in extreme cases ‘letting them go’ can actually work out better for both them and the business.

Patty McCord discusses this topic in-depth in her book on culture building at scale “Powerful: building a culture of freedom and responsibility”.

In essence let’s help people work on things that will realize their ambitions, but not at the expense of the business, more in co-operation with the business. But if your ambitions don’t fit with the business, then maybe it’s best for you and the business to separate ways. I like that as a healthy way for both sides to either work together for a common good or separate ways for an equally better outcome.

When building relationships you should be mindful of not falling too heavily on one side or another with regards to ‘professional vs personal’. Find a balance, and if unsure err on the side of building a respected professional relationship, as friendship and a greater sense of personal appreciation/understanding (in some cases) will naturally occur.

Below I’ve listed some useful resources for learning how to communicate effectively (these are all highly recommended):

Authentic Communication by Fred Kofman.
Radical Candor by Kim Scott.
SBI: Situation, Behaviour, Impact a methodology developed by The Center for Creative Leadership.
The Hard Thing About Hard Things by Ben Horowitz.

ℹ️ NOTE

‘Authentic Communication’ is actually taken from a larger work called Conscious Business: How to build value through values.

Team Leader

As someone who is going to be mentoring and supporting a team of engineers, it is important you understand the motivations and what drives your team. In doing so you’ll be better equipped to find them the right roles within the organisation that brings out their best work and allows them to grow in the areas that matter the most to them.

It’s also important to sometimes think as an ‘engineer’ again. When you design and develop large scale systems you identify bottlenecks and ways of working that reduce complexity and support certain services to scale. Now think about that when you’re handling management duties.

Do you find yourself handling tasks, or using processes, that take up a lot of time? This might be ok for now while you’re responsible for a small team but how will that work out for you once you start taking on more teams and more responsibilities? Be mindful of the tools and processes you use and refine them constantly.

Why?

The single biggest question you should now be asking yourself is:

“Why do I want to become an Engineering Manager?”

This is really important, and requires you to be very honest and open with yourself (not only will you suffer the consequences of a bad decision, but so will everyone else who is relying on you).

If you answered the question with any one of the following (or more), then you need to seriously consider an alternative future career path; simply because you truly won’t be happy in this role otherwise:

I want more money
I want a promotion
I want a better job title
I’m bored of writing code

To me, a good reason to want to progress towards being an Engineering Manager is when you deep down know that your enjoyment comes from helping people and having a positive impact in their ability to do great work and helping them to progress in their career.

If that is your motivation, then you have the right mentality for this type of role. If that’s not your intent, then that’s OK (don’t worry) but you will need to consider alternative options and really figure out what it is about working in tech/engineering that you enjoy, and do what you can to move more in that specific direction.

Ultimately if you want things like more money or a promotion or a better job title, that will require you to have more impact at work, and to do that means you need to lift your head up from your laptop and engage more widely with the organisation and the teams within and start demonstrating more broadly the potential positive impact that you can have.

ℹ️ NOTE

see also my post on ‘Architecture Interviews (and other thoughts)’, where I cover what I look for when interviewing an Engineering Manager candidate.

Incremental Steps

You might not be quite ready for a straight jump up to Engineering Manager, so what can you do in the meantime to help you move towards that goal? Here are some ideas…

Mentoring
- Most companies have a mentoring program, and even if not it doesn’t prevent you from reaching out to colleagues and offering them your help, guidance and the benefits of your experience (if they choose to accept it)
Take Responsibility
- Get involved with cross-team discussions
- Create a working-group for a topic you care about
- Support your team and reach out on their behalf if you’re able to
Lead
- Take the responsibility of leading a project
- Beyond that take the opportunity to lead a team
- Help non-technical stakeholders understand the team’s work

Manager Cheat Sheet

BuzzFeed recently hired a new Engineering Manager 🎉 🙂

Her name is Svetlana Bozhko and she kindly shared with me a ‘cheat sheet’ (of sorts) that helps her stay on-track/focused.

I know of other Engineering Managers/Directors of Engineering (some of whom work for a few of ‘the big 4’) who use a similar approach, and it’s something I personally like.

Here’s an example of what such a ‘cheat sheet’ might look like:

Daily

Personal
- Am I ready for today’s meetings?
- Can I reduce the length of those meetings?
- Can I cancel any of those meetings?
People
- Recruiting, Hiring, Interviewing
- Candidate follow-ups
- Sell the company, its vision, roles etc
- Close great candidates
- Observe team health
- Casual feedback sessions
Processes
- Daily standups
- Am I ready for these standups?
- Any important company-wide updates?
Projects
- Are we on track with critical projects?
- Any docs/guides needed to be written?
- Any PRs waiting my approval/feedback?
- Should I step in on any engineering tasks?

Weekly

Personal
- Have I setup my goals for this/next week?
- How do I progress with my goals?
People
- 1:1’s and enhancing their strengths
- Who needs additional feedback?
- Any tasks I can delegate?
Processes
- Review all tasks in backlog
- Any design/architecture discussions?
- Divide tasks into smaller ones
- Plan team retrospectives
Projects
- Give status reports to my managers
- Who else do I need to update?

Monthly

Personal
- What do I need to do to be more successful in my role?
- Any training I can do, mentoring I can give, books I can read?
People
- Ensure team vision is aligned
- Ensure team expectations are understood
- Any team events we should consider?
- 1:1’s focused specifically on career growth
- Anyone moved into new roles?
- Do I need to set new expectations?
- Any changes to the career ladder?
- What changes would make it more fun to work here?
Processes
- Challenge and question existing processes
- Are the current sprint metrics still useful?
Projects
- Any strategic ideas to improve current projects?
- Do we need a new project?
- Dow we need to stop a project?
- Any strategic hires needed?
- Budget and capacity planning

Conclusion

So what do you think? Is there anything here that you disagree with, or something you feel I missed altogether? Let me know on twitter!

Interview Techniques

Sat, 18 Aug 2018 00:00:00 +0000

Introduction

This post will explain a little bit about a particular type of interview: an architecture interview. It’ll breakdown what it is, why using whiteboards isn’t necessarily a bad thing to use in an interview context, as well as understanding a bit about what it is we’re looking for from a particular candidate (in this post the information will be related to interviewing someone for an Engineering Manager opportunity).

What is an architecture interview?

An architecture review can be a few different things depending on the context of the interview and the type of role being applied for.

Typically, for a software engineering role, a candidate will be asked to design a system architecture based on the interviewer’s given requirements.

ℹ️ NOTE

this is typically carried out on a whiteboard, but pen and paper is sufficent too (although harder for both candidate and interviewer to share and discuss over).

Whereas for a more senior role, like an engineering manager (or even a director of engineering), the format will change to one where the candidate is asked to present a system architecture they’ve been involved with.

The reason for the switch around of formats is that generally the more senior roles won’t necesarily be controlling the system design (they’ll have input, but generally speaking this would be the responsibility of individual teams), and so what you’re trying to gauge from that type of interview is an understanding of whether they have a solid technical grounding, because their other skills are of a bigger priority.

Is one interview all it takes?

No. To clarify: when interviewing a candidate, you won’t give them a single architecture interview and call it a day (i.e. make a decision). There are many interviews that need to happen before you can make such a decision:

Architecture
Organizational
Team Values
Executive
Cross-Discipline
Team Managment

ℹ️ NOTE

the above are based on a ‘Director of Engineering’ day of interviews, but you’d have similar formats for varying engineering roles.

Aren’t whiteboard interviews bad?

In summary: no (at least they’re not inherently bad).

When you say “whiteboard interview”, people generally tend to cringe and think of past experiences of being told to solve X problem using Y algorithm, and writing out chunks of pseudo-code with pen and paper.

But let’s be honest, that rarely correlates to the actual work you typically end up doing, and so to describe that format as “wrong” would be correct †

† although, as with everything, it can depend on the role you’re hiring for of course.

For me, that type of whiteboarding session is a problem. Whiteboard interviews of that nature, the ones that have you jumping through hoops to perform some kind of mathematical trickery are at best useless, and at worst very stressful. They’re also likely to result in a candidate leaving unenthused and saying bad things about the company and its interview process on social media.

You should not be having whiteboard interviews consisting of questions of that kind. Whereas architecture interviews are (in my experience any way) about more about discussing ‘high-level’ concepts, and demonstrating ‘practical’ problem solving skills which are applicable at many different levels of experience, and useful in many different situations.

If structured (and presented) correctly by the interviewer, a whiteboard interview (such as an architecture interview) can actually be fun for both the candidate and the interviewer, and be an enlightening experience rather than a stressful one.

How should an architecture interview be handled?

The goal (for any interview) is to allow the candidate to shine, and to present themselves in the best light that they can. Even if you decide they’re not right for the role or the organisation, you want them to walk away feeling good about themselves and what they did that day.

To hopefully achieve that goal, we want the candidate to feel relaxed and that they have everything they need before the interview begins (do they need a toilet break? do they need a glass of water? make sure you ask and that they’re comfortable).

Next, tell them the agenda. Just as an example it might go something like:

“We have 90mins and then you move to your next interview. So I reckon we should spend five minutes introducing ourselves and what we do here (the teams we work in etc), and then the next hour chatting about the problem to be solved (or the architecture you’re going to present), and then after that it’s up to you: we could either use up the rest of the time by having you ask us questions – anything you like, or we could do that for 15 minutes and the remaining time you can have back to relax and we can show you around the office a bit”.

Doesn’t have to be exactly like that, but basically you want the candidate to know what’s going on (no surprises or ambiguity). So, you might also want to make sure the candidate is ok with you taking notes (my memory is terrible and so much happens during an interview that you want to be able to go back over your notes to be sure you’ve taken everything into consideration).

As far as the actual ‘test’ portion of the architecture interview, remember that people solve problems in different ways to you so be open-minded when critiquing their design.

Lastly, and most importantly, when dealing with a software engineer (i.e. you’re giving them requirements to fulfill as part of the design), make sure you’re working with that person. Help guide them when it looks like they might be losing their way, you don’t have to outright tell them the answer, but you can ask leading questions that should otherwise help kickstart their thinking down a better path. You’d do this for any colleague you work with, so afford this person the same respect and kindness.

What are you looking for from a candidate?

Well, for me, it depends on the role. I’ve carried out quite a few ‘Engineering Manager’ interviews recently and so they’re the most prominent in my mind so I’ll use that as my measuring stick (but just remember, none of this is set in stone and is open for interpretation and tweaking/evolving).

So below are the high-level perspectives I’m looking for, and taking notes on. But realise you won’t be able to incorporate all of these into your single architecture interview. It wouldn’t be practical, nor would it make sense to do that (especially if you have multiple interview formats for this candidate). These topics are more generally applicable.

Communication:
- are they ‘remote aware’? I’m generally interviewing remotely, so when there are other interviewers in the room (in real life), it’s often the case I’m not interacted with in the same way as others.
- do they use gender appropriate language?
- do they communicate clearly and circle back to questions they forgot to answer?
System:
- are they able to describe the history of the project, its reason for existing, the value it’s supposed to offer, the teams and other comms involved?
Architecture:
- is it a good design? fundamental but just be aware that in most cases this becomes a subjective opinion, and so unless there are some horrific consequences to the design, then it’s not actually something to worry too deeply about.
Risk Management:
- Is the system fault tolerant? What scenarios were considered?
- Were risks identified (if so what were they)?
- How were those risks managed/mitigated?
- Was the risk monitored?
Observability:
- What telemetry system was used to gather data?
- What types of instrumentation is in places (logging, metrics, both or other)?
- What monitoring tools are in place?
- See also this post about monitoring best practices.
- How is on-call handled (is there a culture of responsibility or is there a centralized operations team looking after things)?
Strategy:
- What teams were consulted/impacted?
- Who were the stake holders, and what were their involvement?
- How was the system proposed and communicated?
Distribution:
- Is the candidate used to working within a large multi-region organisation where teams are distributed across many different time zones (are they used to those types of challenges)?
Organisation:
- What is the structure and hierarchy of their current employer? Is it fairly flat or lots of layers and red tape? Maybe it’s “goldilocks” (i.e. just right)?
- Has there been any restructuring of the teams (why), and how was it communicated, how was the changes received by staff?
Awareness:
- Was the candidate actively thinking about things outside of just the technical aspect? Were they thinking about, for example, the costs associated with the architecture they were designing and how to reduce those costs whilst trying to still solve the problem at hand?
Team Management:
- How does the candidate handle 1:1 meetings, build relationships and give feedback?
- How big is the team(s) they manage, what conflicts have they needed to resolve, and how?
Organisation Feedback/Inclusion:
- Does their current employer provide staff the means to give them feedback and to understand the health of their organisation?
- If so what are those tools and what has their effectiveness and impact been?
Diversity:
- Is the candidate thinking about diversity and how to improve it in the hiring process?
- What tools do they use to help with that goal?
- Are there any community programs they support or host?
Interest:
- Is the candidate excited to work for/with us? What about our organisation do they like the most? (this isn’t a deal breaker, this is more out of interest)

Anything else?

I don’t know, but maybe you could let me know on twitter if there’s anything I’ve missed or should be doing differently. Feedback appreciated.

Post Mortems

Thu, 09 Aug 2018 00:00:00 +0000

ℹ️ NOTE

for those short on time: here’s the Post-Mortem Template

What is a post-mortem?

When you have a service outage, or any form of unexpected service disruption, you first resolve the issue and then proceed to discuss what happened, why and how.

The process of discussion is referred to as a ‘post-mortem’.

How do we make post-mortems effective?

You shouldn’t just get a bunch of people together in a room to discuss an incident. You need to:

Schedule the meeting at the right time.
Invite the right people.
Have the right attitude.
Talk about the right things.
Know the right follow-up actions to take.
Notify the right people.

Let’s go over each of these items briefly, they might sound obvious but I’ve seen each of them carried out in vastly different ways (hint: not all of them good)…

Schedule the meeting at the right time

You should schedule the post-mortem to occur as soon as possible after the event. DO NOT have this discussion two weeks after the incident because people will forget important details. Have this discussion while everything is still fresh in people’s minds.

The trouble with getting a post-mortem scheduled at the right time is that you’ll need a prepared document (a post-mortem template) that is filled in with as much detail (as can be reasonably recalled from memory †) almost immediately after the incident. This document doesn’t just magically appear, it’s someones responsibility to create it.

† this is harder than you imagine, even with modern day communication tools like Slack, there are so many things going on at once (multiple people chatting and trying to isolate the problem, then finding a quick and safe solution) that it’s easy to forget things or not notice them happening (hence a post-mortem helps bring all this information together).

You should have the post-mortem document filled in (as much as you can) before having the post-mortem as a way to help drive the conversation, and (ideally) whoever was on-call at the time of the incident would be marked as the point-person for prepping this post-mortem document.

One last useful point is to make sure all those who need to be involved in the post-mortem are reminded of the meeting a day before. This allows them to read through the post-mortem document and prep accordingly (and generally makes for a smoother discussion).

Invite the right people

This is a tricky one to get right, and will vary depending on your organisational structure (and even how the incident itself unfolded).

You shouldn’t invite only those people involved during the incident because there are insights that can be gained from Product Managers (just as an example) who might not have been online at the time of the incident, and who might be able to elucidate certain aspects that the engineers/support team were not aware of.

You also don’t necessarily want to invite too many people outside of those involved in the incident. As the old saying goes: “too many cooks spoil the broth”. You could find the ‘noise to signal’ ratio goes up.

Have the right attitude

Post-mortems should be blameless. Do not come into the meeting with an ‘axe to grind’. People make mistakes, we’re dealing with software and often-times complex distributed systems so we should have an attitude of support, understanding, patience and a willingness to want to genuinely improve things.

Talk about the right things

This is where the post-mortem ‘template’ comes in (I’ve linked to it at the top and bottom of this post), as it includes different topics that can help steer the conversation in the right direction.

Things like: observations, symptoms, hypothesis, remediation, impact etc. You should also identify who the meeting moderator is (the person responsible for keeping the meeting on track and not falling into a war of words) and also who the notetaker is (they can’t be the same person).

It’s also probably worth mentioning that you shouldn’t be having a 3 hour meeting (of any kind, let alone a post-mortem), so make sure the short time you have together is yielding the right feedback and information.

Know the right follow-up actions to take

Have real actionable tasks as takeways from the post-mortem. You don’t want to just discuss how things could be improved, you want to actually improve them. It’s very important you take this opportunity to identity things you can do to prevent this issue from occuring again.

If service downtime/disruption is important to your business (and let’s face it, when is it not), then you need to take incidents seriously and put the time and effort into ensuring stability for the future.

Once you have the post-mortem document filled in fully, reviewed and all takeway tasks have been actioned (or at least scheduled to be actioned in the near future), then you can finish up this whole process by sharing what you learnt with your colleagues who were not involved (which leads us onto the next point).

Notify the right people

We generally take the finished post-mortem document and email it around to our development mailing list so that all engineers in the organsation get to see what happened, why, and how it was resolved.

This is super important because there are so many benefits that can come from this sharing of knowledge. Probably top of that list would be that it supports the notion that your organisation respects ‘accountability’ and that it’s honest about mistakes that are made.

It highlights to others (not just tech and engineering) that these mistakes aren’t punished but treated as opportunities for growth and learning. As well as exposing engineers (of varying skill levels) to different architectures, systems and services that are in place and helps them not only understand them a little better but encourages them to investigate more.

Because of all that (and more), who you share the post-mortem document with really does depend on you and your company’s values/standards.

A template for success

Below I link to a Google document we use as a template for our post mortem meetings, and I’ve included the relevant sections below just for an ‘at a glance’ view.

ℹ️ NOTE

you don’t have to include all of these if you don’t want. Take what is useful to you and leave whatever isn’t.

Observations

What empircal things did people see?

e.g. System X was Y, which indicated Z.

Symptoms

What was the physical outcome (the user experience) of these observations?

e.g. Users were seeing X output.

Hypothesis

What do we believe to be the contributing factors?

ℹ️ NOTE

there’s typically never a single ‘root cause’

Remediation

What did we do to resolve the incident?

ℹ️ NOTE

this isn’t the same as ‘fixing’ the problem, which suggests something more long term was put in place.

e.g.

We did X to resolve the problem.
We also did Y to resolve the problem.

Impact

What was the ‘cost’ of this incident?

ℹ️ NOTE

this isn’t the same thing as the symptoms.

e.g.

Thing A broke for N hours.
Thing B broke and all entered data for User C, during the incident, was lost.

Key Points

There’s lots of interactions during the incident, what were the most important things that happened?

e.g.

Person A identified incident at N time.
Person B notified stakeholders at N time.
Person C ramped up resources at N time.

Participants

Who was involved and what are their roles in the company.

ℹ️ NOTE

you don’t know everyone and what they do, so make it easier for people reading the post-mortem to understand the breadth of skills involved.

e.g.

Jane Doe (Engineer)
Joe Bloggs (Support Staff)

Timeline

Describe what happened and when. This should be much more detail than those extracted for the ‘key points’ section.

ℹ️ NOTE

if you work for a distributed company, then identifying the timezone of the incident (or at least the timezone you’re reporting it from) can help clarify when these things happened.

e.g.

Timezone: BST 2018-12-22 (Saturday)

03:00 - Joan said a thing happened.
03:10 – Joe notified the stakeholders.

Additional Details

Not everything said is going to have happened within the incident channel you were looking at during the incident (maybe it happened in an email thread between support staff, product and other stakeholders). This is something that is likely to be fleshed out during the post-mortem itself as more people who were having those conversations give their perspective.

e.g.

Bilbo Baggins (Product Manager): said something useful that not everyone would have seen

Communication

Where was the ‘war room’ (i.e. the incident channel where everyone was gathered to help problem solve)?

e.g.

#some-slack-channel
Link to Logs
Link to Dashboards

Images

If you have any screen shots of the broken system, then that’s useful for people not involved to understand the impact more visually. But also, linking to a specific point in time of a graph that shows a spike (and hopefully dip later) in errors can also be useful to reference back to.

Tasks

Put together a list of tasks for people to complete. Doesn’t have to be done at the time of sending out the post-mortem document to the wider organisation, but ideally you’d have those things done before you sent it out.

Jane: responsible for doing this task.
Another task that has no specific person assigned to it.
Confirm to Bilbo that we’ve done everything we can.

Questions

During the post-mortem questions will be raised and they should be placed here. You’re not necessarily going to have all the answers during that meeting.

Ideally you’d have answers tied to this questions before you sent out the post-mortem document to the wider organisation though.

What we did well

A list of things the team did well during the incident.

e.g.

The incident was identified quickly
The solution was quickly rolled out
No one panicked
Cross team communication was exceptional

What we didn’t do so well

A list of things the team didn’t do so well during the incident.

e.g.

It took a long time to understand what the alarm meant and who was affected
The on-call person didn’t acknowledge the initial alarm and so it kept firing
We didn’t setup an incident slack channel so conversations were happening everywhere

What can we learn from in future?

A list of the things we should pro-actively improve upon. Generally this will be resolutions to the things that didn’t go well.

Template Document

View the Post-Mortem Template

Conclusion

Let me know what you think on twitter. If you have any improvements or you think I have things totally wrong, then I’d love to hear about it.

OpsBot: Operations Slackbot

Tue, 07 Aug 2018 00:00:00 +0000

So this post is a long time coming. It has been pushed to the forefront by the fact that BuzzFeed recently held its annual ‘Hack Week’, and riding on the wind of a massive success I’ve had with my hack project this past week…

“Mark has casually saved the company $60k a year by letting us break away from a commercial dependency without losing any features!” – BuzzFeed UK Newsletter

…I’ve decided to revisit last years hack week project: OpsBot, which is a Slackbot for operational tasks.

ℹ️ NOTE

for those interested, the presentation slides for my 2018 hack can be found here.

What does OpsBot do?

Creates standardized incident channels.
Auto-invite users to incident channel based on emoji reactions.
Looks up service runbooks.

How do you use OpsBot?

/incident <name> [visibility] [reporter]
/runbook <service>

Let’s break down the arguments provided to each command:

name (required):
hyphenated name of the incident, e.g. Service-Foo-5xx
visibility (optional):
whether the channel should be public or private
reporter (optional; has a default channel):
name of channel where incident was first reported
service (required):
searches our company Google Drive for specified runbook

Incident Example

Imagine I’ve noticed something bad has happened. I’ll go to the appropriate Slack channel and report it.

At that point people who are interested in the incident will use an 👀 emoji (we support various types) to indicate their interest in being auto-invited to a new incident channel (if one is to be created).

The following image demonstrates what that might look like…

ℹ️ NOTE

we have automated notifications sent to specific monitoring slack channels, and so people can also use the emoji reaction on those messages.

Now imagine this incident has been triaged and yes it is indeed a problem. We’ll need to spin up an incident channel so we can focus discussions and get a resolution in place.

It’s at that point someone runs the command: /incident bf4life (or whatever you want to name the incident 😉) and we’ll see the following…

Notice that we automatically prefix the given incident name with the current date, so it’s easier to go back and review/identify past incidents.

If we’re dealing with a service that we’re unfamiliar with, then we might also want to look at the runbook(†) for that service.

† a runbook is a compilation of routine procedures and operations that the system administrator or operator carries out – Wikipedia

This is when someone runs the command: /runbook site router (or whatever the affected service is) and we’ll see the following…

How does OpsBot work?

When creating an incident channel, OpsBot will link back to the channel that reported the incident as well as linking to the specific notification in that channel.

This works because OpsBot’s logic is trying to match one of two possible patterns within the message’s body:

The phrase ISSUE: begins the message (see example from earlier ☝️).
The message is identified as a automated NAGIOS CRITICAL notification.

If the message is identified as an ‘incident trigger’ then that’s what we link to within the incident channel…

What did we learn?

Have a pre-hack document with feature specifications in place.
Slackbots are fun to create.
The Google API was ~~kinda tricky~~ a PITA.

What could we improve?

Two issues cropped up fairly early on in the design:

Identify who not to auto-invite to a private incident.
Identify better ‘nagios incident message’ grouping logic.

The first issue is an awkward one because we can’t necessarily stop people from accessing a private incident channel if they’ve gone into a public slack channel and used the emoji reaction on a particular incident.

That being said, we very rarely have to create private incident channels. That’s only when an incident relates to some serious security vulnerability, and nearly all the time those types of issues are raised via HackerOne and dealt with outside of typical monitoring notifications (like 5xx’s).

The second incident occurs when people add an emoji to a NAGIOS automated notification but then a repeat notification message is sent later. In this scenario a message pings to say there is an issue, people add the emoji reaction but later on (before an incident channel has been created) an updated NAGIOS error notification is sent.

Now when we create the incident channel, it’ll look for the first NAGIOS warning that matches and it’ll find no emoji reactions, so it won’t auto-invite people to the incident channel.

Although this isn’t the end of the world as people can still see the incident channel link generated (if it’s a public channel) and click on that to access the incident channel.

What else could OpsBot do?

OpsBot has the potential to do lots of things, it just depends on your needs and use cases.

For us, a few things we planned to do but never quite got round to was…

A /postmortem command for automatically creating our incident ‘Post Mortem’ documents.
A /service command for looking up the on-call team leads for that service(†).
Add runbook info to our deployment platform so OpsBot can pull it in auto-magically.
A damn good refactor and some tests! (this was a ‘hack’ after all).

† this might materialise into another project we have for improving team relations called #WhoWhatWhy (coming soon).

So that’s OpsBot. If you’re interested, the presentation slides for the OpsBot hack can be found here

Thinking about Interfaces in Go

Sat, 21 Jul 2018 00:00:00 +0000

This post is going to explain the importance of interfaces, and the concept of programming to abstractions (using the Go programming language), by way of a simple example.

While treading what might seem like familiar ground to some readers, this is a fundamental skill to understand because it enables you to design more flexible and maintable services.

Interfaces in Go

An ‘interface’ is a contract which describes behaviour (not data).

Andrei Boar said…

When defining interfaces in Go, you don’t define what something is but what it provides — behavior, not things! That’s why there’s no File interface in Go, but a Reader and a Writer: these are behaviors, and File is a thing implementing Reader and Writer.

…which is important because this has a direct effect on the naming of an interface. You name interfaces with an -er at the end to indicate it’s a verb (i.e. this thing does something).

In Go an interface is defined like so:

type Fooer interface {
    Bar(s string) (string, error)
}

ℹ️ NOTE

In Go, a capitalised name (method, field etc) is public, lowercase is private.

If an object in your code implements a Bar function, with the exact same signature (e.g. accepts a string and returns either a string or an error), then that object is said to implement the Fooer interface.

An example of this would be:

type thing struct{}

func (l *thing) Bar(s string) (string, error) {
  ...
}

Now you can define a function that will accept that object, as long as it fulfils the Fooer interface, like so:

func doStuffWith(thing Foo)

This is different to other languages, where you have to explicitly assign an interface type to an object, like with Java:

class testClass implements Foo

Because of this flexibility in how interfaces are ‘applied’, it also means that an object could end up implementing multiple interfaces.

For example, imagine we have the following two interfaces:

type Fooer interface {
  Bar(s string) (string, error)
}

type Beeper interface {
  Beep(s string) (string, error)
}

We can define an object that fulfils both interfaces simply by implementing the functions they define:

type thing struct{}

func (l *thing) Bar(s string) (string, error) {
  ...
}

func (l *thing) Beep(s string) (string, error) {
  ...
}

ℹ️ NOTE

This is a bit of silly example, and so you’ll notice the method signature for each type is effectively the same. Be careful when designing your interfaces, because in this case we could possibly combine these two interfaces into a single (more generic) interface.

Name Your Interface Arguments

Consider the following interface:

type Mover interface {
  Move(context.Context, string, string) error
}

Do you know what the second and third arguments refer to and how the function will use them?

Now consider this refactored version where the arguments have names associated with them:

type Mover interface {
  Move(context.Context, source string, destination string) error
}

Now that is better, because we can clearly see what the expectations are: the second argument is the ‘source’ and the third argument is the ‘destination’.

Keep Interfaces Small

You’ll find in the Go Proverbs, the following useful tip:

The bigger the interface, the weaker the abstraction.

The reason for this is due to how interfaces are designed in Go and the fact that an object can potentially support multiple interfaces.

By making an interface too big, we reduce an object’s ability to support it. Consider the following example:

type FooBeeper interface {
  Bar(s string) (string, error)
  Beep(s string) (string, error)
}

type thing struct{}

func (l *thing) Bar(s string) (string, error) {
  ...
}

func (l *thing) Beep(s string) (string, error) {
  ...
}

type differentThing struct{}

func (l *differentThing) Bar(s string) (string, error) {
  ...
}

type anotherThing struct{}

func (l *anotherThing) Beep(s string) (string, error) {
  ...
}

In the above example we’ve defined a FooBeeper interface that requires two methods: Bar and Beep. Now if we look at the various objects we’ve defined thing, differentThing and anotherThing we’ll find:

thing: fulfils the FooBeeper interface
differentThing: does not fulfil the FooBeeper interface
anotherThing: does not fulfil the FooBeeper interface

Alternatively, if we were to break the FooBeeper interface up into separate smaller interfaces (like we demonstrated earlier), then in our above example, the differentThing and anotherThing would become more re-usable.

That’s ultimately what this Go proverb is suggesting: smaller interfaces allow for greater code reuse.

Accept Interfaces, Return Concrete Types

If your function accepts a concrete type then you’ve limited the consumers ability to provide different implementations.

Consider a function only accepting the concrete type *os.File instead of the io.Writer interface. Now try swapping out the os.File implementation in a test environment, you’ll have a hard time vs mocking this using a struct that has the relevant interface methods.

Unless there is a good reason to, you should return concrete types instead of interfaces. This is because an interface has a tendency to add an unnecessary layer of indirection for consumers of your package (although we’ll discover a few valid scenarios where returning an interface is more appropriate).

Below is an example of what I mean by indirection. We have a function foo that returns the interface Fooer, and yet we want to access a field on the underlying type of the interface (which we can see is a S struct type):

package main

import (
    "fmt"
    "log"
)

type Fooer interface {
    Bar()
}

type S struct {
    Debug bool
}

func (s S) Bar() {
    fmt.Println("bar called")
}

func foo() Fooer {
    return S{true}
}

func main() {
    f := foo()
    fmt.Printf("Type: %T\n", f) // main.S
    fmt.Printf("Representation: %+v\n", f) // {Debug:true}
    f.Bar()
    fmt.Println(f.Debug) // ERROR: f.Debug undefined (type Fooer has no field or method Debug)
}

We can see from the above code that we’re able to call the Bar method (as it’s part of the public interface) but we can’t access the Debug field, even though it’s declared as a public field.

So how can we access the Debug field? We need to use a type assertion to get access to the interface’s underlying value:

s, ok := f.(S)
if !ok {
  log.Fatal("couldn't coerce f to S")
}
fmt.Println(s.Debug) // true

This is the ‘indirection’ I was referring to, and is a tedious step for a consumer of this code. They wouldn’t need to do this if our foo function had returned the concrete S type.

Don’t Return Concrete Types

This is to keep you on your toes 😉

I want to highlight an important ‘design’ decision, which is: if your code returns a pointer to some data, then it means once that pointer has been passed around a few different functions, we now have multiple entities that are able to mutate that data.

So be careful about whether you return a value (immutable) vs a pointer (mutable) as it could help reduce confusion with regards to how the data is modified by your program.

Returning an interface in these cases could be an appropriate solution.

By this I mean: although you might return a pointer to a data structure, by defining an interface around the behaviours attached to that data structure, it means a caller of your function won’t be able to access the internal fields of the struct but it can call the methods defined in the returned interface!

Another example might be that your function needs to return a different type depending on a runtime condition (*cough* generics *cough*). If that’s the case, then returning an interface could again be an appropriate workaround to the lack of generics in the Go 1.x language.

The following code example highlights the principle:

type Itemer interface {
    GetItemValue() string
}

type Item struct {
    ID int
}

type URLItem struct {
    Item
    URL string
}

type TextItem struct {
    Item
    Text string
}

func (ui URLItem) GetItemValue(){
    return ui.URL
}

func (ti TextItem) GetItemValue(){
    return ti.Text
}

func FindItem(ID int) Itemer {
  // returns either a URLItem or a TextItem
}

The FindItem could be an internal library function that attempts to locate an item via multiple data sources. Depending on which data source the item was found, the type returned will change.

In this instance returning an interface allows the consumer to not have to worry about the change in underlying data types.

ℹ️ NOTE

It’s possible the returned types could be consolidated into a single generic type struct, which means we can avoid returning an interface, but it depends on the exact scenario/use case.

Use Existing Interfaces

It’s important to not ‘reinvent the wheel’ and to utilise existing interfaces wherever possible (otherwise you’ll suffer from a condition known as ‘interface pollution’).

The golang toolchain offers a tool called Go Guru which helps you to navigate Go code.

It’s a command line tool, but it’s designed to be utilised from within an editor (like Atom or Vim etc).

Here is a list of the sub commands available:

callees         show possible targets of selected function call
callers         show possible callers of selected function
callstack       show path from callgraph root to selected function
definition      show declaration of selected identifier
describe        describe selected syntax: definition, methods, etc
freevars        show free variables of selection
implements      show 'implements' relation for selected type or method
peers           show send/receive corresponding to selected channel op
pointsto        show variables the selected pointer may point to
referrers       show all refs to entity denoted by selected identifier
what            show basic information about the selected syntax node
whicherrs       show possible values of the selected error variable

This can be really useful for identifying (for example) whether a new interface you’ve defined is similar to an existing interface.

To demonstrate this, consider the following example…

// this is a duplicate of fmt.Stringer interface
type Stringer interface {
    String() string
}

type testthing struct{}

func (t testthing) String() string {
    return "a test thing"
}

The Stringer interface I’ve defined is actually a duplication of the existing standard library interface fmt.Stringer.

So using Guru via my Vim editor I can see (when I have my cursor over the testthing struct and I call Guru) that this concrete type implements not only stringit but a few other interfaces…

/main.go:33.6-33.14:                                                 struct type testthing
/usr/local/Cellar/go/1.10.3/libexec/src/fmt/print.go:62.6-62.13:     implements fmt.Stringer
/main.go:29.6-29.13:                                                 implements Stringer
/usr/local/Cellar/go/1.10.3/libexec/src/runtime/error.go:66.6-66.13: implements runtime.stringer

Now whether you continue to define a new interface is up to you. There are actually quite a few places in the Go standard library where interfaces are duplicated for (what I believe to be) semantic reasoning, but otherwise if you don’t need to make an explicit/semantic distinction, then I’d opt to reuse an existing interface.

ℹ️ NOTE

For more details on how to use Guru, see this gist.

Don’t Force Interfaces

If your code doesn’t require interfaces, then don’t use them.

No point making the design of your code more complicated for no reason. Consider the following code which returns an interface.

ℹ️ NOTE

The following example is modified from a much older post by William Kennedy.

package main

import "fmt"

// Server defines a contract for tcp servers.
type Server interface {
    Start()
}

type server struct{}

// NewServer returns an interface value of type Server
func NewServer() Server {
    return &server{}
}

// Start allows the server to begin to accept requests.
func (s *server) Start() {
    fmt.Println("start called")
}

func main() {
    s := NewServer()
    fmt.Printf("%+v (%T)\n", s, s)
    s.Start()
}

The use of an interface here is a bit pointless. We should instead just return a pointer to an exported version of the server struct because the user is gaining no benefits from an interface being returned by NewServer (see Don’t Return Concrete Types for a possible use case for returning interfaces, but the above example is not one of them).

⚠️ There is also an important performance consideration to using interfaces that is often neglected: method calls on an interface type will be using dynamic dispatch not static dispatch and in a code hot path that can be a problem because the memory associated with the call can escape to the heap (stack memory is much more efficient). Consider r had an interface type io.Reader. A call to r.Read(b) would result in both the value of r and the backing array of the b byte slice to be allocated onto the heap.

Embedding Interfaces

Sometimes a code base will define a very large interface. Now we can probably agree it’s not a good idea but let’s just accept that in the real-world this kind of thing happens.

One place where a large interface can cause problems is with testing.

If a function accepts an interface but in reality only uses one method from the interface, then you might find yourself getting frustrated at the idea of having to implement a mock version of each method!

Well, to avoid that situation try taking advantage of Go’s ability to embed an interface into a struct.

By embedding the interface into your struct, you automatically promote all of the methods to the embedding struct. Now, you can pass your mock struct to the function and the compiler will be happy. You now only need to implement the methods you need to assert the test scenario you’re trying to validate.

type Exampler interface {
    Foo() error
    Bar() error
    Baz() error
    // ...lots more...
}

func example(e Exampler) error {
  err := e.Foo()
  if err != nil {
    return err
  }

  // ...other stuff...

  return nil
}

type mock struct{
  Example // embedded interface
}

// We're only implementing one method, not all three!
func (m *mock) Foo() error {
  return errors.New("whoops")
}

func TestExample(t *testing.T) {
    example(&mock{}) // we expect the function to fail due to our mock behaviour
}

But be aware that because you’re not providing a concrete implementation of the interface when instantiating your struct, it means that the value of that embedded field will be nil. This means that if the function you pass your struct into calls any of the interface methods not implemented by your mock struct, then there will be a runtime ‘nil pointer dereference’ error (but I argue that’s a good thing because in a test environment you want to know if your code is calling something for real).

Upgrading Interfaces

If you use have an interface that’s used by lots of people, how do you add a new method to it without breaking their code? The moment you add a new method to the interface, their existing code that handles the concrete implementation will fail.

Unfortunately there isn’t a completely clean solution to this problem. In essence the original interface needs to stay untouched and we need to define a new interface that contains the new behaviour. Then the consumer’s of an interface will continue to reference the original interface while using a type assertion within their functions for the new interface.

Below is an example of this problem in action:

package main

import "fmt"

type Fooer interface {
    bar() string
    baz() string // new method added, which breaks the code
}

func doThing(f Fooer) {
    fmt.Println("bar:", f.bar())
}

type point struct {
    X, Y int
}

func (p point) bar() string {
    return fmt.Sprintf("p=%d, y=%d", p.X, p.Y)
}

func main() {
    var pt point
    pt.X = 1
    pt.Y = 2

    doThing(pt)
}

In the above code we can see we have added a new method baz to our Fooer interface which means the concrete implementation pt is no longer satisfying the Fooer interface as it has no baz method.

ℹ️ NOTE

I appreciate the example is a bit silly because we could just update the code to support the new interface, but we have to imagine a world where your interface is provided as part of a public package that is consumed by lots of users.

To solve this problem we need an intermediate interface. The following example demonstrates the process. The steps are…

define a new interface containing the new method
add the method to the concrete type implementation
document the new interface and ask your interface consumers to type assert for it

package main

import "fmt"

type Fooer interface {
    bar() string
}

type Newfooer interface {
    baz() string
}

// We want a `Fooer` interface type, but if that valid type can also do the new
// behaviour, then we'll execute that behaviour...

func doThing(f Fooer) {
    if nf, ok := f.(Newfooer); ok {
        fmt.Println("baz:", nf.baz())
    }
    fmt.Println("bar:", f.bar())
}

// Original concrete implementation...

type point struct {
    X, Y int
}

func (p point) bar() string {
    return fmt.Sprintf("p=%d, y=%d", p.X, p.Y)
}

// New concrete implementation of `point` struct (has the new method)...

type newpoint struct {
    point
}

func (np newpoint) baz() string {
    return fmt.Sprintf("np !!! %d, ny !!! %d", np.X, np.Y)
}

func main() {
    var pt point
    pt.X = 1
    pt.Y = 2

    doThing(pt)

    var npt newpoint
    npt.X = 3
    npt.Y = 4

    doThing(npt)
}

ℹ️ NOTE

Again, the example is a bit silly in that we’re handling everything within a single file, whereas in reality the consumer won’t have access to the original interface/implementation code like we do here (so just use your imagination 🙂).

The output of the above code is as follows:

bar: p=1, y=2
baz: np !!! 3, ny !!! 4
bar: p=3, y=4

So we can see we called doThing and passed a concrete type that satisfied the Fooer interface and so that function called the bar method it was expecting to exist. Next we called doThing again but passed a different concrete type that not only satisfied the Fooer interface, but the Newfooer interface and within doThing we type assert that the object passed in is not only a Fooer but a Newfooer.

What would this look like in practice then? Well, if the Go standard library wanted to add a new method to the existing (and very popular) net/http package ResponseWriter interface: they would create a new interface with just the new behaviour defined, then they would document its existence and in that documentation they would explain that if your HTTP handler required the new behaviour, then you should type assert for it.

Imagine if the go standard library just updated the ResponseWriter with the new method? Lots and lots of existing HTTP server code would break as the concrete implementation that was passed through would not support that implementation.

In fact this is exactly what the go standard library authors have done with the Flusher and Hijacker interfaces. The following code demonstrates the use of a type assertion to access the additional behaviour defined by those interfaces:

func(w http.ResponseWriter, r *http.Request) {
        io.WriteString(w, "This will arrive before... ")

        if fl, ok := w.(http.Flusher); ok {
                fl.Flush()
                time.Sleep(1 * time.Second)
        }

        io.WriteString(w, "...this bit does.")
}

Standard Library Interfaces

Imagine we have a function process, whose responsibility is to make a HTTP request and do something with the response data:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func process(n int) (string, error) {
    url := fmt.Sprintf("http://httpbin.org/links/%d/0", n)

    resp, err := http.Get(url)
    if err != nil {
        fmt.Printf("url get error: %s\n", err)
        return "", err
    }

    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("body read error: %s\n", err)
        return "", err
    }

    return string(body), nil
}

func main() {
    data, err := process(5)
    if err != nil {
        fmt.Printf("\ndata processing error: %s\n", err)
        return
    }
    fmt.Printf("Success: %v", data)
}

We can see our process function accepts an integer, which is interpolated into the URL that is requested. We then use the http.Get function from the net/http package to request the URL.

The function then stringify’s the response body and returns it. This is sufficient for a basic example, but in the real world this function would likely do lots more processing to the response data.

It may not be immediately obvious but there are already many instances where interfaces are being utilised. Let’s break down the code and see what interfaces there are.

The http.Get function returns a pointer to a http.Response struct, and from within that struct we extract the Body field and pass it to ioutil.ReadAll.

The Body field’s ‘type’ is set to the io.ReadCloser interface. If we look at that interface we’ll see it’s made up of nested interface types:

type ReadCloser interface {
    Reader
    Closer
}

If we now look at the io.Reader and io.Closer interfaces, we’ll find:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Closer interface {
    Close() error
}

This means that for the response body object to be valid, it must support the Read and Close functions defined by these interfaces (the returned object will likely include other functions, but it needs Read and Close at a minimum).

The next thing that happens in the code is that we pass http.Response.Body to an input/output function called ioutil.ReadAll.

If we look at the signature of ioutil.ReadAll we’ll see that it accepts a type of io.Reader, which we’ve seen already, and so this is another indication of why smaller interfaces enable re-usability.

What the io.Reader interface means for our code is that the input we provide to ioutil.ReadAll must support a Read function, and (because http.Response.Body implements the io.ReadCloser interface) we know it does implement that required function.

So already we’ve seen quite a few built-in interfaces being utilised to support the standard library code we’re using. More importantly, you’ll find the use of these interfaces (io.ReadCloser, io.Reader, io.Closer and others) are used everywhere in the Go codebase (highlighting again how small interfaces enable greater code re-usability).

Tight Coupling

Now there’s an issue with the above code, specifically the process function, and that is we’ve tightly coupled the net/http package to the function.

What this means is that the process function has to intrinsically know about HTTP and dealing with the various methods available to that package.

Also, if we want to test this function we’re going to have a harder time because the http.Get call would need to be mocked somehow. We don’t want our test suite to have to rely on a stable network connection or the fact that the endpoint being requested might be down for maintenance.

The solution to this problem is to invert the responsibility of the process function, also known as ‘dependency injection’. This is the basis of one of the S.O.L.I.D principles: ‘inversion of control’.

Dependency Injection

If we call a function, then it is our responsibility to provide it with all the things it needs in order to do its job.

In the case of our process function, it needs to be able to acquire data from somewhere (that could be a file, it could be a remote procedure call, it shouldn’t matter). The most important aspect to consider is how it acquires that data.

The how is not the responsibility of the process function, especially if we decide later on that we want to change the implementation from HTTP to GRPC or some other data source.

Meaning, we need to provide that functionality to the process function. Let’s see what this might look like in practice:

ℹ️ NOTE

This is just a first iteration, and is a poor design because although it shifts the problem slightly, there will still be tight coupling. I’ll come back to this code later and refactor away the coupling completely. The reason I’ve not done that upfront is because there are learnings to be had from trying to write tests for this code (which we’ll see in a minute).

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

type Getter interface {
    Get(url string) (*http.Response, error)
}

type httpbin struct{}

func (l *httpbin) Get(url string) (*http.Response, error) {
    resp, err := http.Get(url)
    if err != nil {
        fmt.Printf("url get error: %s\n", err)
        return &http.Response{}, err
    }

    return resp, nil
}

func process(n int, g Getter) (string, error) {
    url := fmt.Sprintf("http:/httpbin.org/links/%d/0", n)

    resp, err := g.Get(url)
    if err != nil {
        fmt.Printf("data source get error: %s\n", err)
        return "", err
    }

    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("body read error: %s\n", err)
        return "", err
    }

    return string(body), nil
}

func main() {
    data, err := process(5, &httpbin{})
    if err != nil {
        fmt.Printf("\ndata processing error: %s\n", err)
        return
    }
    fmt.Printf("\nSuccess: %v\n", data)
}

Refactoring Considerations

Let’s start by looking at the interface we’ve defined:

type Getter interface {
    Get(url string) (*http.Response, error)
}

We’ve not been overly explicit when naming this interface Getter. Its name is quite generic on purpose so as not to imply an underlying implementation bias.

Unfortunately the defined Get method is still too tightly coupled to a specific implementation (i.e. it specifies http.Response as a return type).

Meaning, that although the refactored code is better, it is far from perfect.

Next we define our own object for handling the implementation of the Get method, which internally is going to use http.Get to acquire the data:

type httpbin struct{}

func (l *httpbin) Get(url string) (*http.Response, error) {
    resp, err := http.Get(url)
    if err != nil {
        fmt.Printf("url get error: %s\n", err)
        return &http.Response{}, err
    }

    return resp, nil
}

By using this interface as the accepted type in the process function signature, we’re going to be able to decouple the function from having to acquire the data, and thus allow testing to become much easier (as we’ll see shortly), but the process function is still fundamentally coupled to HTTP as the underlying transport mechanism.

The reason this is a problem is because the process function still knows that the returned object is a http.Response because it has to reference the Body field of the response, which isn’t defined on the object we’ve injected (meaning the function intrinsically knows of its existence).

How far you take your interface design is up to you. You don’t necessarily have to solve all possible concerns at once (unless there really is a need to do so).

Meaning, this refactor could be considered ‘good enough’ for your use cases. Alternatively your values and standards may differ, and so you need to consider your options for how you might what to design this solution in such a way that it would allow the code to not be so reliant on HTTP as the transport mechanism.

ℹ️ NOTE

We’ll revisit this code later and consider another refactor that will help clean up this first pass of code decoupling.

But first, let’s look at how we might want to test this initial code refactor (as testing this code allows us to learn some interesting things when it comes to needing to mock interfaces).

Testing

Below is a simple test suite that demonstrates how we’re now able to construct our own object, with a stubbed response, and pass that to the process function:

package main

import (
    "bytes"
    "io/ioutil"
    "net/http"
    "testing"
)

type fakeHTTPBin struct{}

func (l *fakeHTTPBin) Get(url string) (*http.Response, error) {
    body := "Hello World"

    resp := &http.Response{
        Body:          ioutil.NopCloser(bytes.NewBufferString(body)),
        ContentLength: int64(len(body)),
        StatusCode:    http.StatusOK,
        Request:       &http.Request{},
    }

    return resp, nil
}

func TestBasics(t *testing.T) {
    expect := "Hello World"
    actual, _ := process(5, &fakeHTTPBin{})

    if actual != expect {
        t.Errorf("expected %s, actual %s", expect, actual)
    }
}

Much like we do in the real implementation, we define a struct (in this case we’ve named it more explicitly) fakeHTTPBin.

The difference now, and what allows us to test our code is that we’re manually creating a http.Response object with dummy data.

One part of this code that requires some extra explanation would be the value assigned to the response Body field:

ioutil.NopCloser(bytes.NewBufferString(body))

If we remember from earlier:

The Body field’s ‘type’ is set to the io.ReadCloser interface.

This means when mocking the Body value we need to return something that has both a Read and Close method. So we’ve used ioutil.NopCloser which, if we look at its signature, we see returns an io.ReadCloser interface:

func NopCloser(r io.Reader) io.ReadCloser

The io.ReadCloser interface is exactly what we need (as that interface indicates the returned concrete type will indeed implement the required Read and Close methods).

But to use it we need to provide the NopCloser function something that supports the io.Reader interface.

If we were to provide a simple string like "Hello World", then this wouldn’t implement the required interface. So we wrap the string in a call to bytes.NewBufferString.

The reason we do this is because the returned type is something that supports the io.Reader interface we need.

But that might not be immediately obvious when looking at the signature for bytes.NewBufferString:

func NewBufferString(s string) *Buffer

So yes it accepts a string, but we want an io.Reader as the return type, whereas this function returns a pointer to a Buffer type?

If we look at the implementation of Buffer though, we will see that it does actually implement the required Read function necessary to support the io.Reader interface.

Great! Our test can now call the process function and process the mocked dependency and the code/test works as intended.

ℹ️ NOTE

Yes, we should probably use something more obvious and replace bytes.NewBufferString with something like bytes.NewReader, strings.NewReader.

Conclusion

Is there more we can do to improve this code’s design? Sure. But we’ll leave a new refactor iteration to another post.

Hopefully this has given you a feeling for how interfaces are used in the Go standard library and how you might utilise custom interfaces yourself.

Multigrain Services

Thu, 19 Jul 2018 00:00:00 +0000

It seems most people are either talking about “monolith applications” or “micro services”. But recently I’ve noticed yet another new buzzword bubbling up across the internet: “mini services”.

It’s worth mentioning that this isn’t a post about which pattern is the most appropriate to use, as they’re all appropriate to use, when applied in the right context and scenario.

Below is an image that visualises the concept, and makes understanding these various patterns much easier (at a glance):

source: thenewstack.io

A rose by any other name

To me “mini services” isn’t anything new, nor do I think it necessarily needs a new term to be applied to it. For me “mini services” are effectively just SOA (Service Oriented Architecture), but done right.

People complain about SOA and say it has lots of pain points that make it unusable. I disagree. From what I’ve read it simply comes down to designing your SOA correctly so that it has proper fault tolerance and resiliency. Not just throwing together some code, seeing certain dependencies break and then crying that SOA is rubbish 🤦‍♂️

Although maybe a new services lexicon † is useful, at least for giving a consistent and modern terminology to these concepts we collectively understand.

† the vocabulary of a person, language, or branch of knowledge.

Definitions

Let’s breakdown these concepts along with their old/new terminology:

Macro Service (Monolith Application): the traditional architecture model/system design. One big application handling multiple responsibilities and all of them plugged into a single data store.
Mini Service (SOA): an evolution of the monolith where domain ‘boundaries’ were more clearly defined and made into separate services and data stores. Although these services would still handle multiple responsibilities, those responsibilties were at least related.
Micro Service: we breakdown the domain boundaries even further so that services/data stores have very narrow and specific responsibilties. We’ve moved to ‘feature driven’ services.
Nano Service: we go to the extremes of boundary definitions and are working at the ‘component’ level (BBC explain their approach to this here).

Micro services are themselves a new’ish concept at ~7 years of age (at the time of writing). The term was first used in 2011.

Real micro services?

Martin Fowler famously defined microservices as having a very specific design application and that most “micro services” aren’t conforming to that. Specifically he says that they should:

Have a single responsibility
Be loosely coupled
Be independently deployable and scalable

But according to Ross Garrett (VP of Marketing Cloud Elements): “an HTTP-build service has to know more about what’s going on around it in order to communicate”.

So how do we achieve communication without relying on tried and tested HTTP/REST APIs etc? The answer is the ‘observer’ design pattern (or pub/sub - publisher, subscriber model).

With the pub/sub model, the micro service publishes notifications whenever something happens and services can subscribe to those notifications. This means the micro service has no awareness of anything outside of itself, it just throws out a notification and there’s either someone there to catch it and process it, or there isn’t.

Queue systems like RabbitMQ or SQS etc are used heavily in these cases to allow for that decoupling.

What do you use?

I tend to gravitate towards the SOA pattern † as it’s a nice balance between loose coupling and separation of domain boundaries without descending into the co-ordination and infrastructure madness of micro services.

† or “mini services” if that’s the language you prefer, although I personally don’t think I’ll ever be able to bring myself to use it over “SOA”.

Authentication with AWS Cognito

Fri, 15 Jun 2018 00:00:00 +0000

Introduction

In this post I would like to introduce you to the AWS Cognito service, and to explain its various moving pieces and how they fit together.

If you’re interested in a very high-level view of what I was working on, then this architecture diagram should give you the basic idea.

Effectively I co-designed and implemented a new authentication system (using AWS Cognito) for BuzzFeed’s existing community users to utilize and which opened the doors for new BuzzFeed services to also be able to offer additional features built upon authentication to their users.

Cognito is tricky to get up and running with (for a variety of reasons which I’ll explain as we go), and to make things worse there aren’t many reference points outside of the official documentation to help you. Hence this blog post now exists for those weary travellers looking for answers.

ℹ️ NOTE

this post was written approximately five months into a year long project and so a lot has changed in the design of the system and the implementation. But this post is still very relevant and useful for those looking to understand Cognito. This post also was fed back to various internal AWS teams and has resulted in work being carried out to improve various aspects of their services mentioned here.

Let’s start at the beginning…

What is Cognito?

According to the official blurb…

Amazon Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily.

In essence, Cognito provides features that let you authenticate access to your services, while also providing features to let you authorize access to your AWS resources.

Authentication vs Authorization

It’s important to clarify that in this blog post we’re only really discussing authentication, and not authorization. They are two different concepts.

Authentication is the process of verification that an individual, entity or website is who it claims to be.
Authorization is the function of specifying access rights to resources, which is different to (and commonly confused with) the process of authentication.

ℹ️ NOTE

if you’re new to these types of security concepts, then take a look at this glossary document I put together which covers the various terminology.

User Pools vs Identity Pools

In order for you to be able to authenticate and authorize access, Cognito provides two separate services:

User Pools
Identity Pools

User Pools deal with ‘authentication’, whereas Identity Pools deal with ‘authorization’ (and specifically that means AWS based resources only).

For the purposes of this post I’ll only be focusing in on User Pools, as our project requirements did not involve authorizing access for AWS resources to an authenticated user (which is where Identity Pools would typically come into play).

Now, with that said (and what makes this oh the more confusing, due to the design of the mobile SDKs), mobile applications do utilize Identity Pools for authentication, but the Identity Pool would be configured with a ‘provider’ which happened to be our User Pool.

If you’re interested in the various Identity Pool concepts, then please refer to the official documentation.

Implementation Options

There are fundamentally three options available for implementing User Pools:

Client SDK
Server SDK
AWS Hosted UI

Client SDK

The client SDK has a bit of a jagged history, which makes reading the AWS docs a bit confusing at times (or indeed when Googling for help), as you may notice references to ‘Amazon Cognito Identity SDK for JavaScript’ which is now a deprecated library.

What you’ll want to use instead is their new ‘Amplify’ SDK, which you’ll also find AWS has a strong bias towards (or at least their ‘solution architects’ push it really hard).

ℹ️ NOTE

I get the feeling AWS put a lot more time into Amplify and having it be able to abstract away a lot of the Cognito complexity, that they’re keen for consumers to utilise it.

Based on this I decided I would trust their opinion and just try and spin up something that works using Amplify, which unfortunately took a long time and ultimately I ended up dropping the work in favour of a server-side solution.

I don’t keep up with the constant changes to the JavaScript landscape, and so I’m not familiar with React (or Angular) which were the two examples the AWS docs (and most example repos) used the majority of the time. So using Amplify required me to first do some reading up on React, Babel, WebPack and a whole host of other tools. It was painful.

In the end we just had too much trouble trying to deal with Node and the various build systems that we decided to drop the work we had done and pivot to a new solution (see next section).

Server SDK

The server-side solution we chose was to use the Python SDK.

This ended up being a bit of a double edged sword. We were happier with the move to Python, but we really struggled with both the AWS documentation and also the boto3 library documentation that the Python SDK is built upon.

In order to get up and running we initially opted to use a 3rd party abstraction library called ‘Warrant’, which also incidentally helped us to understand the AWS documentation because we were able to reverse-engineer the Warrant code to better understand the boto3 API calls that needed to be made.

ℹ️ NOTE

I think that says a lot about AWS documentation. If people need to read through how an abstraction library is using your API, then your documentation must be pretty bad. I would be the first to suggest maybe I’m just too dumb to understand Cognito, but a lot of people across the internet were having the same problems.

Ultimately, Warrant didn’t provide all the functionality we needed and so we eventually refactored out Warrant and were back to using the underlying boto3 Python SDK.

It’s worth me taking a moment to also explain that some APIs require you to define a specific type of ‘authentication flow’ which is a security feature, and as far as I understand it, is supposed to help you to more safely access data provided by these APIs.

What I didn’t know originally, and was one of the reasons we decided to use a library such as Warrant, was that the code involved with some of these auth flows can be quite complex (I still now struggle to follow exactly what the code does within Warrant when it uses one of these ‘flows’).

Just to give you an example of the type of code AWS Cognito would expect you to write, take a look at the InitiateAuth API call with the USER_SRP_AUTH auth flow.

First of all I don’t think it’s very clear what is expected to be provided in that documentation alone, but also, take a look at Warrant’s implementation and specifically how to generate an SRP_A, which also doesn’t appear to be explained anywhere (no where obvious at least).

ℹ️ NOTE

it wasn’t until much later, we discovered that we could (in the case of InitiateAuth at least) have avoided writing all the SRP generation code and instead used the admin version of that API, called AdminInitiateAuth which allows you to skip SRP in favour of implicitly trusting the caller – which was fine for our use case as we were building a centralized authentication API service.

AWS Hosted UI

AWS Cognito offers a ‘hosted ui’, where by you redirect a user to an endpoint such as:

https://{...}.auth.us-east-1.amazoncognito.com/login?\
response_type={...}&client_id={...}&redirect_uri={...}&state={...}`

ℹ️ NOTE

a custom domain can also be configured, but it requires you use AWS Certificate Manager for the TLS cert.

The hosted ui option gives you all the interactions in a fully functioning interface, which includes: sign-in, sign-up, forgotten username, forgotten password, social logins.

But there are some caveats:

Very limited controls over the ui (very basic font colors and css).
Custom domains only work with TLS certificates via ACM.
State parameter overloading
Can’t access new signup passwords †

† this was necessary for my use case as I needed to co-support a legacy system that wasn’t ready to migrate over to Cognito

There are other issues still that I have with the hosted ui, but in a lot of cases it does the job well enough to put up with them.

The ‘state’ parameter overloading is an interesting issue and I’ll come back to that later on when I discuss a little bit about sign-ins with social providers.

Stateless Authentication

The thing we liked about Cognito was that it would allow us to build a ‘stateless’ authentication system. Due to the use of JWTs we could pass these tokens around (†) and know that if the user had these tokens that they would be valid and untampered with (because when decoding the tokens we could verifiy this using the public signing key AWS uses to sign the tokens at point of generation).

† we only ever pass tokens around ‘server-side’, using secure cookies (with HttpOnly and Secure attributes set) to avoid replay attacks that might occur if we exposed the tokens to the client.

The problem we then stumbled across was: what happens if a user authenticates on a public computer but doesn’t log out (or they authenticate on their laptop but have no password to prevent someone from stealing it and thus their existing authenticated session tokens)?

Well, we would still decode the ID JWT we got back from Cognito (to ensure there was no tampering of the token), but we would then make a simple API call to AWS (specifically the GetUser) as it required us to provide the ACCESS JWT we get back from Cognito.

The reason I mention this is because, we needed a way to invalidate a session, and the only way to do that was to call the GlobalSignOut API.

We originally though that would invalidate all tokens ID/Access/Refresh, but we were wrong. Only the Access and Refresh tokens are invalidated. But that was fine as our system was already being passed the Access token and so once we invalidated the session, if that user tried to reuse the tokens at one of our protected endpoints, we could be sure that it would now fail to give them access as we not only verified the ID token but attempted to use the Access token to call an AWS API to see if it suceeded or not.

Our use case is probably quite unconventional, but otherwise the whole point of having a stateless system was in danger of being made redundant by the fact that if (for whatever reason) we had a set of compromised users we’d otherwise have no way to invalidate their sessions (and we didn’t want to have to build our own session state datastore to track all this).

Logic Processing with AWS Lambda

With the hosted ui option you’ll likely also need to utilise AWS Lambda in order to do some logic processing. The following diagram demonstrates how we were initially using the hosted ui:

We redirect an unauthenticated user to Cognito.
Once the user attempts to sign-in we trigger some additional ‘hooks’.
Cognito redirects the authenticated user to our API service †
Our API service redirects the user back to the CMS (with user tokens).
The CMS asks the API service to validate the tokens.

† this service exchanges the given Cognito auth code for the user’s Cognito User Pool tokens.

Once the tokens are validated, the CMS will allow the user to view the relevant page.

Beware the Lambdas

It’s worth noting the second half of the above diagram (the section after the lambda is triggered). What we have there are two separate lambda’s, and which one is triggered depends on the scenario.

If the user has tried to authenticate using a username/password set of credentials, and those don’t match an existing user within the Cognito User Pool, then the “User Migration” lambda is triggered. In that lambda we attempt to authenticate the user within our legacy system (that is the call over to the “WebApp” in the diagram).

If the authentication with the legacy system is successful, then we’ll modify the user’s User Pool record (which hasn’t actually been created yet) to include auth related details we’ve pulled from their legacy account. We then return the ‘event’ object provided to Lambda, which let’s Cognito know it can now create the user within its User Pool (not returning the lambda event object indicates an error occurred and the whole request flow fails).

ℹ️ NOTE

with the ‘user migration’ for users from our legacy system over to Cognito, before we return the event in the lambda, we make sure to mark the new Cognito user as ‘verified/confirmed’ – that way they don’t need to enter a verification code that gets emailed or sent via SMS (that’s because the user would’ve already verified themselves originally in our legacy system).

The reason I say “beware the lambdas” is because yes, code errors can cause it to bomb out, but more importantly they don’t always fire when you think they will (this is a user error thing, not an AWS bug).

To clarify, let me explain what we saw when testing the migration path of a legacy user account to Cognito, when the user was signing into Cognito using their social provider details.

We had hoped the ‘User Migration’ lambda hook would have been triggered by both a Cognito User Pool account login and also a Social Provider account login, but it doesn’t.

ℹ️ NOTE

when a user signs-in with a social account, they have an account created within the Cognito User Pool, but they are also added to a specific group (such as a Facebook group or a Google group).

We eventually discovered that the ‘Post Confirmation’ hook would fire at the right interval for us to do the processing we needed for users signing in with a Social Account. But that wasn’t immediately obvious.

Before settling on the ‘Post Confirmation’ hook, we originally started using ‘Post Authentication’ for handling first time social logins (the hook sounded reasonable enough), but when we were testing this hook we already had the social account stored in our User Pool (this was from earlier testing, before we decided to do some ‘post-login’ processing).

The reason I mention this is because a week later we decided to clear out our User Pool and start testing our various scenarios again from scratch, and we noticed the ‘post authentication’ hook was no longer firing 🤔

Turns out social accounts only trigger ‘post migration’ hooks when they already exist in the User Pool. In order to do the ‘first time login’ modification we were looking for, we needed the ‘post confirmation’ hook.

Using this hook wasn’t obvious to us because ‘post confirmation’ makes it sounds like an event that happens once a username/password user has entered their ‘verification code’ for the first time (and thus become marked as ‘confirmed’ within the User Pool). Well, turns out social provider logins are automatically considered confirmed once they authenticate for the first time (hence why that event would trigger when we needed it to).

Useful Lambdas

There are some useful lambda’s though, for example, the Custom Message Lambda Trigger is great for intercepting the emails (or SMS) messages that are sent to your users, and allowing you to configure them however you like.

Take a look at the following code for an example…

def lambda_handler(event, context):

    domain = 'https://your.domain.com'
    username = event.get('userName', '')
    code = event['request'].get('codeParameter', '')

    print(event)

    if event['triggerSource'] == "CustomMessage_SignUp":
        event['response']['emailSubject'] = "Validate your account"
        event['response']['emailMessage'] = "Hi <b>" + username + "</b>!<br>" \
                                            "Thank you for signing up.<br>" \
                                            "Click <a href='" + domain + "confirm-account-signup-validation?" \
                                            "username=" + username + "&code=" + code + "'>here</a> " \
                                            "to validate your account."

    elif event['triggerSource'] == "CustomMessage_ForgotPassword":
        event['response']['emailSubject'] = "Reset your password"
        event['response']['emailMessage'] = "Hi <b>" + username + "</b>!<br>" \
                                            "Click <a href='" + domain + "confirm-password-reset?" \
                                            "identifier=" + username + "&code=" + code + "'>here</a> " \
                                            "to reset your password."

    elif event['triggerSource'] == "CustomMessage_UpdateUserAttribute":
        event['response']['emailSubject'] = "Validate your new email"
        event['response']['emailMessage'] = "Hi <b>" + username + "</b>!<br>" \
                                            "Click <a href='" + domain + "/confirm-email-change?" \
                                            "code=" + code + "'>here</a> " \
                                            "to validate your new email."

    if event['triggerSource'] == "CustomMessage_AdminCreateUser":
        user_attr = event['request'].get('userAttributes', {})
        user_status = user_attr.get('cognito:user_status')
        if user_status == 'FORCE_CHANGE_PASSWORD':
            event['response']['emailSubject'] = "Validate your account"
            event['response']['emailMessage'] = "Hi <b>" + username + "</b>!<br><br>" \
                                                "You attempted to signin, but your account is 'unverified'.<br><br>" \
                                                "Your temporary password is <b>" + code + "</b>.<br><br>" \
                                                "Click <a href='" + domain + "/confirm-account'>here</a>."

    return event

What’s good about this lambda is that we’re able to improve the user’s flow a little bit. Otherwise if we relied on AWS to generate the email/SMS we’d have to create a separate UI that allowed (for example, when verifying an account using a code) the user to copy paste their code into the UI and then submit that code to our server to process.

By controlling the email content ourselves we can construct an endpoint that has the verification code as a query param and make a GET request to an endpoint that will process that code for the user (saving them from having to manually enter anything).

Just something to consider when using Cognito: can I use lambda triggers to improve the user flow?

One thing that might not be clear when opting for a server-side solution is how to handle social logins (e.g. users signing in/up using facebook or google).

It might sound a bit strange, but in order to implement social logins you’ll need to make a call to the hosted ui endpoint (mentioned earlier):

https://{...}.auth.us-east-1.amazoncognito.com/login?response_type={...}\
&client_id={...}&redirect_uri={...}&state={...}

The specific endpoint you call will be based upon those supported in Cognito’s User Pools Auth API.

For example, to attempt to sign-in a user with facebook you would provide a button that links to:

https://{...}.auth.us-east-1.amazoncognito.com/oauth2/authorize?\
response_type={...}&client_id={...}&redirect_uri={...}&state={...}&identity_provider={...}

The value we use for the response_type parameter is code. What this does, once the user has authenticated with their social provider (defined by the identity_provider param), is redirect the user back to your service (specified via the redirect_uri param) and then your service is responsible for exchanging the code for the user’s User Pool Tokens (see the following section on JWTs).

The values you can assign to identity_provider are:

Facebook
Google
LoginWithAmazon

ℹ️ NOTE

if you were planning on handling authentication at a very low level (instead of an SDK), then for a User Pool login you would provide the value COGNITO.

Overloading the State Parameter

The state param is used for CSRF protection, and is the only parameter that is persisted when the user is redirected to redirect_uri.

A common problem for people using Cognito is that they need more than one redirect. In my case (see the earlier ‘hosted ui’ architecture diagram) I need to redirect the signed-in user to an API service so we can handle the exchanging of the AWS ‘code’ for the Cognito User Pool tokens before needing to then redirect the user back to our actual origin service.

The only way we can do this is to overload the state param so it has a value like:

state=123_redirect=https://www.example.com

The value 123 is the nonce (for CSRF) and the _ gives us a way to split the query param server-side to extract the secondary redirect endpoint.

ℹ️ NOTE

it’s recommended you do validation on that input (e.g. a whitelist of accepted URIs) so hackers can’t manipulate the endpoint a user is sent to once they’ve authenticated.

Scope

One thing I stumbled across, and which took a while to figure out, was when I tried to call the GlobalSignOut API operation.

It worked fine for users authenticated against the Cognito User Pool, but not for users authenticated via their social provider.

Turns out I needed to enable the right scope within the Cognito User Pool UI console (within “App Integration -> App Client Settings”, and under “Allowed OAuth Scopes”): aws.cognito.signin.user.admin needed to be ticked.

But also, when making the request to the Auth API endpoint (e.g. /oauth2/authorize), I needed to append a scope query parameter: &scope=scope=openid+aws.cognito.signin.user.admin.

See the API docs and the UI docs for more information on the reasoning.

JWTs

When you exchange the cognito ‘code’ for a user pool ‘token’, you’ll actually be returned three tokens:

ID token
Access token
Refresh token

ℹ️ NOTE

see documentation for more details on these three tokens.

The ID token provides details about the user, and the access token indicates the access allowed to that user’s attributes stored within the Cognito User Pool.

Both the ID token and access token will expire after one hour. To use them after that you’ll need the refresh token to refresh the access/id tokens for another hour. The refresh token expires after 30 days.

We use the ID token for verifying the user is authenticated, and we do this by passing the token to an internal service that verifies the token hasn’t been manipulated by checking it against the AWS JWK that cryptographically signed the token.

The JWK is a set of keys that are the public equivalent of the private keys used by AWS to digitally sign the tokens. We acquire these via a standard format endpoint:

https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json.

ℹ️ NOTE

the JWK’s are rotated every 24hrs (approx), and so you need to ensure (if you’re caching the response) your code gets a fresh copy of the JWK. You can check this by inspecting the Cache-Control header set on the JWK response.

API Limits

One issue we stumbled across recently was the API limits, which meant we couldn’t make any further API requests (and for an indeterminate amount of time) 🤔

Seems there is a Cognito API limits reference page, but it’s still unclear how long you have to wait before you can start making requests again.

Logout Issues

AWS provides a /logout endpoint that when visited allows a user to clear any session tracking AWS might have on their browser.

This is different to the ‘signout’ API functionality in that the user can call the /logout endpoint without any special tokens †

ℹ️ NOTE

you have to provide quite specific query params (e.g. client_id and logout_uri – so AWS can redirect back to a pre-configured logout page that you host) and so it’s likely you’ll want to wrap that long and ugly URL within an <a href=''>click to logout</a> link.

This endpoint is useful because if you have a social user in your user pool, and you delete that user (something we would do regularly while we were testing in our stage environment), you would find that AWS is tracking your browser via session and so if you tried to sign-up or sign-in again using that same social user you would get an invalid grant exception back from AWS (this was super confusing behaviour and took a long time to figure out).

So for our use case, we would have a social user (e.g. facebook or google) sign-in for the first time, Cognito would (once the user had authenticated with their social provider) automatically create a social user account in our user pool …BUT we have other post-processing steps and if any of those fail we want to delete the social user and tell the user what went wrong.

But again, we need to now call the /logout endpoint so if that user decides they want to try and sign-up again they don’t get a more confusing invalid_grant error message thrown at them (and we don’t have to translate that message into something that shouldn’t even be a concern for them in the first place).

This brings us to the problem we have with the /logout endpoint…

Our intention, for when we were getting an error from an AWS API operation, would be to catch the error, then make a request to /logout and have it redirect to a failure page we host (the redirection is a built-in feature that AWS provides via a logout_uri query param).

Our logout_uri value (a URL) would itself have a query param specified of err_msg=<SOME_ERROR> so that our failure page could use that param to indicate the original error to the user. Problem was in Cognito you can only specify a logout url that matches what’s predefined in your user pool.

Meaning if we wanted to redirect to https://example.com/failure?err_msg=foo then that’s exactly what needs to be defined in Cognito (even down to the query param). We didn’t realise this and we only had https://example.com/failure defined as a valid logout_uri.

To solve this issue we could have explicitly specified the err_msg param but we have lots of error types to handle and so it wasn’t practical to list each and every variant URL.

So to work around the fact that we can’t specify a query param (because there are too many value variants to be practical for us to explicitly list all of them in the UI) we now catch the original API error and redirect the user to our failure page with the err_msg param passed so we can indicate the error back to the user.

At this point we still need to call the AWS /logout endpoint so we can clear any session tracking. So along with the error message we display to the user, we also show a message to say we will redirect them automatically (via JavaScript) back to another part of our site in N seconds time (enough time for the user to read the error we’re displaying).

Before we redirect the user to that page we first redirect them to the AWS /logout endpoint, this time specifying our failure page as the value for the logout_uri query param but just without an err_msg query param provided.

Our failure page is configured with a conditional check that says “if an err_msg param is passed then show error page with the JS redirect to /logout, otherwise if no err_msg provided just immediately redirect to our /signin page”.

So that’s how we’re resolving this issue currently. It’s not elegant for sure but it works.

Other Concerns?

It’s worth me mentioning a few other concerns we had, the first of which wasn’t directly related to Cognito but was specific to the system we were designing, and that was ‘atomic operations’.

We had to make changes to Cognito and then sync some behaviours back to our legacy system. If there was a network (or other) fault then we needed both systems to be tolerant and to undo any data modifcations in case of failure.

Other than that we would be storing off the JWT tokens we received from AWS into secure cookies (+Secure +HttpOnly attributes), and this caused us issues because our cookies needed to be scoped to the right domains to prevent overloading the HTTP request header limit.

Due to the JWTs being large in size, and the fact that BuzzFeed has many services/properties for users to interact with, we noticed that some users would end up seeing a 400 Bad Request caused by the large cookies. The routing services in front of these upstreams are generally nginx instances and so we would use large_client_header_buffers to allow an increased size until such a time we could figure out an appropriate solution.

Which is the right solution?

The answer: it depends.

For me the server-side solution made the most sense, and although difficult in the beginning (primarily due to documentation and general mis-understandings about the difference between User Pools and Identity Pools) we found it worked the best for our requirements, and gave us the most flexibility.

Updated Architecture

If you’re interested the updated architecture looked something like this…

None of the listed services are public, they’re all internal. The “API Gateway” is an internal tool that allows upstreams (such as the buzzfeed_auth_api to control concurrency and rate limiting) of downstream consumers (such as buzzfeed_auth_ui and user_settings).

The reason we migrated certain ‘user settings’ functionality out of our monolithic webapp and not other user features is because we only wanted to move behaviours that interacted with fields that needed sync’ing between Cognito and our legacy datastore. As times goes on, we’ll start to migrate more and more functionality out into separate services.

We discovered that the social sign-in for native mobile apps doesn’t work as well as the web SDK’s. Mobile apps need to instead do things differently as their SDK’s aren’t as ‘integrated’ like web.

User Pool Configuration

As far as the User Pool is concerned you’ll need a few things:

ℹ️ NOTE

this is based on a server-side solution.

Application Client: this will generate a client ‘id’ and ‘secret’, which your application(s) will need to use when making certain API calls †
Federated Identity Providers: this is where you tell Cognito about your social providers (facebook, google etc).
IAM User: some API calls require AWS credentials (access/secret key), so you’ll need to create an IAM user and define the various Cognito APIs you want it to have access to.

† even if you opt for the ‘hosted ui’ solution, you’ll still need an application client (for two reasons). Firstly you’ll configure which ‘providers’ you want your client app to support, and this will affect what the hosted ui will display to your users. Secondly, the client app id is used as part of the hosted ui uri; meaning you can have different hosted ui’s (all configured slightly differently).

IAM User

The IAM user is necessary as we have to provide some credentials to the boto client (boto is the Python SDK) in order for it to make certain API calls. Below is an example of the code to instantiate a client:

client = boto3.client('cognito-idp', **{'aws_access_key_id': access_key,
                                        'aws_secret_access_key': secret_key,
                                        'region_name': region})

Notice the service name is cognito-idp and not cognito-identity. I mention this as the docs specify two different services “CognitoIdentity” and “CognitoIdentityProvider”, which when we were first learning about Cognito we presumed the latter “CognitoIdentityProvider” was something associated with Cognito Identity Pools.

As we were only interested in the User Pool functionality, we found it strange that the small number of examples we found online all referenced cognito-idp.

So we struggled for a bit to understand the difference, and although we used the “CognitoIdentityProvider” service (i.e. cognito-idp), we were confused for a long time as to why that was the case.

Turns out that Cognito’s “User Pool” is itself fundamentally a identity provider (idp), and because of that you can configure a “Identity Pool” to have a “User Pool” associated within it (along with more common external identity providers such as Facebook and Google).

So with that understanding firmly in place, the fact the SDK uses cognito-idp for interacting with a User Pool makes total sense (because the User Pool is an “idp”), and the Identity Pool is just a tool for handling “identities” via many different providers (whether that be a User Pool or a social ‘provider’ such as Facebook or Google), and so the SDK using cognito-identity for interating with AWS Identity Pools also makes perfect sense.

It’s the little details that can really make a difference to even the simplest aspects of using an SDK/API, and why Amazon’s atrocious documentation is a real detriment to its users.

Lambda IAM Role

Below is an example IAM role policy you can use for AWS Lambda (if you’re using the hosted ui option and need lambda for logic processing):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:AdminUpdateUserAttributes"
            ],
            "Resource": "arn:aws:cognito-idp:us-east-1:{aws_account_id}:userpool/{user_pool_id}"
        }
    ]
}

It simply sets up CloudWatch logs access, and allows us (as an ‘admin’) to update user attributes within our User Pool.

ℹ️ NOTE

if you’re ‘copying and pasting’, don’t forget to update aws_account_id and user_pool_id in the code snippet.

Example Python API code

This is a small slice of some Python code I wrote for abstracting away the API interactions with Cognito. Some readers might find it useful for understanding parts of Cognito (apologies if it’s a bit ambiguous, as it’s taken out of context).

# standard library modules

import json
import re
from functools import reduce
from typing import Dict, List, Optional, Tuple

# application modules

import app.exceptions as exceptions
import app.network
import app.security

# external modules

from bf_auth.utility import extract_status_code, instr_exc

import boto3
from botocore.exceptions import ClientError

# configuration

aws_access_key = config.get('aws_application_access_key')
aws_region = config.get('cognito_region')
aws_secret_key = config.get('aws_application_secret_key')
client_id = config.get('app_client_id')
client_secret = config.get('app_secret_key')
cognito_domain = config.get('cognito_domain')
default_redirect = config.get('default_redirect')
credentials = {'aws_access_key_id': aws_access_key, 'aws_secret_access_key': aws_secret_key, 'region_name': aws_region}
sdk = boto3.client('cognito-idp', **credentials)
user_pool_id = config.get('cognito_user_pool_id')
user_pool_jwk = f'https://cognito-idp.{aws_region}.amazonaws.com/{user_pool_id}/.well-known/jwks.json'


def attr_adapter(attributes: dict) -> List[Dict[str, str]]:
    """Convert dictionary into a Cognito compatible structure."""
    return [{'Name': key, 'Value': value} for key, value in attributes.items()]


def create_user(username: str,
                password: str,
                verified: bool,
                email: str,
                user_info_id: str) -> Optional[dict]:
    """Create the user, then authenticate them to acquire their tokens.

    If the record is created successfully, we'll return user tokens.

    We use the 'Server Side Authentication Flow' for authenticating and
    migrating users with Cognito (this means no need for generating an SRP)

    Documentation:
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminCreateUser.html
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminInitiateAuth.html
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminRespondToAuthChallenge.html
    """

    # default to forcing user to validate
    message_action = {}  # type: ignore
    cognito_password = app.security.random()
    user_verified = False

    if verified:
        message_action = {'MessageAction': 'SUPPRESS'}
        cognito_password = password
        user_verified = True

    attributes = {'email': email,
                  'email_verified': str(verified).lower(),
                  'custom:user_info_id': str(user_info_id)}

    cognito_attributes = attr_adapter(attributes)

    # we use admin_create_user so that we can choose to either suppress or
    # trigger the sending of a 'verify your account' email.
    #
    # but in using this api call, cognito requires us to set a temporary
    # password for the user.
    #
    # so in the case of a verified user we set their password to their actual
    # password (so they notice no difference), where as with an unverified user
    # we will generate them a random password as they'll then reset that
    # password when cognito sends them the account verification email.
    try:
        sdk.admin_create_user(**{'UserPoolId': user_pool_id,
                                 'UserAttributes': cognito_attributes,
                                 'Username': username,
                                 'TemporaryPassword': cognito_password,
                                 **message_action})

    except Exception as exc:
        msg = 'USER_CREATE_FAILED'
        instr_exc(exc, msg,
                  metric_name='create.user',
                  context='normal',
                  scope='cognito',
                  state='failed',
                  verified=user_verified,
                  user_info_id=user_info_id)
        raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    # also for a verified user we'll try to authenticate and acquire their user
    # pool tokens, which will be returned to the caller of this function.
    if user_verified:
        auth_params = {'USERNAME': username,
                       'PASSWORD': password,
                       'SECRET_HASH': app.security.hash(username)}

        challenge_responses = {'USERNAME': username,
                               'NEW_PASSWORD': password,
                               'SECRET_HASH': auth_params['SECRET_HASH']}

        try:
            auth_response = sdk.admin_initiate_auth(UserPoolId=user_pool_id,
                                                    ClientId=client_id,
                                                    AuthFlow='ADMIN_NO_SRP_AUTH',
                                                    AuthParameters=auth_params)

            challenge_name = auth_response.get('ChallengeName')
            session = auth_response.get('Session')

            response = sdk.admin_respond_to_auth_challenge(UserPoolId=user_pool_id,
                                                           ClientId=client_id,
                                                           ChallengeName=challenge_name,
                                                           ChallengeResponses=challenge_responses,
                                                           Session=session)

            result = response.get('AuthenticationResult', {})
            tokens = {'id_token': result.get('IdToken'),
                      'access_token': result.get('AccessToken'),
                      'refresh_token': result.get('RefreshToken'),
                      'token_type': result.get('TokenType')}

            return tokens
        except Exception as exc:
            msg = 'USER_AUTH_FAILED'
            instr_exc(exc, msg,
                      metric_name='create.user',
                      context='normal',
                      scope='cognito',
                      state='failed',
                      verified=user_verified,
                      user_info_id=user_info_id)
            raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    # if we haven't returned some tokens already, then that means we
    # successfully created a cognito account for an unverified user, and now we
    # can raise the relevant exception that will be served back to the
    # application JS to handle (i.e. display a message to the user)
    msg = 'USER_MIGRATE_UNVERIFIED'
    gen_exc = exceptions.CognitoException(msg, code=201)
    raise gen_exc


def delete_user(username) -> bool:
    """Delete user record associated with the given username.

    If the record is deleted successfully, we'll return True.

    Documentation:
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminDeleteUser.html
    """

    try:
        sdk.admin_delete_user(**{'UserPoolId': user_pool_id,
                                 'Username': username})

    except ClientError as exc:
        error = exc.response.get('Error', {})
        error_code = error.get('Code')

        msg = 'USER_DELETE_FAILED'
        exc_tags = {'metric_name': 'user.delete',
                    'state': 'failed',
                    'scope': 'cognito'}

        if error_code == 'UserNotFoundException':
            instr_exc(exc, msg, **exc_tags)
            raise exceptions.CognitoException(msg, code=404)
        else:
            instr_exc(exc, msg, **exc_tags)
            raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    except Exception as exc:
        msg = 'USER_DELETE_FAILED'
        exc_tags = {'metric_name': 'user.delete',
                    'state': 'failed',
                    'scope': 'cognito'}
        instr_exc(exc, msg, **exc_tags)
        raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    return True


@app.network.cache
async def get_keys(endpoint=user_pool_jwk) -> dict:
    """Retrieve JWK (for verifying tokens).

    If successful we return a dict consisting of the cache-control response
    header value and the actual list of JWKs.

    If unsuccessful we return the standard dictionary error format.
    """

    response = await app.network.http_client.fetch(endpoint)
    cache_control = response.headers.get('Cache-Control')

    match = re.search(r'max-age=(\d+)', cache_control)

    if not match or response.code != 200:
        msg = 'JWK_RESPONSE_INVALID'
        gen_exc = exceptions.AsyncFetchException(msg, code=response.code)
        instr_exc(gen_exc, msg, cache_control=cache_control, scope='cognito')
        raise gen_exc

    try:
        response_data = json.loads(response.body)
    except Exception as exc:
        msg = 'JSON_PARSING_FAILED'
        instr_exc(exc, msg, endpoint=endpoint, body=response.body, scope='cognito')
        return {'state': 'error',
                'code': 500,
                'message': msg}

    return {'state': 'success',
            'cache_control': match.group(1),
            'response_body': response_data.get('keys', [])}


async def get_key(kid: str) -> str:
    """Extract signing key from JWK (for verifying tokens)."""

    response = await get_keys()
    keys = response.get('response_body')
    key = list(filter(lambda x: x.get('kid') == kid, keys))

    if len(key) < 1:
        raise exceptions.NoJWK('JWK_FILTER_FAILED')

    return key[0]


def global_signout(access_token) -> bool:
    """Sign out user from all devices.

    If the user is signed out successfully, we'll return True.

    Documentation:
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_GlobalSignOut.html
    """

    try:
        sdk.global_sign_out(AccessToken=access_token)
    except Exception as exc:
        msg = 'USER_GLOBAL_SIGNOUT_FAILED'
        instr_exc(exc, msg, metric_name='user.global_signout.failed', scope='cognito')
        raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    return True


def admin_signout(username) -> bool:
    """Sign out user from all devices as an administrator.

    If the user is signed out successfully, we'll return True.

    Documentation:
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminUserGlobalSignOut.html
    """

    try:
        sdk.admin_user_global_sign_out(**{'UserPoolId': user_pool_id,
                                          'Username': username})
    except Exception as exc:
        msg = 'USER_SIGNOUT_FAILED'
        instr_exc(exc, msg, metric_name='user.signout.failed', scope='cognito')
        raise exceptions.CognitoException(msg, code=extract_status_code(exc))

    return True


def get_attribute(key, user_attrs) -> Optional[str]:
    """Recursively search user attributes for given key."""

    return reduce(lambda x, y: x if x.get('Name') == key else y, user_attrs)


async def exchange_code_for_tokens(code, redirect_host, code_verifier=None) -> dict:
    """Exchange the Facebook/Google authorization code for user tokens.

    Documentation:
    https://docs.aws.amazon.com/cognito/latest/developerguide/token-endpoint.html

    Example response:

    {
      "access_token":"123",
      "refresh_token":"456",
      "id_token":"789",
      "token_type":"Bearer",
      "expires_in":3600
    }
    """

    # Note: the `redirect_uri` field isn't used for redirecting, but for verification.
    # the value needs to match what is configured within the allowed cognito
    # callback endpoints
    data = {'code': code,
            'grant_type': 'authorization_code',
            'client_id': client_id,
            'redirect_uri': f'{redirect_host}/auth/signin/callback'}

    if code_verifier:
        data['code_verifier'] = code_verifier

    metric_tags = {'metric_tags': {'scope': 'cognito', 'context': 'social', 'name': 'token_exchange'}}
    endpoint = f'{cognito_domain}/oauth2/token'
    headers = {'Content-Type': 'application/x-www-form-urlencoded'}

    response = await app.network.fetch(endpoint, post_data=data,
                                       headers=headers,
                                       auth_username=client_id,
                                       auth_password=client_secret,
                                       **metric_tags)

    if response.get('state') == 'error':
        gen_exc = exceptions.AsyncFetchException(response.get('message'),
                                                 code=response.get('code'))
        raise gen_exc

    return response


async def resend_confirmation_code(username) -> bool:
    """Resend account confirmation email

    Documentation:
    https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ResendConfirmationCode.html
    """

    try:
        sdk.resend_confirmation_code(**{'ClientId': client_id,
                                        'SecretHash': app.security.hash(username),
                                        'Username': username})
    except Exception as exc:
        msg = 'CONFIRMATION_RESEND_FAILED'
        instr_exc(exc, msg, metric_name='user.confirmation_resend.failed', scope='cognito')
        return False

    return True

Example Cognito App Settings

This isn’t meant to be an exhaustive example, but it gives you an idea of some of the configuration you’ll need.

Callback URL(s):
https://auth-api.example.com/auth/signin/callback
Sign out URL(s):
https://auth-api.example.com/auth/signout
Allowed OAuth Flows:
Authorization code grant
Implicit grant
Allowed OAuth Scopes:
email
openid
profile
aws.cognito.signin.user.admin

Example Cognito User Pool “Federation: Identity Providers”

For each provider there is a “Authorize Scope” section.

Facebook:
public_profile,email
Google:
profile email openid

Facebook Attribute Mappings

fb: id –> user_pool: Username
fb: email –> user_pool: Email
fb: name –> user_pool: Name

Google Attribute Mappings

google: email –> user_pool: Email
google: name –> user_pool: Name
google: sub –> user_pool: Username

Example Facebook App Configuration

https://developers.facebook.com/apps

App Domains:
https://your-organisation.auth.us-east-1.amazoncognito.com
Privacy Policy URL:
https://www.example.com/about/privacy
Site URL:
https://your-organisation.auth.us-east-1.amazoncognito.com/oauth2/idpresponse

Client OAuth Login:
Yes
Web OAuth Login:
Yes
Enforce HTTPS:
Yes
Valid OAuth Redirect URIs:
https://your-organisation.auth.us-east-1.amazoncognito.com/oauth2/idpresponse https://auth-api.example.com/auth/signin/callback

Example Google App Configuration

https://console.developers.google.com/

Enabled API(s):
Google+ API
Credentials Type:
OAuth client ID
Application Type:
Web application
Authorized JavaScript origins:
https://your-organisation.auth.us-east-1.amazoncognito.com
Authorized redirect URIs:
https://your-organisation.auth.us-east-1.amazoncognito.com/oauth2/idpresponse https://auth-api.example.com/auth/signin/callback

Terraform Example

# Examples for Cognito User Pools can be found here:
# https://github.com/terraform-providers/terraform-provider-aws/blob/master/examples/cognito-user-pool/main.tf

####################################################
# main.tf
####################################################

# TODO: split this file up into separate modules
# e.g.
#
# user_pool/
# identity_pool/

provider "aws" {
  region = "${var.aws_region}"

  assume_role {
    role_arn = "${var.aws_role_arn}"
  }
}

resource "aws_cognito_user_pool" "pool" {
  name = "${var.environment}_${var.name}_user_pool"

  alias_attributes         = ["email", "preferred_username", "phone_number"]
  auto_verified_attributes = ["email", "phone_number"]

  admin_create_user_config {
    allow_admin_create_user_only = false
  }

  # container for the AWS Lambda triggers associated with the user pool.
  # https://www.terraform.io/docs/providers/aws/r/cognito_user_pool.html#lambda-configuration
  lambda_config {
    custom_message = "${aws_lambda_function.custom_message_lambda.arn}"
  }

  mfa_configuration = "OPTIONAL"

  sms_configuration {
    external_id    = "${var.environment}_${var.name}_sns_external_id"
    sns_caller_arn = "${aws_iam_role.cognito_sns_role.arn}"
  }

  password_policy {
    minimum_length    = 6
    require_lowercase = false
    require_numbers   = false
    require_symbols   = false
    require_uppercase = false
  }

  /*
  # email was a required field, but it ended up causing issues for any social
  # users whose identity is actually their mobile number. So to avoid problems
  # authenticating those users, we no longer require an email to be provided.
  schema {
    name                     = "email"
    attribute_data_type      = "String"
    developer_only_attribute = false
    mutable                  = true
    required                 = true

    string_attribute_constraints {
      min_length = 1
      max_length = 2048
    }
  }
  */

  schema {
    name                     = "some_custom_attribute"
    attribute_data_type      = "Number"
    developer_only_attribute = false
    mutable                  = true
    required                 = false

    number_attribute_constraints {
      min_value = 1
      max_value = 50000000
    }
  }
  tags {
    "environment" = "${var.environment}"
    "service"     = "${var.name}"
  }
  depends_on = [
    "aws_iam_role.cognito_sns_role",
  ]
}

resource "aws_cognito_user_pool_client" "pool_client" {
  # Federation > Identity providers
  depends_on = [
    "aws_cognito_identity_provider.facebook_provider",
    "aws_cognito_identity_provider.google_provider",
  ]

  # General settings > App clients
  user_pool_id           = "${aws_cognito_user_pool.pool.id}"
  name                   = "${var.environment}_${var.name}_user_pool_client"
  generate_secret        = true
  refresh_token_validity = 30
  explicit_auth_flows    = ["ADMIN_NO_SRP_AUTH", "USER_PASSWORD_AUTH"]

  # this flag is automatically set to true when creating the user pool using the AWS console.
  # however, when creating the user pool using Terraform, this flag needs to be set explicitly.
  allowed_oauth_flows_user_pool_client = true

  # issue: https://github.com/terraform-providers/terraform-provider-aws/issues/4476
  read_attributes  = ["email", "preferred_username", "profile", "custom:some_custom_attribute"]
  write_attributes = ["email", "preferred_username", "profile", "custom:some_custom_attribute"]

  # App integration > App client settings
  supported_identity_providers = ["COGNITO", "Facebook", "Google"]
  callback_urls                = "${var.callback_urls}"
  logout_urls                  = "${var.logout_urls}"
  allowed_oauth_flows          = ["code"]

  allowed_oauth_scopes = [
    "aws.cognito.signin.user.admin",
    "email",
    "openid",
    "profile",
  ]
}

# aws cert configured in certs.tf
resource "aws_cognito_user_pool_domain" "pool_domain" {
  domain          = "${var.domain}.${var.root_domain}"
  certificate_arn = "${aws_acm_certificate.certificate.arn}"
  user_pool_id    = "${aws_cognito_user_pool.pool.id}"
}

# bug in https://github.com/terraform-providers/terraform-provider-aws/issues/4807 that keep showing changes in plan
resource "aws_cognito_identity_provider" "google_provider" {
  user_pool_id  = "${aws_cognito_user_pool.pool.id}"
  provider_name = "Google"
  provider_type = "Google"

  provider_details {
    authorize_scopes = "profile email openid"
    client_id        = "${var.google_provider_client_id}"
    client_secret    = "${var.google_provider_client_secret}"
  }

  attribute_mapping {
    username = "sub"
    email    = "email"
  }
}

# bug in https://github.com/terraform-providers/terraform-provider-aws/issues/4807 that keep showing changes in plan
resource "aws_cognito_identity_provider" "facebook_provider" {
  user_pool_id  = "${aws_cognito_user_pool.pool.id}"
  provider_name = "Facebook"
  provider_type = "Facebook"

  provider_details {
    authorize_scopes = "public_profile,email"
    client_id        = "${var.facebook_provider_client_id}"
    client_secret    = "${var.facebook_provider_client_secret}"
  }

  attribute_mapping {
    username = "id"
    email    = "email"
  }
}

# The identity pool(s) are used by our mobile apps, and allows them to authenticate
# their users via our Cognito 'user pool'.
#
# Note: we're not sure if we need to configure anything else in facebook/google ui's?
#       we're also not sure what `server_side_token_check` (set below) really means.
resource "aws_cognito_identity_pool" "apps_identity_pool" {
  identity_pool_name               = "${var.environment}_${var.name}_identity_pool"
  allow_unauthenticated_identities = false

  cognito_identity_providers {
    client_id               = "${aws_cognito_user_pool_client.pool_client.id}"
    provider_name           = "cognito-idp.us-east-1.amazonaws.com/${aws_cognito_user_pool.pool.id}"
    server_side_token_check = false
  }

  supported_login_providers {
    "graph.facebook.com"  = "${var.facebook_provider_client_id}"
    "accounts.google.com" = "${var.google_provider_client_id}"
  }

  depends_on = [
    "aws_cognito_user_pool.pool",
  ]
}

# an identity pool (used by mobile apps) requires a role to be assigned to both
# authenticated and unauthenticated access (even if the identity pool is configured
# to not allow unauthenticated access, it still requires a role to be assigned)
#
# https://www.terraform.io/docs/providers/aws/r/cognito_identity_pool_roles_attachment.html
resource "aws_iam_role" "apps_identity_pool_authenticated" {
  name = "${var.environment}_${var.name}_identitypool_authenticated"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "cognito-identity.amazonaws.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "cognito-identity.amazonaws.com:aud": "${aws_cognito_identity_pool.apps_identity_pool.id}"
        },
        "ForAnyValue:StringLike": {
          "cognito-identity.amazonaws.com:amr": "authenticated"
        }
      }
    }
  ]
}
EOF
}

resource "aws_iam_role" "apps_identity_pool_unauthenticated" {
  name = "${var.environment}_${var.name}_identitypool_unauthenticated"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}
EOF
}

# we can then attach additional policies to each identity pool role
resource "aws_iam_role_policy" "apps_identity_pool_authenticated" {
  name = "${var.environment}_${var.name}_identitypool_authenticated_policy"
  role = "${aws_iam_role.apps_identity_pool_authenticated.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "mobileanalytics:PutEvents",
        "cognito-sync:*",
        "cognito-identity:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF
}

# we don't allow unauthenticated access, so just set all actions to be denied
resource "aws_iam_role_policy" "apps_identity_pool_unauthenticated" {
  name = "${var.environment}_${var.name}_identitypool_unauthenticated_policy"
  role = "${aws_iam_role.apps_identity_pool_unauthenticated.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF
}

# finally, we can attach our roles to our identity pools
resource "aws_cognito_identity_pool_roles_attachment" "apps_identity_pool_role_attachment" {
  identity_pool_id = "${aws_cognito_identity_pool.apps_identity_pool.id}"

  roles {
    "authenticated"   = "${aws_iam_role.apps_identity_pool_authenticated.arn}"
    "unauthenticated" = "${aws_iam_role.apps_identity_pool_unauthenticated.arn}"
  }
}

/*
We originally had this policy inlined within the the below iam role,
but then discovered it caused a cyclic reference...

aws_cognito_user_pool -> aws_lambda_function -> aws_iam_role <BOOM!> -> aws_cognito_user_pool

So to avoid that we could have made the policy not depend on that
specific user pool resource, using: "arn:aws:cognito-idp:*:*:*"
but we opted to create a separate policy, which we then attach to
the existing role, and tell the policy it can't be attached until
the user pool has been created.
*/
resource "aws_iam_role_policy" "cognito_lambda_policy" {
  depends_on = [
    "aws_cognito_user_pool.pool",
  ]

  name = "send_user_email_policy"
  role = "${aws_iam_role.iam_for_lambda.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Action": [
        "cognito-idp:AdminUpdateUserAttributes"
      ],
      "Effect": "Allow",
      "Resource": "${aws_cognito_user_pool.pool.arn}"
    }
  ]
}
EOF
}

resource "aws_iam_role" "iam_for_lambda" {
  name = "${var.environment}_${var.name}_sendUserEmailLambdaRole"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

data "archive_file" "generate_custom_message_lambda" {
  type        = "zip"
  source_dir  = "${path.module}/source/"
  output_path = "lambda.zip"
}

resource "aws_lambda_function" "custom_message_lambda" {
  filename         = "lambda.zip"
  function_name    = "${var.environment}_${var.name}_customMessages"
  role             = "${aws_iam_role.iam_for_lambda.arn}"
  handler          = "custom_message.lambda_handler"
  source_code_hash = "${data.archive_file.generate_custom_message_lambda.output_base64sha256}"
  runtime          = "python3.6"
}

# this resource allows lambda to be invoked by our user pool and tripped us up initially because
# it is automatically applied when setting up the lambda trigger in the AWS console.
# however, when creating the lambda trigger via Terraform, this needs to be set explicitly.
resource "aws_lambda_permission" "allow_cognito" {
  statement_id  = "AllowExecutionFromCognito"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.custom_message_lambda.function_name}"
  principal     = "cognito-idp.amazonaws.com"
  source_arn    = "${aws_cognito_user_pool.pool.arn}"
}

####################################################
# certs.tf
####################################################

resource "aws_acm_certificate" "certificate" {
  domain_name       = "${var.domain}.${var.root_domain}"
  validation_method = "DNS"

  tags {
    "environment" = "${var.environment}"
    "service"     = "${var.name}"
  }
}

####################################################
# outputs.tf
####################################################

output "user_pool_id" {
  value = "${aws_cognito_user_pool.pool.id}"
}

output "user_pool_arn" {
  value = "${aws_cognito_user_pool.pool.arn}"
}

output "user_pool_client_id" {
  value = "${aws_cognito_user_pool_client.pool_client.id}"
}

output "user_pool_client_secret" {
  // this is only shown at creation
  value = "${aws_cognito_user_pool_client.pool_client.client_secret}"
}

output "app_user_name" {
  value = "${aws_iam_user.cognito_app_user.name}"
}

output "app_user_arn" {
  value = "${aws_iam_user.cognito_app_user.arn}"
}

output "acm_certificate_arn" {
  value = "${aws_acm_certificate.certificate.arn}"
}

output "acm_certificate_domain_name" {
  value = "${aws_acm_certificate.certificate.domain_name}"
}

output "acm_certificate_domain_validation_options" {
  value = "${aws_acm_certificate.certificate.domain_validation_options}"
}

####################################################
# required.tf
####################################################

terraform {
  # No value within the terraform block can use interpolations. 
  # The terraform block is loaded very early in the execution of Terraform and interpolations are not yet available.
  required_version = "0.10.7"
}

####################################################
# service_iam.tf
####################################################

resource "aws_iam_group" "cognito_app_group" {
  name = "${var.environment}_${var.name}_group"
}

resource "aws_iam_user" "cognito_app_user" {
  name = "${var.environment}_${var.name}_user"
}

# note:
# we don't also create an 'aws_iam_access_key' resource
# because we don't want the access key to be committed
# 
# so we manually create access/secret keys via the console

resource "aws_iam_user_group_membership" "cognito_app_user_groups" {
  user = "${aws_iam_user.cognito_app_user.name}"

  groups = [
    "${aws_iam_group.cognito_app_group.name}",
  ]
}

data "aws_iam_policy_document" "cognito_app_group_policy" {
  statement {
    actions = [
      "cognito-idp:ListUserPools",
      "cognito-idp:ListUsers",
    ]

    resources = [
      "*",
    ]
  }

  statement {
    actions = [
      "cognito-idp:AdminAddUserToGroup",
      "cognito-idp:AdminConfirmSignUp",
      "cognito-idp:AdminCreateUser",
      "cognito-idp:AdminDeleteUser",
      "cognito-idp:AdminDeleteUserAttributes",
      "cognito-idp:AdminDisableProviderForUser",
      "cognito-idp:AdminDisableUser",
      "cognito-idp:AdminEnableUser",
      "cognito-idp:AdminForgetDevice",
      "cognito-idp:AdminGetDevice",
      "cognito-idp:AdminGetUser",
      "cognito-idp:AdminInitiateAuth",
      "cognito-idp:AdminLinkProviderForUser",
      "cognito-idp:AdminListDevices",
      "cognito-idp:AdminListGroupsForUser",
      "cognito-idp:AdminListUserAuthEvents",
      "cognito-idp:AdminRemoveUserFromGroup",
      "cognito-idp:AdminResetUserPassword",
      "cognito-idp:AdminRespondToAuthChallenge",
      "cognito-idp:AdminSetUserMFAPreference",
      "cognito-idp:AdminSetUserSettings",
      "cognito-idp:AdminUpdateAuthEventFeedback",
      "cognito-idp:AdminUpdateDeviceStatus",
      "cognito-idp:AdminUpdateUserAttributes",
      "cognito-idp:AdminUserGlobalSignOut",
    ]

    resources = [
      "${aws_cognito_user_pool.pool.arn}",
    ]
  }
}

resource "aws_iam_policy" "cognito_app_group_policy" {
  name   = "${var.environment}_${var.name}_group_policy"
  policy = "${data.aws_iam_policy_document.cognito_app_group_policy.json}"
}

resource "aws_iam_group_policy_attachment" "cognito_app_group_attachment" {
  group      = "${aws_iam_group.cognito_app_group.name}"
  policy_arn = "${aws_iam_policy.cognito_app_group_policy.arn}"
}

####################################################
# sns_iam.tf
####################################################

data "aws_iam_policy_document" "cognito_sns_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["cognito-idp.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "cognito_sns_role" {
  name               = "${var.environment}_${var.name}_cognito_sns_role"
  assume_role_policy = "${data.aws_iam_policy_document.cognito_sns_assume_role_policy.json}"
}

data "aws_iam_policy_document" "cognito_sns_publish_policy" {
  statement {
    actions = [
      "sns:Publish",
    ]

    resources = [
      "*",
    ]
  }
}

resource "aws_iam_policy" "cognito_sns_role_policy" {
  name   = "${var.environment}_${var.name}_cognito_sns_role_policy"
  policy = "${data.aws_iam_policy_document.cognito_sns_publish_policy.json}"
}

resource "aws_iam_role_policy_attachment" "cognito_sns_role_policy_attachment" {
  role       = "${aws_iam_role.cognito_sns_role.name}"
  policy_arn = "${aws_iam_policy.cognito_sns_role_policy.arn}"
}

####################################################
# vars.tf
####################################################

variable "aws_role_arn" {}

variable "aws_region" {
  default = "us-east-1"
}

variable "environment" {}

variable "name" {
  default = "your_service_name"
}

variable "callback_urls" {
  type = "list"
}

variable "logout_urls" {
  type = "list"
}

variable "domain" {}

variable "google_provider_client_id" {}
variable "google_provider_client_secret" {}

variable "facebook_provider_client_id" {}
variable "facebook_provider_client_secret" {}

variable "root_domain" {
  description = "certificate root domain"
  default     = "your-example-domain.com"
}

Conclusion

There’s so much more to the story, but I think this post is long enough as it is and I don’t want to keep you any longer. If you have any questions, then please reach out to me on twitter.

I personally found the documentation around Cognito (and the various tools) to be both overwhelming and underwhelming. Not to mention confusing in places, as well as just downright frustrating at times.

Hopefully you found this short break down of AWS Cognito useful. There’s so much more still to dive into, but this should give you at least a decent starting point.

A Guide to Effective 1:1 Meetings

Mon, 04 Jun 2018 00:00:00 +0000

What is a 1:1 ?

A 1:1 (“one to one”) is a meeting between a manager and a ‘report’.

What is a ‘report’ ?

A ‘report’ is a member of staff of which the manager is responsible for. Specifically the manager is responsible for that person’s happiness at the company and their career progression.

How important is a 1:1 ?

A manager’s 1:1 meetings are their most important meeting, as the primary goal of this meeting is for the manager and the report to build a strong and trusted relationship.

If you’re a line manager, then you should always attend these meetings. Never skip or reschedule a 1:1 as that has the potential to give a bad impression upon your report (e.g. your report feels like they’re unimportant and not appreciated by the organisation).

As a manager, your most important priority is your staff.

How long should a 1:1 be?

At least thirty minutes, no less as there just isn’t enough time to really get into what’s on someones mind. If you schedule a fifteen minute 1:1, then this again suggests you don’t have time, or care enough, to fit your direct report into your schedule.

When should 1:1 meetings happen?

1:1’s should happen regularly. Typically they are scheduled once a week, although fortnightly (i.e. once every two weeks) is more practical for some people. Just remember that if you have them weekly, then that helps to keep them shorter in length (increase the length of the 1:1 depending on how far apart they’re scheduled).

ℹ️ NOTE

unless you’re an experienced engineer of many years, don’t fall into the trap of thinking “I don’t need regular 1:1’s”. You’ll be surprised to discover what tidbits of feedback and shared experience can help you progress with both your practical and soft skills.

What do we talk about in a 1:1 meeting?

If you’re the report, then quite literally anything that is on your mind.

Got an issue with someone at work? Talk about it!
Got problems at home with a new born baby? Talk about it!
Got ideas for ways to improve company communication? Talk about it!

Remember, what you talk about doesn’t always have to be centered around work. If you’re having trouble sleeping (whether that’s because you have a new born baby or you’ve hurt your back from being sat in an office chair all day), then talk about it!

The reason being is that your line manager can help expedite any tasks that help to resolve these problems. Whether it be chatting with HR and Facilitaties to get you a standing desk or a new chair to help your back, or whether they can talk to the tech lead on your team to allow you some leeway as far as the tasks you’re expected to work on.

Maybe you’re interested in taking on some extra responsibilities yourself and want to do some mentoring or looking after an intern? Your line manager can help investigate all these possibilities on your behalf. They are quite literally there to help you.

These meetings should be a safe space for you to share how you’re feeling, and that’s important because if you’re not feeling great (for whatever reason, whether it be physical or emotional issues), then that’s ultimately going to have an affect on your performance.

If you’re a line manager, then a 1:1 is your opportunity to perform weekly ‘preventive maintenance’ and to understand the overall health of your team. If you’re effective in these meetings, then you (and the company) will be rewarded with good work from your teams and you won’t have any sudden built-up frustrations turning into drama or someone resigning.

Unfortunately most 1:1’s end up being some form of status (or project) update, which isn’t good because the focus of a 1:1 is for a report to communicate with their line manager about how their doing and not how the project is going †

† that’s what project/product managers are for.

What about the silent types?

Now not everyone is going to be overly talkative, and so it’s the line manager’s responsibility to tease out the conversation.

The most common question a line manager will (or should) ask is:

“How Are You?”

But if the response is a nonchalant “Not bad”, then the line manager might have to resort to some other options for kick starting a meaningful conversation. Some examples could be:

Mini performance review
A current disaster
How to improve X

Remember, we don’t want the 1:1 to be a status/project update.

Suggestions

If you are the report, and you have nothing to say or have zero issues with any thing happening in the company (which to me would be doubtful), then just be aware that the meeting (if not handled properly) could turn into a status update and that’s ultimately a pointless use of everyones time.

So try and make an effort to care about what’s going on in the organisation, as there is usually always something that can be discussed.

If you are the line manager, then use the above three points to help guide the conversation to a meaningful topic that benefits both yourself and your direct report.

A mini performance review is great because we should always be looking at keeping staff on track with their goals. In order to do this we need to understand their motivations and what they want from life, so start the discussion there and that will help you to understand how you can guide their progress.

Discussing a current disaster is a good way to get some feedback on a topic that’s important to the company and allows you to get the perspective of someone either not involved in the problem space or who doesn’t even work in the same area of the business. This equally allows the direct report to feel included and that their opinions matter.

Lastly, choosing an area of the business where you feel could use some improvement is also a good option for opening up the discussion and having a meaningful 1:1, and it results in similar benefits as mentioned in the previous comment.

Meeting structure

Here’s the approach I’m currently taking in my 1:1 meetings…

How’s things? (in general)
Any wins this past week? (positivity boost!)
Any updates from last week?
Any team or project concerns? (can I unblock?)
Any projects you’re interested in helping with?
Any broader company wide updates I can share?
Any career progression updates? (either from me or the report)
Any feedback on my performance so far? (make me more effective)
Anything you looking forward to? (can be work related or personal)
Finish by celebrating their hard work ❤️(another positivity boost)

This structure can change depending on what I know going into the meeting, as well as what arises from discussions within the meeting.

Generally I want to be sure to take advantage of any opportunity where I can help a report to learn from mistakes and to share my own experiences as a way of expressing how I believe certain interactions/incidents could have been improved. Especially when it comes to ‘soft skills’ and communication.

I also like to understand what a person’s strong points are because this person might be able to have some greater impact on a project or team that they otherwise might not realize was an opportunity for them to be a part of. I can help them progress by getting them onto projects they wouldn’t normally consider or even know about, or start helping them to prepare and to learn new relevant skills.

Making it personal

I like to find a common interest with my report. This normally happens organically but at the very least I like to have a basic understanding of what my report enjoys doing (i.e. I take an interest in them), and once I do that I feel more of a connection and more invested in helping them succeed.

Do they like to read books in their spare time? Do they like rock climbing? etc. This helps me to keep the conversation light and informal when the meeting starts and I ask “How’s things with you?”, because if I don’t get much output (e.g. I’m dealing with a ‘quiet’ one), then I feel like I can tease out some light conversation by asking about their interests.

Generality can be misleading

If you ask “how are things?” you’ve asked a very general question, and that can sometimes be ok but it can also be too vague a question for your report to answer in a meaningful way.

So to tackle this I like to ask specific questions such as: “ok, what happened in the past week that was…”

Exciting
Frustrating
Rewarding
Challenging
Engaging
Boring

Notice I flip-flop between positive and negative toned queries

This helps me to tease out characteristics of the report and this helps me to then align their interests with what’s potentially happening within the organization.

Understanding

The following questions are useful for getting a clearer understanding of what will and not work with your report. Remember, like how a good teacher will adapt their lesson plan for different types of students (some students have different learning styles), you should adapt your 1:1 to best fit your report (yes this takes effort, but this is your responsibility, and an important one at that).

Most of these questions are taken verbatim from Camille Fournier’s awesome book “The Manager’s Path” and is strongly recommended reading!

How do you like to be praised, in public or in private?
What is your preferred method of communication for serious feedback? Do you prefer to get such feedback in writing so you have time to digest it, or are you comfortable with less formal verbal feedback?
Why did you decide to work here?
What are you excited about working here?
How do I know when you’re in a bad mood or annoyed? Are there things that always put you in a bad mood that I should be aware of? (e.g. does your report fast for a religious holiday which can make them a bit grumpy)
Are there any manager behaviours you hate?
Do you have any clear career goals that I should know about so I can help you achieve them?
Any surprises since you joined, good or bad, that I should know about? (e.g. where are my stock options? you promised me a relocation bonus and I’ve not received it)
Would you like more or less direction from me on your work?
What could I do to make you enjoy your work more?
What part of the day do you have the most energy and focus? What changes could we make to accommodate this?

Soliciting Ideas

Some questions can help the report feel like their voice is heard and that their opinions matter:

How could we change our team meetings to be more effective?
Do you feel your ideas are heard by the team and I?

Evolve your process

You try things. They fail. Tweak them as necessary. If my earlier suggestions aren’t working for you, then obviously make sure you’re comfortable with them and that you feel like you’ve given them your best effort, but if they’re not working don’t keep forging ahead. Try a different approach.

Remember, these meetings aren’t about you and your success as a manager. They are about your report and their success. Drop the ego and start thinking about the best interests of your reports and how you can leverage their best self to make the company as a whole better.

If I’m taking over reporting duties from another manager, I like to talk to the previous manager to understand what worked or didn’t work (in their opinion) when they had 1-1’s with this report. I might ignore their suggestions or I might not, but it’s good to dig into the previous manager’s experience with the report.

Track progress

Some managers like to jot down details of their reports alongside their normal work notes. If that works for you, then fine but I personally feel that you should work with a medium that allows your notes and thoughts to be easily shareable with your report.

You can continue to write private thoughts separately of course, but having a document that’s shared with the report so they can see how they’re progressing and have a place to review previous conversations is a good way for them to feel these regular conversations are providing value.

ℹ️ NOTE

this will also be useful to you (either as a line manager or as a report) when it comes time to give/receive six monthly performance reviews, as you’ll have a complete track record of all discussions.

It’s important for reports to understand that they own their career progression, and so once you’ve had an opportunity to understand your report the next step is for the report to come up with their goals and to pro-actively work through the relevant steps that lead to the realization of those goals.

As a manager, it’s your responsibility to help and support the report with this process.

Dealing with issues

If you’re a line manager, then there are two types of issues:

Mild
Serious

The first is someone just venting frustrations, the second is where they’ve reached their limits and have exploded their anger all over you.

In both cases you should:

Say nothing (for now).
Listen.
Do not attempt to problem solve (yet).
Do not comfort or try to take away their experience.

Once they have had an opportunity to say what they need to, then you can start to identify the various moving pieces of the problem (although, if they’re that angry you may find them repeating themselves and so at some point you might have to interject).

This could include discussing with them topics such as:

What the next steps are (e.g. intermediary and long term resolutions).
How we can reach each each step as quickly and efficiently as possible.
How we might prevent this issue from happening again.

If you’re the direct report and you’re struggling with an issue to the point where you’re now very angry, then this is a good example of why it’s important to have frequent, qualitative and effective 1:1’s. They allow a lot of issues to be caught ahead of time.

Thought leaders

For anyone interested in learning more I would highly recommend following these key players…

Conclusion

Hopefully you’ve found this breakdown of 1:1 meetings interesting and are able to bring some new learnings into your own 1:1’s to make them more effective and successful.

Good luck out there!

Project Management in Five Minutes

Fri, 25 May 2018 00:00:00 +0000

💡 Related information: Project Planning.
Includes specific documents needed and also an example matrix spreadsheet.

Introduction

If you’re a technical lead (aka engineering lead, lead developer, etc), then there comes a time where you’ll be put in charge of a project.

You may also already be quite familiar with the process (read: red tape) of certain types of project management: agile, scrum, lean etc.

So what do you do, follow one of those methodologies to the letter and have to suffer all the process that comes along with it? You could do.

For me, I find the following things are good enough to get you by without having to worry about whether you’re “doing scrum” (or whatever) correctly.

I suggest you take what you want and leave the rest.

But first, let’s take a moment to recognise and understand a few distinct roles that otherwise can appear to have a lot of cross-over responsibilities (this has been summarised from the great work originally published by Lara Hogan - I highly recommend you check out her work).

What Who How Why?

Product Manager: owns the story of “what”
Engineering Manager: owns the story of “who”
Engineering Lead (Tech Lead): owns the story of “how”
All Three: each person shares ownership of the story of “why”

More details… (Lara Hogan’s original post)

Project Management Checklist

This is a short checklist of things you probably should be doing…

🤔 Understand the requirements, and specifically why they are important.

If you don’t understand the project values, then you’ll have a hard time building something that benefits your users.
Does the project even align with your team’s mission statement/responsibilities? You should be focused on building things that bring value to your stakeholders.

📝 Create simple/high-level Gherkin user stories

This is one way of helping you better understand the product values and standards (there are other ways, but this is what I like using).

✅ Break down the requirements

Define milestones and manageable sub tasks.
inc. investigation time, QA (Quality Assurance) and security testing.

🔝 Prioritise tasks

You can do this either by ‘importance/impact’ (High, Medium, Low) or using ‘MSC’ (Must Should Could).
Once categorized, group related items (e.g. group together all high, medium, low items).
A table matrix can help visualise the various tasks and their importance.

Regardless of the approach you choose, you wont be able to prioritize using just a ‘problem/solution/impact’ approach. You’ll need to document and consider both ‘cost’ and ‘risk’ as part of your assessment. Here is an example document you can use.

🗒 Work through the unknowns

Do this over and over, until there is no more value to be gained in spending time on them.

🗣 Figure out who and which teams need to be consulted

Then communicate as often as practical and/or relevant.

📆 Run the project and adjust the plan as you go.

How far has the project come?
How far is it from completion?

Don’t work faster (and or harder) when you discover the team is missing deadlines (e.g. “it’s ok, I know we missed this deadline/milestone, but we’re sooooo close, let’s crank it up and get it done!”). STOP! take a breath and understand why the deadline was missed. Were there any trending patterns that are likely to repeat themselves?

📈 Track changes to requirements

Be clear about the cost of those changes.
How do these changes affect the completion?
Should we cut some existing features in order to accommodate the new work?

📊 Ensure observability and monitoring is in place.

Never release a product you can’t track out in the wild.
Don’t forget your on-call process. Ensure monitors are in place.

⛑ Define a rollout plan and what the roll back steps look like.

What systems need to be integrated with?
Notify appropriate on-call and support teams.
Rollback plan can be a spreadsheet, word doc, doesn’t matter, just have one.

❤️ Retro

What went well, what didn’t, what could we do differently in future?
Communicate with radical candor (care personally, challenge directly).

🎉 CELEBRATE!

Doesn’t matter how big or small the project, or whether it was difficult at times. Always, always, celebrate the finishing line and thank your team for their hard work.

Shielding

Don’t shield the team from issues that are arising around them (e.g. don’t think “oh, they’re stressed and missing deadlines, so I need to pretend like everything is ok” – it’s not ok). Treat the team with respect, as adults, and let them know things aren’t working and that we need to adjust the process to resolve that.

Yes, shield the team from distractions, but that is something altogether different than shielding them from bombs exploding around them. They can help you negotiate those mindfields if you let them. We’re working with adults, and they don’t need to have scary things hidden from them.

Positive Mindset

I’ve found that swapping the word “problem” for “challenge” a good thing to do in general, whether it be talking about an actual technical challenge or discussing a challenging interaction with another employee.

The subtle switch in language helps me refocus on a more positive and motivated projectory (rather than setting myself up to be in a negative mindset for the conversation).

Recognising Trends

As time goes on, you might start having issues with your team (for various reasons - we’re humans, we’re notoriously difficult creatures).

When reporting to your line manager (e.g. 1:1’s where you discuss things that are on your mind), it can be difficult sometimes to voice concerns without explicit examples.

Depending on the situation, explicit examples aren’t always possible to recall. In those cases where you have a niggling feeling something isn’t quite right but you couldn’t point to an exact moment in time where an incident occurred, then being able to see a trend of something negative happening can help you to raise it up to leadership.

Be aware of trends in people, otherwise you might find yourself in a bad situation and not sure how or why you got there in the first place. If you catch problems early enough, you can help work towards a solution that gets your team back on track.

User Stories

In case you’re not familiar with user stories…

Gherkin is plain-text with a little extra structure and is designed to be easy to learn by non-programmers, yet structured enough to allow concise description of examples to illustrate business rules in most real-world domains. – https://cucumber.io/docs/reference

I find it’s good for documentation, but it can also be helpful to some teams to use these user stories as a foundation for their own integration testing systems (although I personally wouldn’t, I prefer just using them as a simple reference for what it is we want to achieve at a high-level).

Below is an example ‘feature’, broken down into various scenarios.

Feature: User Authentication
  
  ...optional description about this feature here...

Scenario: authenticated user requesting a page
  Given I am a BuzzFeed user (internal or external)
  And I’m already signed in
  When I visit www.buzzfeed.com or www.buzzfeed.com/post
  Then I am directed to my destination page

Scenario: unauthenticated user requesting a page
  Given I am a BuzzFeed user (internal or external)
  And I’m not signed in
  When I visit www.buzzfeed.com or www.buzzfeed.com/post
  Then I am able to login with <method>

  Examples:
    | method            |
    | Facebook          |
    | Twitter           |
    | Google            |
    | Username/Password |

Scenario: unauthenticated user successful login
  Given I am a BuzzFeed user (internal or external)
  And I provide valid credentials
  When I attempt to login
  Then I am directed to my destination page

Scenario: unauthenticated user failed login
  Given I am a BuzzFeed user (internal or external)
  And I provide invalid credentials
  When I attempt to login
  Then I am presented with a login error

Scenario: unauthenticated user sign-up
  Given I am a BuzzFeed user (internal or external)
  And I am not already registered in the system
  When I visit www.buzzfeed.com/cms
  Then I am directed to a legacy sign-up flow

Engineering and Tech Recommended Reading List

Fri, 25 May 2018 00:00:00 +0000

I’ve long kept a list of books I’ve enjoyed reading, and decided I would share them as a blog post so others could benefit from their learnings.

ℹ️ NOTE

I generally keep the original gist the most up-to-date.

Algorithms

Grokking Algorithms: An illustrated guide

Best Practices

Client-Side

Clojure

Communication

Concurrency

Culture

Functional Programming

Go

The Go Programming Language

JavaScript

Management

Patterns

Mastering Regular Expressions

PHP

Practical Design Patterns in PHP

Python

Effective Python

Ruby

Shell

Classic Shell Scripting

System Design, Networking and Security

Testing

Growing Object Oriented Software Guided by Tests

Tools

Security with Python

Tue, 15 May 2018 00:00:00 +0000

Introduction

I recently implemented a Python library which acts as an abstraction layer on top of an existing security algorithm (in this case scrypt).

The motivation was for allowing teams to have a consistent experience utilising encryption (and hashing) in their applications and services without necessarily having to know the ins-and-outs of what’s important with regards to salts, key lengths etc.

ℹ️ NOTE

I always encourage people to understand what it is they’re doing, but in some cases that’s not always a practical mindset.

The library provides three functions:

generate_digest
decrypt_digest
validate_digest

Before we start looking at the three functions provided by this library/interface, let’s very briefly talk about KDF and PBKDF2.

A KDF (Key Derivation Function) accepts a message + a key, and produces a digest for its output. They are designed to be more computationally intensive than standard hashing functions, and so they make it harder to use dictionary or rainbow table style attacks (as they would require a lot of extra memory resources and become more unfeasible as an attack vector).

By default the KDF will generate a random salt (thus output is non-deterministic) and have a maximum computational time of 0.5 (although this can be overridden using a maxtime argument, as we’ll see later).

A PBKDF2 on the other hand is able to provide deterministic output (as well as the ability to specify an explicit salt value). The internal implementation will repeat its process multiple times, thus reducing the feasibility of automated password cracking attempts (similar to a KDF).

I mention both of these (KDF and PBKDF2) because the generate_digest function I’ve written is a multi-arity function that will switch implementation based upon the provided arguments in the method signature.

Originally I had two separate functions to distinguish them a bit more clearly but realised if this library is to make life easier for developers who don’t understand encryption or hashing concepts, then I need to provide a single function that intelligently handles things internally.

Because KDF accepts a key and is able to return the original message (given the same key) it’s acting as a form of symmetrical encryption, whereas a PBKDF2 is more like a one-way hash function. Hence I named the function in this library generate_digest rather than something like encrypt_message which wouldn’t have made sense when dealing with PBKDF2.

generate_digest

This is a multi-arity function that will generate a digest using either a password-based key derivation function (KDF) or a PBKDF2 depending on the input given.

If a password argument is provided, then KDF will be used (along with a random salt) to generate a non-deterministic digest.

If a salt is provided, then a PBKDF2 will be used to generate a deterministic digest.

ℹ️ NOTE

salts should be a minimum of 128bits (~16 characters) in length. Also, when specifying a maxtime with generate_digest, ensure you include that same value when decrypting with decrypt_digest or validating via validate_digest.

decrypt_digest and validate_digest

The decrypt_digest and validate_digest functions only apply to digests that have been generated using a password (i.e. KDF). Given the right password decrypt_digest will return the original message, and thus is considered more a form of symmetrical encryption than a straight one-way hash function. The validate_digest function will return a boolean true or false if the given password was able to decrypt the message.

Dependencies

This abstraction library requires scrypt, which itself requires the following dependencies to be installed within the context of your service: build-essential, libssl-dev, and python-dev. If your service has a Dockerfile, adding these dependencies should be as simple as adding a line like the following:

RUN apt-get update && apt-get install -y build-essential libssl-dev python-dev

Usage

I suggest looking at the test suite (see below) to get an idea of how you would use the functions in this library.

ℹ️ NOTE

for a glossary of security terms, refer to this document.

Tests

Before we look at the implementation of the library, let’s take a moment to sift through its test suite.

ℹ️ NOTE

I named the library secure and have it running on a private PyPy instance. This code is made available via GitHub.

import pytest

from secure.interface import ArgumentError, generate_digest, validate_digest, decrypt_digest


message = "my-message"
password = "my-password"
salt = "my-salt-is-long-enough"


def test_generate_digest_with_both_a_password_and_a_salt():
    """Providing both a password and a salt should raise an exception."""

    with pytest.raises(ArgumentError):
        generate_digest(message, salt=salt, password=password)


def test_generate_digest_with_a_password():
    """Generating a digest with a password should be non-deterministic."""

    digest1 = generate_digest(message, password=password)
    digest2 = generate_digest(message, password=password)
    digest3 = generate_digest(message, password=password, maxtime=1.5)
    digest4 = generate_digest(message, password=password, maxtime=1.5)
    digest5 = generate_digest(message, password=password, maxtime=int(1))
    digest6 = generate_digest(message, password=password, maxtime=int(1))

    assert digest1 != digest2
    assert digest3 != digest4
    assert digest5 != digest6


def test_generate_digest_without_a_password():
    """Generating a digest without a password should be deterministic."""

    digest1 = generate_digest(message)
    digest2 = generate_digest(message)
    digest3 = generate_digest(message, salt=salt)
    digest4 = generate_digest(message, salt=salt)
    digest5 = generate_digest(message, length=128)
    digest6 = generate_digest(message, length=128)

    assert digest1 == digest2
    assert digest3 == digest4
    assert len(digest5) == len(digest6)


def test_generate_digest_with_different_salt_lengths():
    """Salts should be at least 128bits (~16 characters) in length."""

    generate_digest(message, salt=salt)

    with pytest.raises(ArgumentError):
        generate_digest(message, salt="too-short")

def test_validate_digest():
    """Validation only applies to digests generated with a password."""

    digest1 = generate_digest(message, password=password)
    digest2 = generate_digest(message, password=password)
    digest3 = generate_digest(message, password=password, maxtime=1.5)
    digest4 = generate_digest(message, password=password, maxtime=1.5)
    digest5 = generate_digest(message, password=password, maxtime=int(1))
    digest6 = generate_digest(message, password=password, maxtime=int(1))

    assert not validate_digest(digest1, 'incorrect-password')
    assert validate_digest(digest1, password)
    assert validate_digest(digest3, password, maxtime=1.5)
    assert validate_digest(digest5, password, maxtime=int(1))


def test_decrypt_digest():
    """Decryption is possible given the right password."""

    digest = generate_digest(message, password=password)

    assert decrypt_digest(digest, password) == message

Implementation

OK, time to see the library code itself.

ℹ️ NOTE

I like to use MyPy for type hinting.

import scrypt

from typing import Union


class ArgumentError(Exception):
    pass


def generate_digest(message: str,
                    password: str = None,
                    maxtime: Union[float, int] = 0.5,
                    salt: str = "",
                    length: int = 64) -> bytes:
    """Multi-arity function for generating a digest.

    Use KDF symmetric encryption given a password.
    Use deterministic hash function given a salt (or lack of password).
    """

    if password and salt:
        raise ArgumentError("only provide a password or a salt, not both")

    if salt != "" and len(salt) < 16:
        raise ArgumentError("salts need to be minimum of 128bits (~16 characters)")

    if password:
        return scrypt.encrypt(message, password, maxtime=maxtime)
    else:
        return scrypt.hash(message, salt, buflen=length)


def decrypt_digest(digest: bytes,
                   password: str,
                   maxtime: Union[float, int] = 0.5) -> bytes:
    """Decrypts digest using given password."""

    return scrypt.decrypt(digest, password, maxtime)


def validate_digest(digest: bytes,
                    password: str,
                    maxtime: Union[float, int] = 0.5) -> bool:
    """Validate digest using given password."""

    try:
        scrypt.decrypt(digest, password, maxtime)
        return True
    except scrypt.error:
        return False

Conclusion

Let me know what you think on twitter. Have fun.

Static Search With Lunr.js

Thu, 10 May 2018 00:00:00 +0000

Introduction

You have a statically generated website (like mine!) and you want to implement some kind of search facility that is:

Free
Doesn’t Suck (e.g. no ads or iframe)
Quick (i.e. no server-side communication)

The solution is to use Lunr.js.

You might be wondering about Lunr’s origins? Well, it’s based loosely on the idea made popular by Solr, which is an open-source search platform built on a Java library called Lucene.

Since then we’ve also seen the release and rise of ElasticSearch, which is an open-source, distributed, and RESTful search engine built on top of the Apache Lucene library.

Background

I’ll explain this from the perspective of Hugo, which is the static site generator I use to produce this website. Hugo stores the metadata for each post (e.g. title, date, categories, tags etc) in something it calls Front Matter.

I use YAML, but you can use JSON or TOML, and that’s important to note because the implementation I use is based on my metadata being in YAML format. So if yours is JSON or TOML, for example, then you’ll need to modify the code shown in this post to reflect your use case.

Below is the ‘front matter’ for this post you’re reading.

---
title: "Static Search With Lunr.js"
date: 2018-05-10T16:54:08+01:00
categories:
  - "code"
  - "development"
  - "guide"
  - "search"
tags:
  - "elasticsearch"
  - "javascript"
  - "js"
  - "lunr"
  - "solr"
  - "static"
draft: false
---

What typically follows the front matter is the content of your post.

Tasks

OK, so here are the tasks we have to get a working solution:

Generate a search index.json file.
Create a search HTML page.
Write some JavaScript to load the index JSON and populate Lunr.
Write some more JavaScript to accept user’s input.
Write even more JavaScript to query your index JSON.

ℹ️ NOTE

I’m not a JavaScript fan, but needs must.

Setup

Step 1, generate an index.json file. To do that I’m going to use Grunt because it ties easily into NPM’s package.json format and luckily for me someone had already done a lot of the (no pun intended) ‘grunt’ work and I just needed to modify the code to suit my needs.

Here’s the relevant portion of the package.json which we run with npm run dev:

"scripts": {
  "index": "hugo && grunt lunr-index",
  "dev": "npm run index && hugo server"
}

Here is the lunr-index Gruntfile task that’s executed:

var yaml = require("js-yaml");
var S = require("string");

var CONTENT_PATH_PREFIX = "content";

module.exports = function(grunt) {

    grunt.registerTask("lunr-index", function() {

        var indexPages = function() {
            var pagesIndex = [];
            grunt.file.recurse(CONTENT_PATH_PREFIX, function(abspath, rootdir, subdir, filename) {
                pagesIndex.push(processMDFile(abspath, filename));
            });

            return pagesIndex;
        };

        var processMDFile = function(abspath, filename) {
            var content = grunt.file.read(abspath);
            var pageIndex;

            // separate the Front Matter from the content and parse it
            content = content.split("---");

            var frontMatter;
            try {
                frontMatter = yaml.load(content[1]);
            } catch (e) {
                grunt.log.writeln(e.message);
            }

            var href = S(abspath).chompLeft(CONTENT_PATH_PREFIX).chompRight(".md").s;

            if (filename === ".DS_Store") {
              return
            }

            if (filename === "_index.md") {
                href = "/"
            }
            var m = abspath.match(/^content\/page\/(.+)\.md/);
            if (m != null) {
              href = "/" + m[1]
            }

            // build Lunr index for this page
            pageIndex = {
                title: frontMatter.title,
                tags: frontMatter.tags,
                href: href.toLowerCase(),
                content: S(content[2]).stripTags().stripPunctuation().s
            };

            return pageIndex;
        };

        grunt.file.write("static/js/lunr/index.json", JSON.stringify(indexPages()));
    });
};

The key part to that Gruntfile, other than the parsing out of the metadata (front matter), is where we store it: static/js/lunr/index.json. That location is something that’s included in Hugo’s build step and so when I’m working locally on a new post I’ll use npm run dev instead of hugo server because I’m guaranteed to generate a new search index based on the latest blog content I’ve just added and that the file will always be available when my static code is deployed.

Step 2, create a search page:

---
title: Search
description: Lookup articles of interest.
comments: false
menu: main
weight: -170
---

<p><input id="search" type="text" placeholder="type something here"></p>

<ul id="results"></ul>

<script src="https://code.jquery.com/jquery-2.1.3.min.js"></script>
<script src="https://unpkg.com/lunr/lunr.js"></script>
<script>
  ...script here...
</script>

Step 3, 4, 5, lots of JS…

var lunrIndex,
    $results,
    documents;

function initLunr() {
  // retrieve the index file
  $.getJSON("../../js/lunr/index.json")
    .done(function(index) {
        documents = index;

        lunrIndex = lunr(function(){
          this.ref('href')
          this.field('content')

          this.field("title", {
              boost: 10
          });

          this.field("tags", {
              boost: 5
          });

          documents.forEach(function(doc) {
            try {
              this.add(doc)
            } catch (e) {}
          }, this)
        })
    })
    .fail(function(jqxhr, textStatus, error) {
        var err = textStatus + ", " + error;
        console.error("Error getting Lunr index file:", err);
    });
}

function search(query) {
  return lunrIndex.search(query).map(function(result) {
    return documents.filter(function(page) {
      try {
        console.log(page)
        return page.href === result.ref;
      } catch (e) {
        console.log('whoops')
      }
    })[0];
  });
}

function renderResults(results) {
  if (!results.length) {
    return;
  }

  // show first ten results
  results.slice(0, 10).forEach(function(result) {
    var $result = $("<li>");

    $result.append($("<a>", {
      href: result.href,
      text: "» " + result.title
    }));

    $results.append($result);
  });
}

function initUI() {
  $results = $("#results");

  $("#search").keyup(function(){
    // empty previous results
    $results.empty();

    // trigger search when at least two chars provided.
    var query = $(this).val();
    if (query.length < 2) {
      return;
    }

    var results = search(query);

    renderResults(results);
  });
}

initLunr();

$(document).ready(function(){
  initUI();
});

The key part of the above JS is the bit after we’ve retrieved the search index.json file, as this is what takes the search index file and uses it to populate Lunr:

lunrIndex = lunr(function(){
  this.ref('href')
  this.field('content')

  this.field("title", {
      boost: 10
  });

  this.field("tags", {
      boost: 5
  });

  documents.forEach(function(doc) {
    try {
      this.add(doc)
    } catch (e) {}
  }, this)
})

…and that’s it really.

Interview Topics

Sun, 08 Apr 2018 00:00:00 +0000

UPDATE: I moved on from BuzzFeed October 2020 and Joined Fastly.
See my resume for details.

Introduction

An organisation reached out to me about whether I would like to work for them. The focus of this post isn’t that company (hence I’ve not mentioned their name), but more about a set of experiences I’ve had when changing jobs and how I think they’re important to keep in mind, along with specific topics of interest you should look to include, when considering future opportunities.

What Happened?

A recruiter, let’s say they were calling on behalf of the “Foo” company, contacted me to say they (and their CTO) had stumbled across and enjoyed a selection of my blog posts, and found my profile on LinkedIn to fit the candidate they were looking for.

My current work situation

I work at a company called BuzzFeed.

I’ve been here for almost two years now, and I came to them from the BBC.

Am I happy at BuzzFeed?

Yes. 🙂

BuzzFeed is the first company I’ve worked for where they:

Treat me fair.
Treat me like a human being.
Support my need to work remotely.
Are truly diverse and culturally supportive.

As far as interviewing, BuzzFeed also got brownie points for…

Paying me for my time interviewing with them.
Paying for my accommodation and travel to NY (I live in the UK).
Not taking advantage of me †

† one example being, they already knew my salary and still increased it by a significant percentage

Did you talk to the Foo company?

Yes. I wanted to see how they compared.

How did they compare?

Well, I didn’t get past the initial conversation because they didn’t have a “remote role” for me, so I couldn’t say with 100% certainty how they would have compared.

But that said, there were some key aspects I took away from the conversation that I feel are worth mentioning, as these were specifically firing my internal alarm bell…

They tried to sell me on “we’re all engineers here”, which for me isn’t something we should be aspiring to (conversation for another day, but to me it’s an unhealthy attitude).
There was a seeming lack of post-mortems and/or blameless culture. Their response, if you can imagine it, was like a confused raised eye brow.
They confused the concept of teams functioning differently to each other (which is fine) with inconsistency and lack of direction.
Money focused. The biggest takeaway by far was they asked constantly “What would it take for you to leave BuzzFeed?” †.
No interest in techniques that could help improve their ability to work more efficiently.

† I realised they cared less about me as a ‘person’, someone the company could build a long term relationship with, and more about me as a ‘number’. I was a resource, an entity they could own.

What was the problem?

This company is (as they themselves stated) entirely made up of engineers, with the lightest touch of management. Through my discussion it became obvious that building new products with new tech was the focus and drive for those working there. But in that environment, being able to work effectively, efficiently and with good communication (along with making sure you’re working on the right thing and at the right time) is not usually the case (in my experience).

For me, I’m at a point in my career where discussions about programming languages or tech stack X isn’t something I worry about too much, or get giddy about, because most of the time it’s how you solve a problem that’s important/interesting.

In most situations, if you do your due diligence and exhaust all means available to you to make the given tech work, you should be able to empirically prove it’s not working, and thus are in a better position to justify a more appropriate language or tech stack. This all becomes trivial.

ℹ️ NOTE

in my experience, people tend to fall back on subjective opinions and hand wavey ‘facts’ rather than cracking on with solving the problem at hand.

Remember, we’re engineers, we can solve these issues in different ways. But I personally find the ‘human’ problem is something that is typically much harder to solve and ultimately will cause a lot more detrimental effect on our ability to learn, progress in, and emotionally stay connected to our jobs.

After my conversation with the Foo company I already had an unsettling feeling about them, but decided that I would do some research on Glassdoor to see if my gut feelings, stemming from the feedback I received, were correlating to other people’s past experiences working at the company. †

† it’s important to realise that with sites like Glassdoor you’re getting half the story, so take it with a pinch of salt. For me, I already had a lengthy conversation with the company and so I felt like I had enough information on which to corroborate my experience against.

Turns out my gut instincts right. Others, and by that I mean: engineers either working still (or who used to work), at Foo had voiced concerns that if you want to get paid lots of money to work on software with lots of tech debt, and with management that doesn’t care about improving the situation with how teams collaborate and/or communicate with each other, then company Foo might be the place for you.

Now, what I’m about to say, I can say because of privilege (I’m totally aware of that and I try to keep that in check wherever possible)…

For me, getting paid is important (because ultimately it allows me to look after my family), but in truth there’s more to life than getting paid which is (if not more) important for being happy.

I’ve been fortunate enough with my privilege to be able to have that opinion, and I appreciate that for others this isn’t necessarily going to be the case, and so for those of you who don’t feel that way because of circumstances: that’s of course ok and valid. So with that in mind, I will cover what I feel are those other important topics next…

What’s important to you?

First, I want to take a moment to re-iterate, that you need to find what’s important to you.

If you’re starting out in your career, or your mid-way through, or you’re 20+ years in like me, then your priorities, your standards, and your values are going to be different depending on the stage of life you’re at.

For me (in a general sense), I’m looking for:

A well paid job (to enable me to support my family)
A stable work environment (again, family)
A diverse cultural environment (because diversity is important to me)
To be able to have impact (I like helping people)
To be happy (i.e. not stressed)

Now let’s be honest, who of us isn’t looking for those qualities? Everyone wants a good paying job, everyone wants to be happy and in some cases most people prefer stability over chaos. Maybe diversity, or having an impact, isn’t high up on your list of things because you’re just starting out and to be honest, at that stage in life just getting paid is probably going to be your highest priority.

At any rate, figure out your priorities, your standards and your values, when considering working opportunities.

But also remember that these perspectives will change over time, and that’s ok too (sometimes we need stability in a fragile industry and nothing else).

What should you talk about?

It’s a good idea, at the start of any conversation, to ask:

What’s the focus of today’s conversation?

i.e. What’s important to the company (or individual) you’re talking with, and what insights do they want to take from this? Does it align with your agenda? If not, clarify what you hope to get out of the conversation, so it’s clear (on both sides) what intentions there are.

Otherwise, here are the list of questions/topics I like to cover…

What is the company’s story?
- Give me an insight into the company’s history/background.
Presence
- What number of offices do you have and what are their locality?
Culture and Diversity
- The company’s “values” will highlight what’s important to them.
Hierarchy and Organisation Structure
- Are you ‘flat and lean’ or ‘tall and fragmented’?
Visibility & Openness
- How does leadership share and handle critical/internal business topics?
Collaboration
- Building new features across different cross-discipline teams (avoiding duplication)
- Distributed Timezones
Community
- How do internal & external staff interact?
- How do teams across offices/locality bond?
Process
- What’s your ‘onboarding’ process?
- How are new features discussed, designed, evolved, released?
  - How do you deterimine a priority (e.g. impact/value for end-users)?
- What is the leadership like?
- How is documentation handled here?
  - Do they care about documentation?
  - Are they sharing information, or is there silos of knowledge?
- Do you practice post mortems? (blameless retrospection, handling of failures)
- How do you handle tech debt (i.e. sustainability of your software)?
Communication
- What do you consider to be good communication?
- Do you work with any of the following practices?
  - SBI: Situation, Behaviour, Impact (Methodology, The Center for Creative Leadership)
  - Radical Candor (Book, Kim Scott)
  - Conscious Business (Video, Fred Kofman)
  - Authentic Communication (Book, Fred Kofman)
Responsibility and Ownership
- What are your expectations of me when I start?
  - What is your indicator that I was a successful hire?
- What does the responsibilities look like for each of the following types of teams?
  - Engineering (Features/Products)
  - Site Reliability
  - Operations
- How do teams work with regards to an ‘on-call’ rota?
Job Progression & Opportunities
- Learning budgets?
- Conferences?
Remote Working
- How are remotes kept feeling included?
Work life balance and support
- How do staff balance their work-life?
- How do you work in a multi-region/distributed organisation?
  - e.g. cross-over hours between UK and US?
Tech
- Finally we get to something engineering related…
  - What does the infrastructure look like?
  - What build systems do you use?
  - What is your deployment platform?
  - How often do/can you deploy?
  - How do you handle rollback processes?
Vision and Future
- Where is this company going?
- What are the end goals?

Conclusion

I think that pretty much covers everything of interest. To me at least these broad topics will give you a good indicator of what an organisation is like (or the potential they have), depending on how they answer these questions.

What do you ask in an interview? What matters to you? Let me know on twitter.

Go Reverse Proxy

Sat, 03 Mar 2018 00:00:00 +0000

Introduction

I was struggling to find a good (or just simple) reverse proxy solution written in Go, so I decided to take what I had learnt from a work colleague of mine and put together a simple example for others to build upon if they needed a quick reference point.

In this example I have an origin server written in Python (for no other reason than to have a clearer distinction between the proxy and the origin) and which supports the endpoints /, /foo and /bar/* (where the wildcard glob means we support multiple variants of that, such as /bar/baz).

Each origin handler will print the http request headers, followed by sending a response body that correlates to the handler name (so for example, the FooHandler class will respond with FOO!, while the BarHandler class will response with BAR!).

Example Python Origin Code

Here is our Python code using the Tornado web framework.

import tornado.ioloop
import tornado.web


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        print("MAIN HEADERS:\n\n", self.request.headers)
        self.write("MAIN!")


class FooHandler(tornado.web.RequestHandler):
    def get(self):
        print("FOO HEADERS:\n\n", self.request.headers)
        self.write("FOO!")


class BarHandler(tornado.web.RequestHandler):
    def get(self):
        print("BAR HEADERS:\n\n", self.request.headers)
        self.write("BAR!")


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
        (r"/foo", FooHandler),
        (r"/bar.*", BarHandler),
    ])


if __name__ == "__main__":
    app = make_app()
    app.listen(9000)
    tornado.ioloop.IOLoop.current().start()

Example Golang Proxy Code

There are two versions of the code, a simple version and a more advanced version that aims to handle more specific use cases.

The simple version uses just the Go standard library, whereas the advanced version uses the standard library as well as a few a few external packages such as httprouter and logrus for routing and logging respectively.

One difference between them that’s worth mentioning is that in the simple version we use the httputil.ReverseProxy http handler directly, whereas in the advanced version we use httputil.NewSingleHostReverseProxy to construct this for us. The advanced version also tries to normalise the paths by stripping trailing slashes and joining them up with the base path (if there was one, although ironically I don’t define one in the advanced example).

Simple

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    origin, _ := url.Parse("http://localhost:9000/")

    director := func(req *http.Request) {
        req.Header.Add("X-Forwarded-Host", req.Host)
        req.Header.Add("X-Origin-Host", origin.Host)
        req.URL.Scheme = "http"
        req.URL.Host = origin.Host
    }

    proxy := &httputil.ReverseProxy{Director: director}

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        proxy.ServeHTTP(w, r)
    })

    log.Fatal(http.ListenAndServe(":9001", nil))
}

Advanced

package main

import (
    "net/http"
    "net/http/httputil"
    "net/url"
    "os"
    "strings"

    "github.com/Sirupsen/logrus"
    "github.com/julienschmidt/httprouter"
)

func singleJoiningSlash(a, b string) string {
    aslash := strings.HasSuffix(a, "/")
    bslash := strings.HasPrefix(b, "/")
    switch {
    case aslash && bslash:
        return a + b[1:]
    case !aslash && !bslash:
        return a + "/" + b
    }
    return a + b
}

func main() {
    logrus.SetFormatter(&logrus.TextFormatter{})
    logrus.SetOutput(os.Stdout)
    logrus.SetLevel(logrus.InfoLevel)
    logger := logrus.WithFields(logrus.Fields{
        "service": "go-reverse-proxy",
    })

    router := httprouter.New()
    origin, _ := url.Parse("http://localhost:9000/")
    path := "/*catchall"

    reverseProxy := httputil.NewSingleHostReverseProxy(origin)

    reverseProxy.Director = func(req *http.Request) {
        req.Header.Add("X-Forwarded-Host", req.Host)
        req.Header.Add("X-Origin-Host", origin.Host)
        req.URL.Scheme = origin.Scheme
        req.URL.Host = origin.Host

        wildcardIndex := strings.IndexAny(path, "*")
        proxyPath := singleJoiningSlash(origin.Path, req.URL.Path[wildcardIndex:])
        if strings.HasSuffix(proxyPath, "/") && len(proxyPath) > 1 {
            proxyPath = proxyPath[:len(proxyPath)-1]
        }
        req.URL.Path = proxyPath
    }

    router.Handle("GET", path, func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
        reverseProxy.ServeHTTP(w, r)
    })

    logger.Fatal(http.ListenAndServe(":9001", router))
}

Demonstration

In order to run this example you should follow these instructions:

run the tornado application (e.g. python tornado-origin.py)
run the go application (e.g. go run main.go)
make http requests (shown below)

curl -v http://localhost:9001/
curl -v http://localhost:9001/foo
curl -v http://localhost:9001/foo/
curl -v http://localhost:9001/bar/baz

You should see output from the Python server that looks something like this:

FOO HEADERS:

Host: localhost:9001
User-Agent: curl/7.54.0
Accept: */*
X-Forwarded-Host: localhost:9001
X-Origin-Host: localhost:9000

Explanation

OK, so let’s step through the main function of the advanced example code to see what’s going on.

The core reverse proxy code and its concepts are effectively the same between the advanced and simple versions.

First we set up our basic logging configuration:

logrus.SetFormatter(&logrus.TextFormatter{})
logrus.SetOutput(os.Stdout)
logrus.SetLevel(logrus.InfoLevel)
logger := logrus.WithFields(logrus.Fields{
  "service": "go-reverse-proxy",
})

Next we create a new httprouter instance, we define the origin host (http://localhost:9000/) and the ‘pattern’ we want httprouter to look out for (/*catchall, which is a special syntax that represents a catchall wildcard/glob):

router := httprouter.New()
origin, _ := url.Parse("http://localhost:9000/")
path := "/*catchall"

Next we create a new reverse proxy instance, passing it the origin host (http://localhost:9000/):

reverseProxy := httputil.NewSingleHostReverseProxy(origin)

Followed by configuring the ‘director’ for the reverse proxy. The director is simply a function that modifies the received incoming request, while the response from the origin is copied back to the original client.

In this example, we attach a few common proxy related headers to the incoming request and then modify its Scheme/Host to reflect the origin we wish to proxy it onto.

Next, we change the request path to the origin. What we do is ensure the path we request from the origin is whatever the base origin path is + the requested path (i.e. not just directing the request to the root/entrypoint of the origin).

In our example, our origin’s path is just / whereas the client will be requesting things like /foo and /bar/baz, so these would be appended to the origin’s defined /. But we also make sure that when joining the origin’s path with the incoming request path, that we avoid double slashes in the middle.

Lastly, we ensure that any trailing slash is removed as well:

reverseProxy.Director = func(req *http.Request) {
  req.Header.Add("X-Forwarded-Host", req.Host)
  req.Header.Add("X-Origin-Host", origin.Host)
  req.URL.Scheme = origin.Scheme
  req.URL.Host = origin.Host

  wildcardIndex := strings.IndexAny(path, "*")
  proxyPath := singleJoiningSlash(origin.Path, req.URL.Path[wildcardIndex:])
  if strings.HasSuffix(proxyPath, "/") && len(proxyPath) > 1 {
    proxyPath = proxyPath[:len(proxyPath)-1]
  }
  req.URL.Path = proxyPath
}

Finally, we setup the handler for the /*catchall httprouter path. In this case we don’t do anything other than call the reverse proxy’s ServeHTTP method and pass it the original ResponseWriter and http Request. We then kick start the httprouter using ListenAndServe:

router.Handle("GET", path, func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
  reverseProxy.ServeHTTP(w, r)
})

logger.Fatal(http.ListenAndServe(":9001", router))

Handling Errors

In order to handle errors the reverse proxy needs to construct a new response object, which means if you wanted the error response you generate to have all the same response headers as were provided by the upstream service, then you’d have to programmatically add those to the new response object.

Let’s see how we can handle errors in the basic sense, just to get an idea for how the code looks. Now we don’t need to use another programming language to do this, we can do it all in Go (we could have done this earlier instead of using Python, but I wanted to highlight how you could use another language if you wanted).

package main

import (
    "errors"
    "fmt"
    "io/ioutil"
    "log"
    "net"
    "net/http"
    "net/http/httptest"
    "net/http/httputil"
    "net/url"
    "strings"
    "time"
)

// copied from https://golang.org/src/net/http/httputil/reverseproxy.go?s=3330:3391#L98
func singleJoiningSlash(a, b string) string {
    aslash := strings.HasSuffix(a, "/")
    bslash := strings.HasPrefix(b, "/")
    switch {
    case aslash && bslash:
        return a + b[1:]
    case !aslash && !bslash:
        return a + "/" + b
    }
    return a + b
}

func main() {
    backendServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "backend server handled the request!")
    }))
    defer backendServer.Close()

    backendServerURL, err := url.Parse(backendServer.URL)
    if err != nil {
        log.Fatal(err)
    }

    proxy := &httputil.ReverseProxy{
        Director: func(r *http.Request) {
            r.URL.Scheme = backendServerURL.Scheme
            r.URL.Host = backendServerURL.Host
            r.URL.Path = singleJoiningSlash(backendServerURL.Path, r.URL.Path)
        },
        Transport: &http.Transport{
            Dial: (&net.Dialer{
                Timeout: 10 * time.Second,
            }).Dial,
        },
        ModifyResponse: func(r *http.Response) error {
            // return nil
            //
            // purposefully return an error so ErrorHandler gets called
            return errors.New("uh-oh")
        },
        ErrorHandler: func(rw http.ResponseWriter, r *http.Request, err error) {
            fmt.Printf("error was: %+v", err)
            rw.WriteHeader(http.StatusInternalServerError)
            rw.Write([]byte(err.Error()))
        },
    }

    frontendServer := httptest.NewServer(proxy)
    defer frontendServer.Close()

    resp, err := http.Get(frontendServer.URL)
    if err != nil {
        log.Fatal(err)
    }

    b, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("\n\nbody: \n%s\n\n", b)
}

Here we can see that our httputil.ReverseProxy#ModifyResponse function is hardcoded to return an error type, which then causes the httputil.ReverseProxy#ErrorHandler function to be called.

From there we use the http.ResponseWriter to create a new response. In this case we do nothing other than print the original error, but you could do pretty much anything you like at this point.

If you needed the original response object that came from the upstream then you’d need to make sure the error you returned from ModifyResponse was a custom error type so that you attach a field such as OriginalResponse to it and thus assign it the original http.Response that was available to you within ModifyResponse.

NGINX-Lite (not-really)

Below is an example that demonstrates using httpbin as our origin.

Specifically we use its /anything endpoint, which allows you to provide any value as the final path segment. So for example, /anything/foo or /anything/beep, both work with the httpbin.org service.

package main

import (
    "fmt"
    "log"
    "net/http"
    "net/http/httputil"
)

func main() {
    proxy := &httputil.ReverseProxy{Director: func(req *http.Request) {
        originHost := "httpbin.org"
        originPathPrefix := "/anything"

        req.Header.Add("X-Forwarded-Host", req.Host)
        req.Header.Add("X-Origin-Host", originHost)
        req.Host = originHost
        req.URL.Scheme = "https"
        req.URL.Host = originHost
        req.URL.Path = originPathPrefix + req.URL.Path

        fmt.Printf("final request\n\n %+v \n\n", req)
    }}

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        proxy.ServeHTTP(w, r)
    })

    log.Fatal(http.ListenAndServe(":9001", nil))
}

Now let’s elaborate on this example a little bit and ensure that our reverse proxy has a client timeout specified (see this article for details as to why you would want to do this).

We also use gorilla/mux as it supports utilising regular expression path matching (we could do this ourselves, but using a library in this case helps to keep the code we have to write down).

One last thing you’ll notice is that we’re using a configuration object that allows us to configure override behaviour. For example, if our request includes a HTTP header of X-BF-Testing and its value is integralist, then we’ll proxy the request to a different endpoint.

You can do more complex things if necessary, but this gives you a good idea of how to replicate something like NGINX with very little code (obviously to replicate something like NGINX is waaay beyond the scope of this post) 😉

package main

import (
    "log"
    "net"
    "net/http"
    "net/http/httputil"
    "time"

    "github.com/gorilla/mux"
)

type override struct {
    Header string
    Match  string
    Host   string
    Path   string
}

type config struct {
    Path     string
    Host     string
    Override override
}

func generateProxy(conf config) http.Handler {
    proxy := &httputil.ReverseProxy{Director: func(req *http.Request) {
        originHost := conf.Host
        req.Header.Add("X-Forwarded-Host", req.Host)
        req.Header.Add("X-Origin-Host", originHost)
        req.Host = originHost
        req.URL.Host = originHost
        req.URL.Scheme = "https"

        if conf.Override.Header != "" && conf.Override.Match != "" {
            if req.Header.Get(conf.Override.Header) == conf.Override.Match {
                req.URL.Path = conf.Override.Path
            }
        }
    }, Transport: &http.Transport{
        Dial: (&net.Dialer{
            Timeout: 5 * time.Second,
        }).Dial,
    }}

    return proxy
}

func main() {
    r := mux.NewRouter()

    configuration := []config{
        config{
            Path: "/{path:anything/(?:foo|bar)}",
            Host: "httpbin.org",
        },
        config{
            Path: "/anything/foobar",
            Host: "httpbin.org",
            Override: override{
                Header: "X-BF-Testing",
                Match:  "integralist",
                Path:   "/anything/newthing",
            },
        },
    }

    for _, conf := range configuration {
        proxy := generateProxy(conf)

        r.HandleFunc(conf.Path, func(w http.ResponseWriter, r *http.Request) {
            proxy.ServeHTTP(w, r)
        })
    }

    log.Fatal(http.ListenAndServe(":9001", r))
}

Conclusion

That’s all there is to it.

You could also wrap the function passed to router.Handle in a middleware function so that you’re able to do extra processing. A common example of this is to authenticate the incoming request before it is proxied to the origin (meaning you can reject the request if necessary).

Hashing, Encryption and Encoding

Fri, 16 Feb 2018 00:00:00 +0000

Introduction

I’ve written previously (and in-depth) on the subject of security basics, using tools such as GPG, OpenSSH, OpenSSL, and Keybase. But this time I wanted to focus in on the differences between encryption and hashing, whilst also providing a slightly more concise reference point for those already familiar with these concepts.

Terminology

OK, so using the correct terminology is essential and helps us to be explicit and clear with what we really mean.

hash function:
calculates a deterministic, irreversible, fixed-size alphanumeric string (based on input).
message:
a message is the data (e.g. the ‘input’ provided to a hash function).
digest:
the hexidecimal output generated by a hash function (contextually referred to as “checksum” or “fingerprint”).
symmetric algorithm:
a cryptographic algorithm that uses the same key to encrypt and decrypt data.
asymmetric algorithm:
a form of encryption where keys come in pairs (what one key encrypts, only the other can decrypt).
integrity:
the message transported has not been tampered with or altered.
confidentiality:
the communication between trusted parties is confidential.
authenticity:
the communication is with who you expect it to be (not a man-in-the-middle).

For a longer “Security Glossary”, please see this Google doc I created.

Hashing vs Encryption

In essence:

hashing: provides integrity.
encryption: provides confidentiality.

Often cryptographic primitives need to be combined. For example, public-key cryptography uses RSA (a slow, but very secure algorithm) for communicating securely, while internally using AES (a faster, but less secure algorithm †) for encrypting data with a shared key, while using a hash function for generating a message digest to ensure both parties can verify the integrity of the payload sent/received.

† less secure in the sense that you have to share a secret key with the person you wish to communicate with, but that’s what public-key cryptography helps to secure.

Why use a hash function?

Hash functions (or more specifically their output: digests) can be used for many things, like indexing data in a hash table, fingerprinting (i.e. detecting duplicate data or uniquely identifying files), or as a checksum (i.e. detecting data corruption).

Message authentication (i.e. message integrity) involves hashing the message to produce a digest and encrypting the digest with the private key to produce a digital signature.

In order to verify this ‘signature’ the recipient of the encrypted message would need to compute a hash of the message, then decrypting the signer’s public key and comparing the computed digest against the decrypted digest sent within the encrypted message.

If the digest you generated is the same as the decrypted digest, then we can be sure the message was delivered unmodified whilst in transit (e.g. ‘man-in-the-middle’).

Base64 Encoding

Base64 is a way of taking binary data and transforming it into a text-based format. It is commonly used when there is a need to transfer the binary data over a medium that only supports textual data (e.g. you can Base64 encode images so they can be inlined into HTML).

How it works: Base64 encoding takes three bytes, each consisting of eight bits, and represents them as four printable characters in the ASCII standard.

ℹ️ NOTE

Base64 encoded strings are NOT secure.
Remember, it encodes data, not encrypt it.

MAC vs HMAC

A ‘MAC’ (Message Authentication Code) uses symmetrical cryptography with an encryption algorithm (such as AES †) to verify the integrity of a message, whereas a ‘HMAC’ will use a hash function (such as SHA256) internally instead of an encryption algorithm.

† encryption algorithms: AES (Advanced Encryption Standard), Blowfish, DES (Data Encryption Standard), Triple DES, Serpent, and Twofish.

Below is an example HMAC written in Bash and using the OpenSSL command-line tool.

function hmac {
  digest="$1"
  data="$2"
  key="$3"
  shift 3
  echo -n "$data" | openssl dgst "-$digest" -hmac "$key" "$@"
}

The way you would use it is as follows:

hmac sha256 "message to be hashed" secret-key

ℹ️ NOTE

you can swap sha256 for any supported digest algorithm (see openssl dgst -h for details).

Which would generate the digest output:

44db14fe496c4bc4af5e8e6e3683e5db7acffa555897cf4b2b4345abaaf1ace3

Now because the implementation is using the openssl command, you can also choose to convert the hexidecimal output into binary and then Base64 encode that binary output, like so:

hmac sha256 "message to be hashed" secret-key -binary | base64

Which outputs:

RNsU/klsS8SvXo5uNoPl23rP+lVYl89LK0NFq6rxrOM=

You don’t have to use an abstraction around the command obviously, you can just use:

cat plaintext.txt | openssl dgst -sha512 -binary | base64

ℹ️ NOTE

base64 could be replaced with openssl’s base64 encoding command: openssl enc -base64 -A

Random Password Generation

Generating random passwords that are complex enough to make automated attacks difficult can be a bit tedious, yet important. But if you install a program such as pwgen (brew install pwgen) you’ll be able to generate random and complex passwords very easily.

Once installed, add the following alias to your shell:

alias psw="pwgen -sy 20 1"

Now when you execute psw you’ll get output that looks something like the following:

|93<3(M;r?~40c$A@>{\

Hash Functions

There are many different ways of accessing a hash function, two options we’ll look at will be using the executable shasum (provided by macOS) and the hashlib package provided by the Python programming language.

shasum

Let’s generate a hexidecimal digest of the message foobar using the SHA512 hash algorithm:

echo -n foobar | shasum -a 512

ℹ️ NOTE

see shasum -h for all available algorithms.

Which outputs:

0a50261ebd1a390fed2bf326f2673c145582a6342d523204973d0219337f81616a8069b012587cf5
635f6925f1b56c360230c19b273500ee013e030601bf2425

hashlib

Let’s again generate a hexidecimal digest of the message foobar using the SHA512 hash algorithm, now using Python:

import hashlib
message = hashlib.sha512()
message.update(b"foobar")
print(message.hexdigest())

Which outputs the same digest as the shasum command produced:

0a50261ebd1a390fed2bf326f2673c145582a6342d523204973d0219337f81616a8069b012587cf5
635f6925f1b56c360230c19b273500ee013e030601bf2425

cksum

Remember hash functions generate a digest of some message input, and one such use of that digest output is data corruption (i.e. a checksum).

The macOS also provides a cksum command which let’s you generate a checksum, like so:

echo foobar | cksum

Which outputs:

857691210 7

The first number is the checksum and the second number is the amount of data in bytes.

OpenSSH

OpenSSH provides secure and encrypted tunneling capabilities and is typically used to enable secure shell connections from your machine to external servers.

In order to generate a cryptographically secure key pair, execute the following command:

ssh-keygen -t rsa -b 4096 -C "your.email@domain.com"

This uses the RSA algorithm (which is the default, so the -t can be omitted) along with a key size of 4096 bits (the default is 2048).

The output of this command will be a public and private key pair.

It’s usually best to generate these keys (or at least move them when generated) within the ~/.ssh directory.

SSH Agent

One thing that catches me out all the time is when I open a new terminal tab or shell instance and I go to push up some code changes to a remote server only to discover an error saying I’m not authenticated. This is because the new terminal/shell instance doesn’t have the SSH agent running which is what makes my SSH key pair available.

This happens so often I’ve created an alias to make starting up the SSH agent and loading my SSH private key very quick and easy:

alias sshagent='eval "$(ssh-agent -s)" && ssh-add -K ~/.ssh/github_rsa'

ℹ️ NOTE

the use of the -K flag is macOS specific, it means it’ll add the key into the macOS keychain program.

OpenSSL

OpenSSL is designed to provide a method for securing web based communication (think HTTPS/TLS/SSL).

ℹ️ NOTE

for a full list of commands see: openssl -h and openssl <command> -h.

Key Exchanges

There are two popular key exchange algorithms:

RSA
Diffie-Hellman

For the specific details of each I recommend you read this post on the differences. In short RSA uses the person’s public key to encrypt the secret, while Diffie-Hellman uses a mathematical function to ensure only those two people communicating can calculate the secret based on the information that’s publicly available.

Generating a key pair

In order to generate a RSA based public/private key pair, execute the following commands:

# generate a private key
openssl genrsa -out private_key.pem 4096

# generate a public key, from the private key
openssl rsa -pubout -in private_key.pem -out public_key.pem

Encrypting and Decrypting

The following examples use symmetric encryption, and so you’ll be asked for a secret key when encrypting and decrypting (although you could also use the -pass flag like so -pass pass:<your_password>, yeah the syntax is odd and it’s the same for decrypting):

# symmetric encryption (you'll be asked for a key)
echo foobar | openssl enc -aes-256-cbc -out message.enc

# decrypt that encrypted message
openssl enc -aes-256-cbc -in message.enc -d

ℹ️ NOTE

.enc is a commonly used format to indicate a file is encrypted (.asc is specifically used for asymmetric encryption).

I’m passing in the message via stdin (when encrypting), but specifying a file for the output (when decrypting), but you could use a file for both by explicitly specifying the -in and -out flags to provide a text file instead.

Annoyingly with openssl the same thing can be done a million different ways, so (for example) you might also find that you can do the above without the enc portion of the command (and thus removing the - prefix from the selected algorithm):

# symmetric encryption
echo foobar | openssl aes-256-cbc -out message.enc

# decrypt that encrypted message
openssl aes-256-cbc -in message.enc -d

Encoding

You can also generate Base64 output of the encrypted data, by using the -a flag like so:

$ echo foobar | openssl aes-256-cbc -a

U2FsdGVkX19/L0WtkvCNlpMiQnvD1SWGM19lm4m6xK4=

ℹ️ NOTE

see man enc for details

Salts

It’s also worth mentioning that the default behaviour for OpenSSL is to use a ‘salt’ when using encrypting the message. A salt is random data appended to your already hashed message and then that is hashed itself. In pseudo-code it would look like this:

$pwd = hash(hash($password) + salt)

You would then store the value of $pwd in your database along with the salt itself.

The security doesn’t come from obfuscating the salt, but more that a rainbow table attack can’t now automatically loop/check its collection of hashed passwords. An attacker would need to incorporate your (per-user) unique salt value into their check against a predetermined list of hashes, and they also wouldn’t know if the salt was prefixed or suffixed to the password itself. Making it computationally very expensive and time consuming to attempt.

You can also see that a salt is used by trying to read an encrypted file (cat message.enc):

Salted__MJin¨MàÍ£?è,random¡:~randomW!5µõ

Asymmetrical Encryption

If you need to you can use a public key to encrypt data with (i.e. asymmetrical encryption) by utilising the openssl rsautl command, which stands for “RSA Utility” and is commonly used to sign, verify, encrypt and decrypt data using the RSA algorithm.

In the following example we have a file plaintext.txt we encrypt using a public key. It will now only be possible to decrypt the secret.enc file if you have the corresponding private key:

# encrypting
openssl rsautl -encrypt -pubin -inkey public_key.pem -in plaintext.txt -out secret.enc

# decrypting
openssl rsautl -decrypt -inkey private_key.pem -in secret.enc

Randomness

OpenSSL also offers a way to generate random binary data which you can then export as either hexidecimal or base64 formats:

ℹ️ NOTE

in the following examples, 64 is the number of bytes to be generated.

$ openssl rand 64

RR_wK[=q5}VrdMܾj{8(Ty]7;file://Integralist-MBPr/tmp

$ openssl rand 64 -hex

660baf33c189ced722a07c6a29d35a7e4584bb954c8c86f2cfd4ea8d892bff32fc188b0c56cbe0a5
6d60b628cdee697308b0cf3806cd95052b743bec5ccc5240

$ openssl rand 64 -base64

JIPU5SiCgKP3XVrnef1gY+PxjBvjdQgSN+OJoBAdWmCa/cRvDdFl01GQiSwFimQ5
1lVa/7hfYIK6Z5jjHNauaQ==

GPG

GPG is a tool which provides encryption and signing capabilities, and supports both symmetrical and asymmetrical encryption + digital signing of your encrypted content to ensure the integrity.

Generating a key pair

To generate a new GPG key pair you would execute the following command and interactively fill in the details:

gpg --gen-key

Automate

If you prefer to automate this you can create a file to contain the details and pass that into the command-line instead. The following code generates a new batch_file that will contain the information we would otherwise have to enter manually:

$ cat > batch_file <<EOF
     %echo Generating a basic OpenPGP key
     Key-Type: RSA
     Key-Length: 4096
     Subkey-Type: Default
     Name-Real: Your Name
     Name-Comment: Integralist testing
     Name-Email: foo@example.com
     Expire-Date: 0
     Passphrase: foobar
     %commit
     %echo done
EOF

Once we have this file we can pass it along with the --gen-key command:

$ gpg --gen-key --batch batch_file

gpg: Generating a basic OpenPGP key
gpg: key 4BCAEAAD199B5FE8 marked as ultimately trusted
gpg: directory '/Users/Integralist/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/Users/Integralist/.gnupg/openpgp-revocs.d/\
  CFE96536285D83C990567BF64BCAEAAD199B5FE8.rev'
gpg: done

Now if we check our list of keys we’ll see the new one we just generated:

$ gpg --list-keys

/Users/Integralist/.gnupg/pubring.gpg
---------------------------------------
pub   rsa4096 2018-02-17 [SCEA]
      CFE96536285D83C990567BF64BCAEAAD199B5FE8
uid           [ultimate] Your Name (Integralist testing) <foo@example.com>
sub   rsa2048 2018-02-17 [E]

Revocation

When you generate a new key pair, if you intend on publishing your public key online, then you’ll want to generate a revocation certificate. Doing this will mean you can revoke your original key pair if your private key becomes compromised (or you just want to decommission it):

gpg --gen-revoke your.email@domain.com

When you’re ready to decommission it, just import the certifcate into your keyring:

gpg --import revocation.cert

You can then also push up your key identifier to a key server to force it to recognise the key has been revoked:

gpg --keyserver pgp.mit.edu --send-keys <key_id>

Asymmetrical Encryption and Decryption

In order to encrypt some data using someone elses public key (i.e. so only they can decrypt the data) you first need access to their public key and have it imported to your gpg keyring:

gpg --import public.key

If you want to verify the integrity of the public key you have acquired, then you should speak securely with the recipient who owns the public key and ask them to give you their digital ‘fingerprint’. You can then verify it matches what you have using the following command:

gpg --fingerprint <pub_key_id>

You’ll then look for the fingerprint in the gpg output. The fingerprint should look something like this:

FDFB E9B5 24BA 6972 A3AA 44B9 A1B1 7E6F DD86 E7F5

The command for encrypting a file plaintext.txt using their public key would be:

gpg --encrypt -u "Sender User Name" -r "Receiver User Name" plaintext.txt

As you’ve encrypted the file using that person’s public key, it means they can decrypt the file simply with:

gpg -d plaintext.txt.gpg

Symmetrical Encryption and Decryption

By default gpg uses the AES algorithm for its symmetrical encryption. The command to use is (you’ll be asked to provide a passphrase):

gpg --symmetric plaintext.txt

You can specify a different algorithm, as the default isn’t as secure as it could be. Let’s use a 256bit encryption key:

gpg --symmetric --cipher-algo AES256 plaintext.txt

ℹ️ NOTE

see gpg --version for all available ciphers

Signing keys

If you want to explicitly trust a public key you have imported, you can ‘sign’ it. You do this using the --sign-key flag. Doing this can also be beneficial for the owner of that public key (Bob), because if a friend of yours (Alice) trusts you and they see you’ve signed Bob’s public key, then Alice is more likely to trust Bob as well.

In order for Bob to benefit from this ‘web of trust’ you need to send him back his public key which you signed. Bob would need to import that version of his public key back into his gpg keyring, so that he can then republish it online for others to see the you trust him.

The following example demonstrates how you would export Bob’s public key, which you previously imported and signed:

gpg --export --armor bob@example.org

ℹ️ NOTE

--armor simply outputs the binary data as ASCII

Signing encrypted files

It can be useful to sign a file that you encrypt, so that the person who will decrypt the file can verify it was you who sent it to them, and also check that the integrity of the file is still intact.

ℹ️ NOTE

this provides a combination of authenticity and integrity (as defined within the terminology section)

You do this by using the --sign flag:

gpg --local-user Bob --encrypt --recipient Alice --sign plaintext.txt

ℹ️ NOTE

I’m using --local-user because I have many different key pairs setup for testing.

This will generate a plaintext.txt.gpg encrypted file.

The recipient (Alice), can either decrypt the file using Bob’s public key and this will both decrypt and verify the signature, or Alice could just use the --verify flag if she didn’t want to decrypt the file.

$ gpg --verify plaintext.txt.gpg

gpg: Signature made Mon Feb 19 10:16:38 2018 GMT
gpg:                using RSA key F2G91BE243E405E5B64B08A1CB5EBDB2561C861B
gpg: Good signature from "Bob <bob@example.com>" [ultimate]

Keybase

Keybase is a public-key directory that maps social media identities to encryption keys in a publicly auditable manner. Keybase offers an end-to-end encrypted chat and cloud storage system, called Keybase Chat and the Keybase filesystem.

In order to use the command-line tool keybase you’ll need to register for an account on their website.

To install keybase on macOS:

brew install keybase

Once installed you’ll need to login:

keybase login

At this point you can either generate a fresh key pair or select an existing gpg key pair:

# generate new key pair
keybase pgp gen

# select existing key pair
keybase pgp select

You can search for other keybase users:

keybase search sthulb

You can then encrypt data for another keybase user, like so:

keybase encrypt -i info.txt -o info.txt.asc sthulb

If you receive an encrypted file you can decrypt it, like so:

keybase decrypt -i info.txt.asc -o info.txt

If you receive an encrypted file (info.txt.gpg) using your keybase pub key but the senders not using keybase (e.g. they’ve encrypted the file using their own gpg private key), then you’ll need to have their public key in your gpg keyring:

keybase pgp decrypt -i info.txt.gpg

Computers 101: terminals, kernels and shells

Tue, 30 Jan 2018 00:00:00 +0000

Before we get started, here’s a diagram to give you an overview of what we will be covering in this post:

One thing I want to clarify in the above image is that each box is within the parent box. So for example, The “Program: terminal” is interacting with a contained shell, while the shell then spawns either a “Built-in” or “Program”.

ℹ️ NOTE

Nearly four years after this post was published someone else published https://www.poor.dev/blog/terminal-anatomy/ which is an excellent write up that provides a really good animation of how the terminal interacts with the shell, and it goes into much deeper explanation that I do here. So I recommend reading that also.

Kernel

A computer has a kernel.

The kernel is responsible for managing the computer’s system.

The kernel has no user interface.

To interact with the kernel you use an intermediary “program”.

Program

A “program” is a structured collection of instructions (machine code) that a computer can execute.

Your computer has many programs. One such example would be the ‘terminal emulator’ program.

ℹ️ NOTE

see next section for explanation of what a “terminal” is.

Depending on the programming language used to create the program, either the program is compiled down into binary so it can be understood by the computer, or it’ll be interpreted by another program that then generates machine code out of the human readable program.

Executables

Executables (or ‘executable binaries’) are programs.

More specifically, an ‘executable’ is a file that contains a program.

ℹ️ NOTE

these are also often referred to as just ‘binaries’.

Executables are generally the result of a program being turned into something that can be ‘executed’ by the computer.

Executables can be found in multiple locations, e.g. look at the $PATH environment variable in a terminal.

$ echo $PATH

/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

ℹ️ NOTE

separate the output by : and you see there are six directories

Terminal

A terminal is an input/output device.

Traditionally terminals would have been a real ‘hardware’ device that you used to interact with a computer.

e.g. the computer would be a large box in a server room, and the terminal would be a monitor/keyboard connected to the computer.

In modern computing we have electronic terminals.

The modern equivalent of a terminal is known as a ‘terminal emulator’.

Terminal emulators ironically (or confusingly) are part of the computer they would have previously been plugged into separately.

If you don’t want to use a GUI (graphical user interface) to interact with your computer, you can use a terminal emulator.

Shell

A shell is a program which is accessed via a terminal emulator.

The terminal accepts input, passes it to the shell, and the shell’s output is sent back to the terminal to be displayed.

The shell accepts input as a set of commands.

The available commands vary depending on the shell (e.g. different shells have different commands).

In order to fulfil your instructions, commands are interpreted by the shell and the shell determines if it should either:

load another program

execute a ‘builtin’ function

Shell Builtins

A ‘builtin’ function is one that is provided by the shell.

If a command is provided and the shell has no corresponding builtin associated with the given command, it will lookup the command via a separate list of available external ‘executables’.

A builtin command can affect the internal state of the shell.

This is why a command such as cd must be part of the shell (i.e. a builtin), because an external program can’t change the current directory of the shell.

Other commands, like echo, might be (and are in this case) built into the shell for the sake of performance (it’s quicker to call the builtin echo than it is to load and manage the external executable echo).

Documentation

Most people are aware of ‘manuals’.

e.g. man bash returns the documentation for the Bash shell.

Manuals do not cover shell builtins.

The exit command is a shell builtin, so what happens when looking up a manual for it?

e.g. man exit returns a generic ‘BUILTIN’ documentation page.

If you see that page, then you know the command is a shell builtin.

Another way to tell if a command is a builtin vs an executable is to use the type command.

$ type exit

exit is a shell builtin

One other reason I like to use type is when trying to figure out what a shell alias is set to (in case you’re unfamiliar, in most shells you can assign a long or hard to remember command to a short variable name). Imagine I’ve created an alias like so:

alias gb="git branch --list 'integralist*'"

I can find out later what I assigned to the alias using the type command:

$ type gb

gb is aliased to `git branch --list 'integralist*''

To read the documentation for a builtin, you need to use the help command:

$ help exit

exit: exit [n]
    Exit the shell.

    Exits the shell with a status of N.  If N is omitted, the exit status
    is that of the last command executed.

The help command is itself a builtin (hence it knows about builtins, unlike man which isn’t a builtin).

$ type help

help is a shell builtin

You can use the help command to read its documentation:

$ help help

help: help [-dms] [pattern ...]
    Display information about builtin commands.

    Displays brief summaries of builtin commands.  If PATTERN is
    specified, gives detailed help on all commands matching PATTERN,
    otherwise the list of help topics is printed.

    Options:
      -d        output short description for each topic
      -m        display usage in pseudo-manpage format
      -s        output only a short usage synopsis for each topic matching
                PATTERN

    Arguments:
      PATTERN   Pattern specifying a help topic

    Exit Status:
    Returns success unless PATTERN is not found or an invalid option is given.

If you run the help command by itself you’ll see a list of commands that can be passed to help (you’ll see in the list exit, hence why we could run help exit earlier):

$ help

GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.5.0)
These shell commands are defined internally.  Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.

A star (*) next to a name means that the command is disabled.

 job_spec [&]                                                                                                           
 (( expression ))                                                                                                       
 . filename [arguments]                                                                                                 
 :                                                                                                                      
 [ arg... ]                                                                                                             
 [[ expression ]]                                                                                                       
 alias [-p] [name[=value] ... ]                                                                                         
 bg [job_spec ...]                                                                                                      
 bind [-lpsvPSVX] [-m keymap] [-f filename] [-q name] [-u name] [-r keyseq] \
 [-x keyseq:shell-command] [keyseq:readlin>  
 break [n]                                                                                                              
 builtin [shell-builtin [arg ...]]                                                                                      
 caller [expr]                                                                                                          
 case WORD in [PATTERN [| PATTERN]...) COMMANDS ;;]... esac                                                             
 cd [-L|[-P [-e]] [-@]] [dir]                                                                                           
 command [-pVv] command [arg ...]                                                                                       
 compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat] [-W wordlist]  \
 [-F function] [-C command] [-X filterpat>  
 complete [-abcdefgjksuv] [-pr] [-DEI] [-o option] [-A action] [-G globpat] \
 [-W wordlist]  [-F function] [-C command]>  
 compopt [-o|+o option] [-DEI] [name ...]                                                                               
 continue [n]                                                                                                           
 coproc [NAME] command [redirections]                                                                                   
 declare [-aAfFgilnrtux] [-p] [name[=value] ...]                                                                        
 dirs [-clpv] [+N] [-N]                                                                                                 
 disown [-h] [-ar] [jobspec ... | pid ...]                                                                              
 echo [-neE] [arg ...]                                                                                                  
 enable [-a] [-dnps] [-f filename] [name ...]                                                                           
 eval [arg ...]                                                                                                         
 exec [-cl] [-a name] [command [arguments ...]] [redirection ...]                                                       
 exit [n]                                                                                                               
 export [-fn] [name[=value] ...] or export -p                                                                           
 false                                                                                                                  
 fc [-e ename] [-lnr] [first] [last] or fc -s [pat=rep] [command]                                                       
 fg [job_spec]                                                                                                          
 for NAME [in WORDS ... ] ; do COMMANDS; done                                                                           
 for (( exp1; exp2; exp3 )); do COMMANDS; done                                                                          
 function name { COMMANDS ; } or name () { COMMANDS ; }                                                                 
 getopts optstring name [arg]                                                                                           
 hash [-lr] [-p pathname] [-dt] [name ...]                                                                              
 help [-dms] [pattern ...]                                                                                              
 history [-c] [-d offset] [n] or history -anrw [filename] or history -ps arg [arg...]
 if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMANDS; ]... [ else COMMANDS; ] fi
 jobs [-lnprs] [jobspec ...] or jobs -x command [args]
 kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
 let arg [arg ...]
 local [option] name[=value] ...
 logout [n]
 mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] \
 [-c quantum] [array]
 popd [-n] [+N | -N]
 printf [-v var] format [arguments]
 pushd [-n] [+N | -N | dir]
 pwd [-LP]
 read [-ers] [-a array] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] \
 [-t timeout] [-u fd] [name ...]
 readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] \
 [-c quantum] [array]
 readonly [-aAf] [name[=value] ...] or readonly -p
 return [n]
 select NAME [in WORDS ... ;] do COMMANDS; done
 set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
 shift [n]
 shopt [-pqsu] [-o] [optname ...]
 source filename [arguments]
 suspend [-f]
 test [expr]
 time [-p] pipeline
 times
 trap [-lp] [[arg] signal_spec ...]
 true
 type [-afptP] name [name ...]
 typeset [-aAfFgilnrtux] [-p] name[=value] ...
 ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]
 umask [-p] [-S] [mode]
 unalias [-a] name [name ...]
 unset [-f] [-v] [-n] [name ...]
 until COMMANDS; do COMMANDS; done
 variables - Names and meanings of some shell variables
 wait [-fn] [id ...]
 while COMMANDS; do COMMANDS; done
 { COMMANDS ; }

If we want to see the documentation for the type builtin, use help type:

$ help type

type: type [-afptP] name [name ...]
    Display information about command type.

    For each NAME, indicate how it would be interpreted if used as a
    command name.

    Options:
      -a        display all locations containing an executable named NAME;
                includes aliases, builtins, and functions, if and only if
                the `-p' option is not also used
      -f        suppress shell function lookup
      -P        force a PATH search for each NAME, even if it is an alias,
                builtin, or function, and returns the name of the disk file
                that would be executed
      -p        returns either the name of the disk file that would be executed,
                or nothing if `type -t NAME' would not return `file'
      -t        output a single word which is one of `alias', `keyword',
                `function', `builtin', `file' or `', if NAME is an alias,
                shell reserved word, shell function, shell builtin, disk file,
                or not found, respectively

    Arguments:
      NAME      Command name to be interpreted.

    Exit Status:
    Returns success if all of the NAMEs are found; fails if any are not found.

Explicit Requests

When we used the type command earlier on the exit command it returned a single response (exit is a shell builtin).

Let’s try again with a different command (echo):

$ type echo

echo is a shell builtin

But if we also apply the -a flag we get more output:

$ type -a echo

echo is a shell builtin
echo is /bin/echo

This indicates that the shell found a builtin first, but that there was also an external executable called echo.

If you were to execute echo foo you would be calling the builtin echo command.

You could be explicit by executing it via the builtin command:

$ builtin echo foo

foo

You could also explicitly request the executable and not the builtin by using the command command:

$ command echo foo

foo

Locating programs

To locate a program you use the which executable.

We know it’s an executable by using the type builtin to check it against:

$ type -a which

which is /usr/bin/which

If we use which to lookup the location of the echo command, will it find the builtin or the external executable?

$ which echo

/bin/echo

We can see it only found the external executable.

The which command isn’t a builtin, and so it has no idea of where to look for builtins.

Because, by nature, builtins are built into the shell itself.

Hashed Types

If you open a fresh terminal screen and execute type man you would see the response man is /usr/bin/man.

If you now execute the man command (e.g. man echo) and try type man again you’ll see:

man is hashed (/usr/bin/man)

The reason for this is because in order for the shell to locate the executable it needs to look it up from various locations.

These locations are defined in the $PATH (as we saw earlier).

To avoid having to do that lookup every time, it caches the result in a hash table.

If we read the Bash manual (man bash) you’ll see the following comment:

Bash uses a hash table to remember the full pathnames of executable files 
(see hash under SHELL BUILTIN COMMANDS below). 

A full search of the directories in PATH is performed only if the command is not found in the hash table.

So it seems there is a hash builtin command, let’s take a look at that:

$ help hash

hash: hash [-lr] [-p pathname] [-dt] [name ...]
    Remember or display program locations.
    
    Determine and remember the full pathname of each command NAME.  If
    no arguments are given, information about remembered commands is displayed.
    
    Options:
      -d    forget the remembered location of each NAME
      -l    display in a format that may be reused as input
      -p pathname    use PATHNAME as the full pathname of NAME
      -r    forget all remembered locations
      -t    print the remembered location of each NAME, preceding
            each location with the corresponding NAME if multiple
            NAMEs are given
    Arguments:
      NAME    Each NAME is searched for in $PATH and added to the list
            of remembered commands.
    
    Exit Status:
    Returns success unless NAME is not found or an invalid option is given.

So the documentation informs us of how we can look inside of the shell’s hash table by using the -l flag:

$ hash -l

builtin hash -p /usr/bin/which which
builtin hash -p /usr/bin/man man
builtin hash -p /usr/bin/clear clear

From this you can see I’ve already executed the which, man and clear executables, hence they’re now cached.

Also in the Bash manual is the following comment:

BASH_CMDS
    An associative array variable whose members correspond to the internal 
    hash table of commands as maintained by the hash builtin.
    Elements added to this array appear in the hash table; 
    however, unsetting array elements currently does not cause command names to 
    be removed from the hash table.  
    If BASH_CMDS is unset, it loses its special properties, even if it is subsequently reset.

This informs us that there is another way to view the hash table contents.

In this case we can view the internal array the hash builtin appends to:

$ declare -p BASH_CMDS

declare -A BASH_CMDS=([which]="/usr/bin/which" [man]="/usr/bin/man" [clear]="/usr/bin/clear" )

ℹ️ NOTE

it’s not as clear to read as the hash output, but this is probably more useful for interacting with programatically.

List of all builtins vs executables

For a list of builtins you can use (in the Bash shell at least):

$ enable -a

enable .
enable :
enable [
enable alias
enable bg
enable bind
enable break
enable builtin
enable caller
enable cd
enable command
enable compgen
enable complete
enable compopt
enable continue
enable declare
enable dirs
enable disown
enable echo
enable enable
enable eval
enable exec
enable exit
enable export
enable false
enable fc
enable fg
enable getopts
enable hash
enable help
enable history
enable jobs
enable kill
enable let
enable local
enable logout
enable mapfile
enable popd
enable printf
enable pushd
enable pwd
enable read
enable readarray
enable readonly
enable return
enable set
enable shift
enable shopt
enable source
enable suspend
enable test
enable times
enable trap
enable true
enable type
enable typeset
enable ulimit
enable umask
enable unalias
enable unset
enable wait

ℹ️ NOTE

an online reference can be found here

To list out all available executables is a little more tricky.

First you need to access only those directories you’re interested in:

$ echo $PATH | tr ':' '\n' | sort | egrep '^/(usr|bin)'

/bin
/usr/bin
/usr/local/bin
/usr/local/sbin
/usr/local/sbin
/usr/local/sbin
/usr/sbin

ℹ️ NOTE

tweak the regex as you see fit

Then you need to list all the commands within those directories.

The following alias’ give you an idea of how you might approach doing that.

alias commands_dir='echo $PATH | tr ":" "\n" | sort | egrep "^/(usr|bin)"'
alias commands='for i in $(commands_dir):; do eval "ls -l $i"; done'

Does anyone know of a better way? I’d ❤️ to hear about it

Statistics and Graphs: The Basics

Tue, 19 Dec 2017 00:00:00 +0000

Introduction

I started learning about statistics because I found myself doing a lot of operational monitoring (i.e. making systems more observable, instrumenting individual services, and monitoring that data via custom built dashboards).

Although, for the most part, everything worked as expected and I generally understood the data I was seeing visualised, I wanted to be sure I wasn’t missing any important information (or worse, mis-representing the data).

This began my journey into learning about statistics (as a complete beginner). I picked up a book on the subject and started taking notes.

ℹ️ NOTE

I highly recommend reading Head First Statistics which is where the majority of this information has stemmed. There you’ll find a lot more detail and better break downs of the ideas.

This blog post is the result of what I have learnt so far, and its motivation is to explain some of the basic concepts behind utilising statistics to represent data. This post is aimed at beginners, as I myself am very much a beginner in this space.

So let’s begin by defining what ‘statistics’ means…

Statistics is a branch of mathematics dealing with the collection, analysis, interpretation, presentation, and organization of data – Wikipedia

OK, that seems reasonable enough. Statistics is an ‘umbrella’ term that encapsulates the complete pipeline of how data is acquired, analysed and visualised. So what is data and what does it look like? Let’s move onto the next section where we can begin to clarify and understand it a bit more…

Information vs Data

If you’re like me, you may well have confused the words “data” and “information” as being the same thing. But there are actually important differences that should be understood:

Data: is the raw facts/figures.
Information: data that has extra context/meaning applied.

For example, raw data might look like:

[3, 5, 7]

Whereas information would be the taking of that raw data and giving it extra context. So, in this example, those three data points could represent the age of three children. In a classic CSV (comma separated values) format (which is used for storing ‘tabular’ data), it might be represented like so:

Age
3,
5,
7

ℹ️ NOTE

data is either “numerical” (dealing with numbers), “quantitative” (describing quantities) or “categorical” (data is split into categories that describe qualities or characteristics - also referred to as “qualitative”).

Frequency

When dealing with statistical data, the first thing you typically learn about is data “frequency”. Let’s start with a definition…

In statistics the frequency (or absolute frequency) of an event is the number of times the event occurred in an experiment or study. – Wikipedia

Consider the following data:

3,5,5,5,7,7,2

The first number 3 and the last number 2 both have a frequency of one, in that they only appear once throughout the entire dataset. Whereas the number 5 has a frequency of three and the number 7 has a frequency of two for similar reasons (i.e. 5 appears three times and 7 appears twice).

Another way to represent this data is by defining a separate column for the frequency (which allows you to more clearly see the unique numbers that are present in the dataset):

Age, Frequency
3,   1
5,   3
7,   2
2,   1

ℹ️ NOTE

strictly speaking you might want to refer to the above snippet as information rather than data simply because the raw data now has ‘context’ added to it which clarifies what the numbers mean.

The data/information can be visualised in many ways depending on the graph type you wish to use. Some graphs are better suited for representing certain types of data than others.

We’ll take a look at some different graph types to see how they work, but first let’s take a moment to consider how data can trick us…

Watch out for misleading data

The following example is very contrived and silly, but it does illustrate the point about being aware of how data can be manipulated to represent what you want it to.

Below are two ‘line’ graphs. We’ll talk about this type of graph in more detail later, but effectively we create two axis and then plot our data onto the graph and draw a line between the dots.

Now both graphs use the same datapoints, but the first graph is misleading the viewer, while the second graph is more accurate. Take a look and see why that might be?

At a quick glance (or if you just didn’t know any better), you would see the first graph and think the company has some incredible profit growth. But the second graph doesn’t look as impressive?

This is because the first graph is zoomed in from the starting point 2.0 whereas the first graph is zoomed out at the correct level so you can see the data in a more accurate and representative form.

See also my comment in the next section about pie charts. You’ll notice there that the data can be mis-represented if not all the information is made available to the user.

So the data isn’t lying, it’s just the view the user has of the data isn’t accurately portrayed due to purposeful data ommission (this is a trick newspapers and academic papers use to represent a point of view they wish to push).

Pie Chart

A graph, such as a pie chart, will split your data up into distinct ‘groups’. These groups are represented as relative percentages of the total group, meaning the total area adds up to 100%.

Below is an example pie graph that uses the data [150,200,50], which could represent (just for example) usage of certain programming languages within an organisation. The pie chart is generated based off the percentage representation of the underlying data.

Here is how the graph data is calculated:

Start by finding the total (i.e. sum all the data)
Then calculate 1% of that total
Finally, calculate each group’s individual percentage

Total: 150+200+50 = 400
1 Percent: ⁴⁰⁰⁄₁₀₀ = 4

Now we know the total is 400 and 1% of that is 4 we can calculate each group’s individual percentage…

Python group = 50%
i.e. (400/100) * 50 = 200

Go group = 37.5%
i.e. (400/100) * 37.5 = 150

Bash group = 12.5%
i.e. (400/100) * 12.5 = 50

We’ll know if we’ve calculated things correctly, if the sum of the group percentages results in 100%

50% + 37.5% + 12.5% = 100%

Pie charts are useful for understanding ‘at a glance’ the relative difference between groups of data. But they become less useful when the data is close together, as each ‘slice’ becomes effectively the same size.

Misleading?

As mentioned in the previous section, all graphs can be presented in such a way as to mislead you and make you think that the reality is different to what it really is. Pie charts are no exception.

Pie charts visually ‘display’ their information in relative percentages, but if they don’t also show (or include somewhere near the graph) the frequency for each group within the pie chart, then the graph could be misleading.

The reason it can be misleading is because without the frequency you can’t identify whether the data being presented is consistent. This may or may not indicate whether it’s fair to compare the data in this way (as it might not be truly representative).

Whether that is the case or not depends on the type of data being presented. In our case it’s not really an issue for two reasons:

We include the frequency data in our pie chart.

hover over or click on the chart to view frequency.

Our data is simple enough to not be mis-represented.

So you have to be careful with data to make sure it’s as inclusive as possible or that you’re explicit about what you’re focusing on or how the data might be lacking.

Earlier we mentioned a problem of data groups percentages being too close together resulting in a graph that was hard to distinguish subtle differences. In those scenarios you might find a more suitable option would be the bar chart…

Bar Chart

If you wanted to see the ‘programming language’ data in a format that is more suitable for subtle differences, then one option would be a bar chart.

Let’s view the dataset we have:

Language, Frequency
Go,       150
Python,   200
Bash,     50

When dealing with a horizontal bar chart (directly below), the individual data groups (in this case: the programming languages) are placed along the y axis, while with a vertical bar chart (example below that) they’re placed along the x axis.

One thing to notice about a bar chart is that the width of the bar is the same (doesn’t matter if it’s a horizontal or vertical variation, their widths stay consistent). It’s the length of each bar that’s actually important.

This is because the length represents the frequency of the group’s data. This can trip people up, as they might mistake a bar chart for a ‘histogram’, where the width of the bar does change in relation to the frequency (don’t worry, we’ll come back to histograms later).

Stacked Bars

Now imagine you wanted to visualise data that represented how much people liked or disliked specific programming languages (this is different to the previous data which was the general usage of programming languages).

The dataset might look something like the following:

Language, Like, Dislike
Go,       290,  10
Python,   150,  100
Bash,     50,   250

To represent this multifaceted data you could use a specific type of bar chart known as a ‘stacked’ bar chart (see below). This type of graph is useful because it represents both the frequency and the relative percentage of the various data types.

ℹ️ NOTE

stacked bar charts are also known as ‘segmented’.

It’s less likely for this chart to be mis-represented because the length of each bar is based on the data frequency.

You can see for the groups “Go” and “Bash” we’ve had a consistent number of reports (300 in total), and so when the values are visualised as a percentage the length of the bars are the same.

Whereas the “Python” group has less data frequency compared to the other groups (only 250 in total, where the other groups were 300), and so although it correctly represents that data as a percentage (60% were “like” vs 40% “dislike”) it’s still not as representative as a whole in comparison to the other data groups we have. Ideally each group would have consistent frequencies.

Split Bars

Another type of bar chart is called ‘split-category’ and is useful for comparing frequencies (unlike the stacked/segmented bar chart which compares frequency but represents them visually in percentages).

Histograms

I’ve yet to find a charting library that let’s me create an actual histogram in the strict definition (most libraries seem to call standard bar charts ‘histograms’?), which makes it hard for me to visually demonstrate them, unless I hand draw a chart (which is what I’ve had to resort to below).

Differences?

Based on what I’ve read, there are some key differences between histograms and standard bar charts. These are:

The ‘area’ (width and height) of each bar is proportional to its frequency.
There should be no gaps between each bar.

The reason for histogram bar sizes being proportional (unlike a traditional bar chart †) is because histograms are usually best suited to dealing with grouped numerical data.

† remember: with a traditional bar chart, each bar width is the same and only the bar ‘length’ is relevant (as it’s determined by the frequency).

Calculating dimensions

When constructing a histogram, you’ll place the groups on the x axis and make the width of each bar the same as the range it covers; while placing the frequencies for each group on the y axis and make the length of the bar match the frequency value for that group.

Doing this is fine, as long as the groups have a consistent range (i.e. they all have the same interval size, like: 0-5,5-10,10-15 and whose range/interval distance are all five).

Inconsistent Ranges

But in some datasets a single group can cover a much wider range than the other groups. For example, consider a dataset for gaming hours played by a group of 17 users:

Hours, Frequency
0-1,   5
1-20,  2
20-22, 10

Here ‘hours’ is the grouped numerical data. As explained above, we would put the frequency on the y axis and the grouped data along the x axis (see below for an example graph).

ℹ️ NOTE

we can see our first bar “0-1” spreads only one interval and “20-22” spreads only two intervals, whereas the middle bar “1-20” spreads over nineteen intervals!

The problem with the above graph is that the height of the middle bar is wrong as we’ve set it to be as high as the frequency value itself (and the third bar is incorrect too!), which normally would be fine if the interval range were the same across all groups, but in this case it isn’t the correct approach due to each group covering a different range from each other.

In using this data, the 1-20 group’s area could mistakenly look disproportionately large (the 20-22 group also has a different range so that would be a problem as well).

So to solve the problem of disproportionate sizes (when dealing with multi-range groups), we have to make sure that both the width and height (or ‘area’) of the group is proportional.

How to fix the proportions?

To fix the bar area size problem we need a different calculation for determining the ‘height’ of each bar (also known as the frequency density):

frequency / range = frequency density

For example, the range for 1-20 is 19 and its frequency is 2:

2 / 19 = 0.10

Meaning the height of the bar (its frequency density) for 1-20 should be set to 0.10 and not to the frequency value itself (which was 2).

You would then apply this calculation to the height of each group/bar (see below). You’ll see we also needed to change the height of the 20-22 group based on this new area calculation (10 / 2 = 5).

ℹ️ NOTE

I appreciate the difference between the original height of 2 vs the correct height of 0.10 (for the range 1-20) and 10 vs 5 (for the range 20-22), when using this contrived example, isn’t exactly ground breakingly different; but this is just to help you understand the general idea behind calculating histogram areas for groups that have inconsistent ranges.

We can now see the overall area (or shape) for each bar represents the actual frequency, and can be calculated like so:

frequency = range × frequency density

Example: the group 20-22 had a range of 2 and a frequency density of 5.

If we were to look at the graph by itself, which only shows the proportional area relative to the other groups (i.e. it only shows the frequency density and the range) we could reverse engineer/calculate the actual frequency value with the abstract calculation above (2 * 5 = 10).

Frequency Density?

As alluded to earlier, the height of each bar indicates the frequency density, which itself is best explained via an analogy:

Imagine you have a tall glass and you pour a set amount of liquid into it that fills the glass to approximately ³⁄₄. If you now pour that liquid into a much wider glass, the liquid amount hasn’t changed but the height has changed. It’ll be at a lower level as the liquid has spread out more across the available glass space.

The frequency density is related to frequency but is focused more around its ‘concentration’.

If we wanted to calculate the total frequency for the entire dataset then we would use the following calculation: group range x group frequency density for each group (which gives us the frequency for each group) and then we sum each of the frequencies together.

ℹ️ NOTE

this is as far as I got with histograms, as I felt I had learnt enough to be dangerous in a conversation. There may well be nuances, pros/cons and other aspects to histograms I’ve not fully understood. I’d welcome anyone who knows more to educate me on this :-)

Line Graphs

We’ve already seen a couple of example line graphs at the start of this post. To create a line graph you require two axis (x and y) and the data to be mapped onto different points across these axis’ which allow us to plot a line between the dots we mark.

These types of graphs are best for identifying trends in your data and are most useful when applied across numerical data (such as time, which again helps with overarching trending patterns).

ℹ️ NOTE

compare this to bar charts, which are generally better for comparing values or categories.

There are various types of line graphs, one is known as an “accumulative frequency graph” which we can demonstrate using the earlier grouped dataset of hours played online. If we were to take the hours data, we could graph a specific subset view of that data.

For example, we can visualise the number of people who played for a specific number of hours. So we could say: “how many users were playing online for up to five hours?”.

The way we do this is by adding up all the previous groups frequencies, which determines the upper limit for each group. So for example, using our previous data:

Hours, Frequency
0-1,   5
1-20,  2
20-22, 10

We can calculate the cumulative frequency like so:

0-1:   upper limit = 1  (total/cumulative frequency: 5)
1-20:  upper limit = 20 (total/cumulative frequency: 5+2=7)
20-22: upper limit = 22 (total/cumulative frequency: 5+2+10=17)

Resulting in the following cumulative frequencies:

0-1:   5
1-20:  7
20-22: 17

We can now plot these onto a line graph by placing the cumulative frequencies onto the y axis and the groups across the x axis. Then mark onto the graph the upper limits against the relevant cumulative frequency and finally join up the dots:

In the above example graph, if we asked “how many people were playing for up to 21 hours?” the answer would be approximately 11 people.

Averages

When looking at data and graphs, people are generally interested in ‘averages’ because they help us better gauge what/where the majority is. But there are actually three different types of ‘average’, and each one has a different purpose:

mean average
median average
mode average

Mean

The ‘mean’ average is the average that most people think about. To calculate the mean you would sum every number in your dataset and then divide the result by the number of elements in the dataset. For example:

dataset = [2,5,9]
sum = 16 (2 + 5 + 9)
mean = 3 (16 / 3)

ℹ️ NOTE

the mean average can be expressed mathematically by Σx/n. When broken down Σx (pronounced “sigma x”) is a quick way of saying “add together the values of all the x’s” without having to say what the values are. This also can be expressed with the Greek symbol μ.

When dealing with datasets that have frequencies we need to ensure the frequencies are included as part of the calculation. For example, consider the following data:

Age:       19, 20, 21
Frequency: 1,  3,  1

What the frequencies indicate is that 19 and 21 only appear once, whereas 20 appears three times. So when summing the values (Σx) this doesn’t mean 19+20+21, instead it means 19+20+20+20+21.

Similarly, when dividing by the number of items in the dataset (the n in Σx/n) it means dividing by 5 (1+3+1) and not taking the number of frequencies literally, so not dividing by 3 (1,3,1).

Resulting in a calculation that gives us the mean average as 20:

(19+20+20+20+21) / (1+3+1) = 20

100 / 5 = 20

ℹ️ NOTE

this type of mean average (i.e. one applied to data that has frequencies associated with it) can be expressed mathematically with Σfx/Σf and equates to “multiply each number by its frequency, then add the results together” (Σfx), then “divide the result by the sum of frequencies” (Σf).

There’s an issue with using the mean average, and that’s outliers.

Consider the following dataset:

Age:       19, 20, 21, 145, 147
Frequency: 3,  6,  3,  1,   1

If we wanted to calculate the average age (using Σfx/Σf), that would look like the following:

((3×19) + (6×20) + (3×21) + 145 + 147) / (3+6+3+1+1) = 38

This is telling us the average age is 38. That number doesn’t actually exist in the dataset!?

What has happened is that the outliers (the large numbers at the end of the dataset: 145, 147) have pulled the mean higher, meaning the data is “skewed”. Data can be skewed to the left (the mean is pulled lower) or it can be skewed to the right (the mean is pulled higher - as in the case of our example above).

ℹ️ NOTE

we say skewed “left” or “right” because when sorting the data in ascending order you would see the outliers are either mainly to the left or the right (depending on the data).

If we were to look at this on a line histogram graph we would notice this ‘pulling’ of the mean:

ℹ️ NOTE

in the above graph you can see the longer ‘tail’ that indicates the outliers.

How can we deal with outliers? Well, this is where the median can help…

Median

The median gives us the exact middle of our data.

To calculate this you would take all the numbers (inc. their frequencies) sort them, and then select whatever is the middle number. Let’s see this using the following dataset:

Age:       19, 20, 21, 145
Frequency: 3,  6,  3,  1

Let’s include the frequencies and sort the numbers in ascending order:

19, 19, 19, 20, 20, 20, 20, 20, 20, 21, 21, 21, 145

There’s 13 numbers in total (i.e. sum the frequencies Σf like so: 3+6+3+1).

So to find the exact middle we just divide the number by 2 (which gives us 6.5) and then round it (so it becomes 7). Meaning the seventh number in the dataset is the exact middle: 20.

ℹ️ NOTE

alternatively it can be abstracted to (n+1)/2

But what happens with a dataset with an even set of numbers? Well, in that case you need to take the mean of the two middle numbers (i.e. sum them and divide the result by 2). Consider the following example:

19, 19, 20, 20, 20, 21, 22, 23, 145, 147

There are 10 numbers so if we divide by 2 we’ll get 5. If you count five items inwards from either the start or the end of the dataset, you’ll find you land on two different numbers (in this case 20 and 21).

So in this scenario you sum 20 and 21 (20+21 = 41) and divide the result by two (41/2 = 20.5). Meaning the median of the above dataset is 20.5.

ℹ️ NOTE

what you’ll typically find is that if the data is symmetrical then the mean and the median will result in the same value.

But what happens if a dataset has multiple clusters of values, so for example the dataset includes values that are at both low and high ends?

Consider a baby swimming group; this would include very young ages (i.e. the babies) and older ages (i.e. the parents) and so you might find the mean and median both result in an average age for that dataset as being mid-teens (which traditionally wouldn’t necessarily be correct - depending on the social/cultural environment). Also the results can vary significantly if there are more babies or parents joining the class.

The solution to this problem is to return the most popular value (the one with the highest frequency). This is where the “mode” average can help…

Mode

The mode is the third and final type of average, and it aims to return the value with the highest frequency. Which means the value that it gives as the result MUST exist within the dataset (unlike the mean which could result in a value that doesn’t exist).

ℹ️ NOTE

if there are multiple frequencies with the same value, then the dataset is considered “bimodal”. Meaning there are multiple modes for the dataset.

The mode average is the only average that works not just on numerical data but categorical data. When you’re dealing with categorical data, the mode is the most frequently occurring category.

Here are the steps for calculating the mode:

Identify all the distinct categories (or values) in your dataset.
Document the frequency for each categoy (or value).
Select the values with the highest frequency to find the mode.

The mode is considered most useful when dealing with data that has a small number of modes, or when the data is categorical instead of numerical (remember: neither the mean nor the median can be used with categorical data). But if there are lots of modes identified, then the mode average can end up being less useful over other averages.

Let’s see some example data and identify the mode average in each:

Values:    1, 2, 3
Frequency: 3, 7, 4

The mode for the above dataset is 2 (as it is the value that has the highest frequency).

Categories: Red, Yellow, Green
Frequency:  20,   7,      4

The mode for the above dataset is the “Red” category (as it is the category has the highest frequency).

Values:    1, 2, 3
Frequency: 3, 7, 7

The above dataset actually contains multiple modes (i.e. multiple modes with the same frequency which happens to be the highest in the set). The modes being 2 and 3 (whose frequency is 7).

Which average to use?

Imagine we have staff, managers and a CEO and they’re all determining how pay rises should be calculated when based on the average salary. Below is a suggestion as to which average each one of them might suggest using and why:

Staff: median
staff prefer the median average as it helps reduce the effect of the CEO’s salary as an outlier.
Managers: mean
the large CEO salary is an outlier, thus resulting in the mean being skewed to the right - so this is the preferred average of the managers as it brings a much larger pay rise!
CEO: mode
considering there are many more staff than managers (or CEOs), the CEO prefers the mode because it results in the staff salary being the basis of a pay rise - resulting in paying out a lot less than is fair.

ℹ️ NOTE

the median is the most ‘fair’ when calculating a pay rise in this example scenario.

Ranges

Averages help identify the center of our data (and via various perspectives as we’ve already seen: mean, median and mode). But this isn’t useful when it comes to understanding how the data itself varies.

Ranges help explain how data is distributed and informs you of how far apart your highest and lowest values are. In other words, the range is a way of measuring how spread out a set of values are (almost like we’re measuring the width of the dataset).

To calculate the range of a dataset you require the “upper bound” and the “lower bound”. The upper bound is the highest value and similarly the lower bound is the lowest value. The range calculation is as follows:

upper bound - lower bound = range

But you can be in a situation where two separate datasets have different distribution of data and yet they have the same ‘range’ value. This is because the range only indicates the width of the data and not how it’s dispersed inbetween the higher/lower bounds.

ℹ️ NOTE

the range value is very sensitive to outliers and so it can be misleading if used to identify data distribution.

Consider this example dataset:

1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5

On a bar chart this data could be mapped like so (where the frequencies are set on the y axis and the numerical values are set on the x axis):

The range for the above data would be 4 (5 - 1).

Now append an outlier value:

1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 10

The range for the above data would now become 9 (10 - 1) even though there are no new values inbetween. The outlier has distorted the range.

Quartiles

In order to avoid outliers we should focus on the central tendency of the data. In order to do that we need to split our data into quarters. The first and last quarters (which typically contain outliers) can subsequently be discarded, allowing us to focus on the central quarters.

Given the earlier data (which included an outlier on the far right):

1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 10

This would be split into quarters like so:

[1, 1, 1, 2, 2,]
[2, 2, 3, 3, 3,]
[3, 3, 4, 4, 4,]
[4, 5, 5, 5, 10]

The location within the data where splits have been inserted are known as “quartiles”:

[...] quartile [...] quartile [...] quartile [...]

The lowest quartile is called the “lower quartile” or Q1, while the highest quartile is called the “upper quartile” or Q3. The middle quartile is called the “median” or Q2 (effectively splitting the data in half).

Here are those splits but using the relevant names applied (Q1, Q2 and Q3):

[...] Q1 [...] Q2 [...] Q3 [...]

The range of the values in the two median quarters (inbetween Q1-Q3) are referred to as the “interquartile range”, and provides a way to measure how values are dispersed while being less sensitive to the outliers that would have been found in the lower/upper quartiles.

Below is a (very) rough outline of where the quartiles would be found using our example dataset (each quartile is highlighted in blue). We can see that the interquartile range (the section inside of the outer blue lines) chops off the outlier that is pulling (skewing) the data to the right:

Locating the lower/higher quartiles

Consider the following sorted data:

4, 7, 1, 0, 3, 8, 9, 2, 5, 10, 6

ℹ️ NOTE

there are 11 values in the above data

The calculation for finding the lower quartile is as follows:

N / 4 = round(result)

So this would be 11 / 4, which gives us the result of 2.75 that we then round up to 3. Meaning the lower quartile is the third value in the dataset, so that would be the value 1.

The calculation for finding the higher quartile is as follows:

lower quartile x (N / 4) = round(result)

So this would be 3 x 2.75, which gives us the result of 8.25 that we then round up to 9. Meaning the higher quartile is the ninth value in the dataset, so that would be the value 5.

Meaning the interquartile range can be found in-between the lower/upper quartile positions, thus giving us the central 50% of the data to focus on.

Percentiles

When dealing with quartiles (see above), you’re splitting your data into quarters. When dealing with percentiles, you’re splitting your data into percentages. Each percentage is a percentile. For example, the 20th percentile is the value that is found 20% into your data.

ℹ️ NOTE

quartiles are a type of percentile. For example, the lower quartile is the 25th percentile. Whereas the upper quarter is the 75th percentile. The median (or Q2) is the 50th percentile.

Percentiles are generally used as a means for identifing values for a specific percentage of your users. For example, if your dataset reported performance numbers, then you could look at the value at the 95th percentile of the data and say “95% of our users are seeing this measure of performance”. Which is more relevant than looking at the average (i.e. mean/median/mode) of that performance data.

Another typical example given for understanding the purpose of percentiles is to imagine you scored 50 in a test, and the value in the test results dataset at the 90th percentile was 50, then this would mean you had scored the same as (or beat the score of) 90% of your class who took the test.

Calculating Percentiles

In order to calculate percentiles of a dataset you first need to sort the data in ascending order. Once this is done you need to identify N which is the number of values you have in your dataset. From there you can do the following calculation (where K is the percentile you want the value for):

K x (N / 100)

If the result is not an integer, then round it up and that will give you the position of your percentile. Otherwise if it is an integer, then the value can be found in-between the integer position and the next position along so take the average of the two numbers at those positions to find the percentile value.

For example, if you have 125 numbers in a dataset and you wished to see the 10th percentile for those values, then the calculation would be:

10 x (125 / 100) = round_up(12.5) -> 13

So the 10th percentile would be the value at position 13 in your dataset.

Variance

Measuring the ‘spread’ of data isn’t always as useful as being able to measure the consistency of that spread, in that if our data represented player scores in a game, then we might want to identify how consistent (or reliable) a player was. Did they perform better and more reliably than another player?

The way to measure this is by looking at the mean and then looking at the distance of other values from the mean. This will give us the “variation” we’re looking for. The closer values are to the mean, the more consistent they are.

So going back to the idea of a dataset that reports game player performance. Imagine we have two players (A and B). Both have a mean value that is the same, but player A has a closer spread of values than player B. This means player A is more consistent with their scoring.

The approach for measuring the spread of data that we will look at is called the “variance” and it specifically measures, not from the ‘mean’, but the ‘mean squared’.

The reason for deciding to measure the spread of values from the ‘mean squared’ is because maths (read Head First Statistics for the gory details). To be a little more specific, it’s because if we don’t calculate things from the mean squared it can result in an incorrect value such as zero.

The variance calculation is:

Σ(μ-x)²/n

That might look a little hairy, but most of these symbols we’ve already seen.

μ: this is the dataset mean (which was shorthand for Σx/n).
- Σ represents the sum of a dataset’s values.
- x is the value for a particular dataset item.
- n is the number of items in the dataset.
- Σx thus means: sum all values in the dataset.
- Σx/n thus means: divide the total sum of numbers in the dataset by the number of items in the dataset.
Σ(μ-x)² thus means: we square the result of μ-x and sum Σ all the resulting values.

For example, imagine we have the following data:

1, 2, 9

The mean average for this data is 4. The distance between 1 and 4 is 3 (e.g. 4 is three numbers away from 1: so count from 1 three numbers away and you’ll reach 4 like so, 2, 3, 4). The distance between 2 and 4 is 2, and the distance between 9 and 4 is -5.

Hence the resulting distances we get from the mean are: 3, 2, -5. There’s more to do, but first let’s clarify the negative/postive aspect of these numbers (i.e. if you’re confused about why the distance between 9 and the mean is -5 and not 5, then read on)…

You need to realise when measuring the distance to the mean that you’re actually counting either negative or positive in direction (this becomes clearer where you look at numbers visually):

1, 2, 3, [4], 5, 6, 7, 8, 9

ℹ️ NOTE

I’ve put a square bracket around the number 4 to indicate that it’s the mean average.

If we look at the number 1 and count the distance to 4, we’re effectively counting forward (i.e. positive) towards 4 (similarly with the number 2). But with 9 we have to count backwards (i.e. negative) towards 4.

For example: counting 8,7,6,5,4 shows that 9 is five numbers away from 4. But as we’re treating this as a negative distance, the value is -5. Whereas when calculating the distance from either 1 or 2 to the mean, the mean (4) is ahead of it and so you count positive not negative.

OK so we now have the distances 3, 2, -5, from here we need to square these numbers and then add them up (we have to square the numbers otherwise the sum result would be zero).

Finally, we divide by N (the number of items in the dataset).

This ultimately gives us the “variance” of 12.67.

ℹ️ NOTE

here’s a shorter calculation for the variance Σx²/n - μ²

The reason the variance is useful is because it has provided the measure of distance from the mean based on every value in the dataset. The downside to this approach is that you’ve calculated the variance from the mean squared of the dataset and not just the mean.

In order to have a more relative variance number (i.e. our variance result is based on the mean squared, but in practice we’ll be dealing mainly with a normal mean value, and not a squared mean value) we’ll want to use something called “standard deviation”…

Standard Deviation

All standard deviation means is: take the variance result and find the square root.

If we calculate the standard deviation of the variance value 12.67 we’ll find the result is 3.56 and so that value is the actual distance most values are away from the mean.

You would then compare this standard devication to a standard deviation of another player’s results to see which player ultimately performed better overall (i.e. if the standard deviation for player A was smaller than player B, then player A is a more consistent player).

ℹ️ NOTE

standard deviation has its own Greek symbol σ (referred to as the lowercase Sigma). Remember to calculate σ you start by calculating the variance, and then take the square root.

Some other real-world examples would be a company that manufactures machine parts. For them, they want the standard deviation for their data to be small so they can be sure all the parts they build are the same size. Whereas if you were inspecting wages across a large organisation you would likely find that the standard deviation naturally becomes quite large.

Standard deviation is also measured in the same units as your data. So if your dataset values are, for example, centimeters and the standard deviation is 1, this means that the values are typically 1 centimeter away from the mean.

Conclusion

So there we have it, a run through of some statistic basics. We’ve covered a few different graph types (pie, bar, histograms, line) and explained what averages are and how they’re calculated in different scenarios. We’ve also looked at how data is distributed and how you might measure that to identify whether some results are more consistent than others.

Statistics is such a large topic area, and as far as I can tell, this is really only scratching the surface. To be honest I’m not sure how much further I myself will go with my learning. I feel I now have enough understanding to help me get by in my work and know enough to be dangerous in conversation. Otherwise if there’s anything I discover is missing in future, I’ll be sure to return and update this post accordingly.

Thanks for reading. Hopefully there was something useful in here for you.

NSQ Queue Reader Best Practices

Sun, 26 Nov 2017 00:00:00 +0000

Introduction

This post should serve as a guide for best practices when dealing with services that consume messages from queues and process those messages (we refer to them as QRs or ‘Queue Readers’). The best practices detailed below are from the perspective of both general programming idioms as well as useful performance patterns.

We also are focusing primarily on QRs that use the NSQ data pipeline and specifically for services written in Python (although I imagine most of the items discussed could translate well enough to your queue mechanism of choice).

As with all ‘best practice’ guidelines, they are just that …guidelines. Not everything listed here will be applicable for your needs. So remember to start by verifying your own application’s requirements and specific use cases.

Ephemeral Channels?

Imagine your server instance needs to be restarted, or its nsqd daemon (which receives, queues, and delivers messages to clients) is unexpectedly terminated, or maybe the nsqd exceeds the allocated mem-queue-size (which determines the number of messages that should be kept in memory).

Normally this would mean messages in the queue would be lost. If you’re OK with that scenario and its outcome, then you should append #ephemeral to your channels…

nsq_channel: 'qr_name_goes_here#ephemeral'

Otherwise, the default behaviour for NSQ queues is to persist messages on disk. Which you choose will depend on your application and how critical you feel the messages are.

Fail quickly

When processing a high throughput of messages it’s beneficial to identify invalid messages quickly, then mark them as “processed” so you can exit your handler as quickly as possible and so not cause undue processing stress on your application and/or upstream service(s).

You should wrap potentially problematic code in a try/except (e.g. a function that makes HTTP requests can have multiple types of exceptions raised). Doing this means you can isolate that specific call and handle the failing scenarios appropriately.

Verify your message handling logic

You should understand the complete request flow of your message handling function(s) and be sure you are correctly dropping and/or re-queuing messages at the appropriate places within your application code. It’s very easy to not re-queue (or drop messages) by mistake.

When processing messages synchronously you typically just return True (message was processed) or False (requeue this message) from your handler. But in order to process messages asynchronously you need to call nsq_msg.enable_async() and then you’ll need to make sure you explicitly return either nsq_msg.finish() or nsq_msg.requeue().

Be wary of global variables

Most of the time global variables can be more performant as you’re reusing a pointer to some data, but there are some cases where a long-living (and large) global object (such as a boto S3 connection) might end up leaking memory. This is something that should be measured and verified using the appropriate Python profiling tools first though.

Instrument timers around your primary message handler

It’s important to be able to identify anomalies in the performance of your message handlers. By using a decorator to time the function you can set up appropriate dashboards and alarms.

from your_metrics_abstraction import metrics

@metrics.timed("message_handler.time")
async def message_handler(nsq_msg):
    ...

Pynsq doesn’t support coroutines

The pynsq library only supports a ‘callback’ form of asynchronous message processing. Meaning if you were to define a message handler using a decorator like @gen.coroutine or a native async syntax (either one will convert the function into a coroutine) it will end up breaking the QR application by exiting the handler immediately.

See the next section for an example code snippet that works around this issue by utilising Tornado’s ioloop directly to schedule the handler’s asynchronous execution.

Prevent messages backing up in the queue

Messages can build up and cause alarms to fire if they are not pulled from the queue and successfully processed by your application in a timely fashion. You can help resolve this by either configuring the nsq.Reader#max_in_flight attribute and/or processing your messages asynchronously.

from tornado import ioloop

async def coro_handler(msg):
    # ... do stuff
    return msg.finish()

def handler(msg):
    msg.enable_async()
    ioloop.IOLoop.current().add_callback(coro_handler, msg)

You can also look to tweak the nsq.Reader#max_tries attribute, which defines the number of times a message can be requeued before it is permanently dropped (this prevents cyclic errors).

There is also the nsq.Message#touch method which lets you indicate to the NSQ daemon that you need more time to process the message and thus postpone (for a little while at least) the message processing from timing out and being automatically re-queued (depending on the setting of the max_tries attribute).

Avoid API libraries auto-retrying expensive operations

Some API libraries, such as boto, allow you to configure it so that operations are retried N number of times before finally failing. This can be helpful to ensure a temporary network blip or error doesn’t cause a message to be unnecessarily dropped or requeued. But this can also bring a performance overhead if the operation in question is very slow. Review the API calls you are making and evaluate how expensive they are. In some cases you might prefer to configure “retries” off and have NSQ handle these temporary errors (i.e. by re-queuing messages).

Below is an example of how to configure boto to not retry operations:

s3_resource = session.resource("s3", config=Config(
    connect_timeout=2, 
    read_timeout=2,
    retries={'max_attempts': 0}
  )
)

ℹ️ NOTE

as per the example above, it’s worth tweaking the connection/read timeouts as well. For example we noticed that calls for .xml files from S3 were really slow and so in that service we had to increase the read_connection by a significant amount (but not too much; you don’t want the client to sit hanging for a long period of time, so it requires some fine tuning to get it right).

Place blocking IO operations into a thread pool

Some libraries do not provide asynchronous support (such as Python’s redis library). So if your message handler is asynchronous, and you’re also executing a potentially long running blocking operation (such as an S3 object GET), then this will end up causing your application to block the ioloop and prevent concurrently handling multiple messages.

from app.threadpool import run_on_executor

async def message_handler():
    result = await run_on_executor(fn, arg1, arg2, ...)

Then the app.threadpool referenced in the above snippet would look something like:

from tornado import gen
from concurrent.futures import ThreadPoolExecutor

from bf_rig import settings


THREAD_POOL = ThreadPoolExecutor(settings.get('pool_max_workers'))  # 10


@gen.coroutine
def run_on_executor(*args, **kwargs):
    result = yield THREAD_POOL.submit(*args, **kwargs)
    raise gen.Return(result)

The above example needs to use a Tornado decorator as ThreadPoolExecutor doesn’t work with native coroutines. It would require the use of asyncio.wrap_future which isn’t much better than just using Tornado’s own decorator.

ℹ️ NOTE

the ThreadPoolExecutor will only help you deal with IO bound tasks that need to be handled asynchronously (and whose library doesn’t support natively). If the task to be executed is actually CPU bound then you’ll want to utilise a ProcessPoolExecutor instead.

Rate limit yourself

In a service where there’s a potential for lots of duplicate messages it can be useful to implement some simple rate limiting logic. In one of our QR services we use Redis to track duplicate requests and then execute some basic rate limiting logic in order to prevent overwhelming any upstream services that would otherwise be called.

ℹ️ NOTE

be aware that the rate limit you set can cause unwanted side-effects. For example, if you start to requeue messages during a rate limit period, you may start to see that messages aren’t being processed quickly enough and so the queue depth will begin to increase (i.e. the queue will start to backup and fill up) and this might cause monitors (e.g. systems like Datadog/Nagios) to trigger.

Disable yourself

Consider your upstream services and identify if there’s ever a point where your service needs to stop making requests to it. Most services will be sat behind an API Gateway so they’ll likely enforce rate limiting on you. But that might not always be the case.

One example of this is a QR service which makes requests to a separate rendering service for HTML content to be backed up into AWS S3. There are periods where this rendering service will dynamically purge its cache (both its internal application cache, and also the outer CDN cache layer). In order to prevent the QR service from overloading the rendering service during this period where it’s vulnerable(†), we automatically disable the QR service (we use a shared redis cluster to identify the switch in a key value; so we change it from disabled to enabled).

† due to it having no cache! none of these services we have are vulnerable in the security sense, as they’re internal access only within a VPC

The below example demonstrates an implementation used in one of our QR services, which was to use a Python decorator:

from app.foo import toggle

@toggle('status')
def _message_handler(nsq_msg, *args, **kwargs):
    nsq_msg.enable_async()
    ioloop.IOLoop.current().add_callback(message_handler, nsq_msg)

The app.foo code then looked something like the following:

def toggle_qr(status_key: str) -> Callable:
    """
    When status_key is set to 'stopped' in redis, 
    this decorator will finish the nsq message and return immediately, 
    otherwise it will proceed with event handling.

    Arguments:
        status: name of the status key in redis.
    Returns:
        Wrapped message handler.
    """
    def deco(function):
        @wraps(function)
        def wrapper(nsq_msg: Message, *args, **kwargs):
            assert 'redis' in kwargs, 'redis parameter is required'
            redis = kwargs['redis']
            try:
                status = redis.get(status_key)
            except Exception as e:
                status = None
            if status == b'stopped':
                nsq_msg.finish()
                return
            return function(nsq_msg, *args, **kwargs)
        return wrapper
    return deco

Drop or Requeue?

Consider the previous section about disabling a QR service in times where it might be necessary to protect an upstream (e.g. where rate limiting yourself maybe doesn’t make sense, or being rate limited by the upstream isn’t possible), you might then need to make a decision about what you do with the messages that are building up in the message queue.

Those messages will eventually reach a threshold and in some cases it might make more sense to not requeue messages while the QR service is in ‘disable’ mode, but instead just drop them completely. The answer, and your approach, will depend on the message source itself: are they messages that you can afford to drop? are they generated fairly regularly?

Observability and Monitoring Best Practices

Wed, 15 Nov 2017 00:00:00 +0000

This post aims to discuss key monitoring discussion points and to summarise the relevant best practices when instrumenting application performance monitoring.

Below are some of the areas we’ll be focusing in on…

ℹ️ NOTE

we primarily work with Datadog so you’ll see them mentioned a lot throughout this post.

Terminology

There is a lot of confusion around the difference between certain terms such as “observability”, “monitoring”, “instrumentation” and “telemetry”. Let’s start with defining what each of these mean…

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs – Wikipedia

In that context, “observability” is the word you use when talking about how well your systems are doing in a broad overarching sense (is the system observable?). Beneath the umbrella term “observability” we’ll then find “monitoring” and “instrumentation”.

Monitoring is the translation of IT metrics into business meaning – Wikipedia

In that context, “monitoring” is the word you use when talking about tools for viewing data that has been recorded by your systems (whether that be time series data, or logging etc). These monitoring tools are supposed to help you identify both the “what” and the “why” something has gone wrong.

Instrumentation refers to an ability to monitor or measure the level of a product’s performance, to diagnose errors and to write trace information – Wikipedia

In that context, “instrumentation” is the word you use when talking about how you’re recording data to be viewed and monitored.

Telemetry is the process of gathering remote information that is collected by instrumentation – MSDN

In that context, “telemetry” is the word you use when talking about the mechanisms for acquiring the data that has been gathered by your instrumentation (e.g. tools like FluentD or Syslog).

Understand the different types of monitoring

Although most of this document is based around one specific type of monitoring (APM), it’s good to be aware of the various types of monitoring available across an entire system.

Server monitoring:
monitor the health of your servers and ensure they stay operating efficiently.
Configuration change monitoring:
monitor your system configuration to identify if and when changes to your infrastructure impact your application.
Application performance monitoring:
look inside your application and services to make sure they are operating as expected (also known as APM tooling).
Synthetic testing:
real time interactions to verify how your application is functioning from the perspective of your users (hopefully to catch errors before they do).
Alerting:
notify the service owners when problems occur so they can resolve them, minimizing the impact to your customers.

Data collection methods

There are fundamentally two methods for data collection:

Push: sending metrics to an analysis tool.
Pull: configuring a health check endpoint, that a centralised tool pulls data from.

When dealing with the ‘pull’ model you’ll hear people suggest that rather than a simple ‘200 OK’ response you should add extra information that gives humans more understanding of the overall state of the service.

So this could be things like a successful database connection was opened. But a possible concern would be the performance overhead that might need to be accounted for (remember: health endpoints are generally pinged every few minutes).

There are also various metric types you can collect data as. Two common ones are:

Counter: an ever increasing value.
Gauge: a point-in-time value (can arbitrarily go up and down).
Histogram: samples observations and counts them in configurable buckets.

Histograms might require a little extra clarification: they sample observations (e.g. request durations) and count different perspectives on the data. In the case of ‘request duration’ you’d likely see the different ‘perspectives’ being: count, avg, median, max and 95percentile.

For more information, see these Datadog articles: Metric Types and DogStatsD.

Frontend monitoring

There are two main approaches to frontend monitoring:

Real User Monitoring (RUM).
Synthetic.

The difference between them has to do with the type of traffic that is triggering the data collection. For example, with RUM the requests being processed are from real users, whereas with synthetic monitoring the requests are fake (i.e. they are generated by your own software).

Synthetic monitoring causes data to be collected for analysis, thus allowing you to identify the availability and performance of your system by constructing very specific test cases.

Make it useful, then actionable

Let’s start with a quote from Charity Majors (author of Database Reliability Engineering and CEO of honeycomb.io).

Don’t attempt to “monitor everything”. You can’t. Engineers often waste so much time doing this that they lose track of the critical path, and their important alerts drown in fluff and cruft.

When a monitor triggers an alarm, it should first and foremost be “useful”. Secondly, it should be “actionable”. There should be something you can do to resolve the alarm and also be a set of steps (post-resolution) to prevent that alarm from triggering again.

If the alarm isn’t actionable, then it just becomes noise.

Focus on user impact

Below is a quote from Mike Julian (author of Practical Monitoring and Monitoring Weekly)

Go as deep and wide with your instrumentation as you want, but always be asking yourself, “How will these metrics show me the user impact?”

Or put another way: “Who is your customer”?

Depending on the services you build, your customers might be people who pay money to use your service, or they might be other engineers within your organisation who are consuming an API you’ve developed. Either way, your customer is the most important thing to you and your service. You need to keep them happy.

For most users, whether they be a non-technical paying customer or an engineer, they will have certain criteria that will make them happy. You could imagine the sorts of questions they’ll ask to be something like…

I want the service to work as expected
i.e. you should monitor whatever is determined to be a ‘success rate’ for your service.
I want the service to be fast
i.e. you should monitor the service latency.
I want to use ‘this much’ of the service
i.e. you should monitor things like ‘requests per second’ - do you have SLA’s defined?

In essence you should start by creating monitors for things that have a direct impact on your users.

For example, measuring OS level metrics such as CPU and memory consumption is great for diagnostics & performance analysis, as they help to highlight the underlying system behaviour.

But you should probably not be using these stats for alarming, as their values can have relative meaning in different situations and they don’t necessarily offer much in the way of understanding the root cause of the problem.

Instead, try monitoring 5xx errors and very slow latency times. These metrics are much more useful indicators of problems with the system and real negative user experiences.

Ultimately, the deeper you go, the more specific your alarms become, and the less useful they are at identifying trends and patterns.

Favour organic changes over static thresholds

Static thresholds such as “the number of errors reported has exceeded N” have a habit of raising false alarms, due typically to unexpected spikes in data (anomalies).

This happens so frequently that most monitoring tools/services (such as nagios) offer “flapping detection” to help prevent these deviations.

To help with this services such as Datadog offer a feature called “recovery thresholds” which helps to quieten monitor state changes so you can be confident when a monitor switches back to OK state that it has in fact definitely resolved itself.

The way it works is like so: you give Datadog a threshold value that must be met to consider the monitor “back to normal”. Once the monitor state switches to ALARM it will now never flip-flop between OK and ALARM. It will only ever go back to OK if the set recovery threshold goes below the specified value.

They also offer “anomaly detection”, which detects when a metric is behaving differently than it has in the past, taking into account trends, seasonal day-of-week and time-of-day patterns. This can be more useful for organically identifying issues, as it allows for buffer zones around your static thresholds.

Datadog also offers “outlier monitoring” which detects when a specific member of a group (e.g., hosts, availability zones, partitions) is behaving unusually compared to the rest. They are useful for noticing when a given group, which should behave uniformly, isn’t doing so.

ℹ️ NOTE

A summary of Datadog’s various detection methods can be found here.

Send critical and noncritical alarms to different channels

At BuzzFeed I work in the software infrastructure team and there we have two separate Slack channels for handling monitoring notifications:

#oo-infra-alarms
#oo-infra-warnings

Only alarms that require immediate review (such as a 5xx monitor) goes into #oo-infra-alarms. Everything else is sent to #oo-infra-warnings because although important, they aren’t surfacing immediately as user issues.

If you don’t do this, then you’ll find people become fatigued by the sheer amount of noise coming from multiple alarms. Especially alarms that are firing due to unactionable anomalies.

To quote (again) Charity Majors…

In the chaotic future we’re all hurtling toward, you need the discipline to have radically fewer paging alerts - not more.

You should also consider sending a monitor’s “warning” state to a different channel for similar reasons. You can define different channels in Datadog using the following template code:

{{#is_alert}}
@slack-my-channel-for-serious-alarms
{{/is_alert}}

{{#is_warning}}
@slack-my-channel-for-just-warnings-to-keep-an-eye-on
{{/is_warning}}

Give context

When an monitor triggers an alarm, and you’re on-call that night, then you might be unfamiliar with the system and its dependencies. One quick way to help people on-call is to provide them with additional context about the alarm and the affected system.

A general message template could look something like the following…

<Alarm Title>
<Alarm Summary>
<Concise service description>
<Monitoring links>

An example might look something like:

Foo-Service 5xx

Foo-Service is serving a high number of 5xx responses, 
these will not be cached at CDN, possibly resulting in further cache misses.

Foo-Service serves responsive article pages for the BuzzFeed website (www.buzzfeed.com) 
and is fronted by a CDN. It has an upstream dependency on Redis and the internal Buzz API v2.

Please refer to the monitoring for more details:

- Logs
- Dashboard
- Timeboard
- Request Breakdown

Alarms highlight the symptom and not the cause. So if at all possible, try to include information or data that might aid the on-call person in identifying the root cause of the issue.

Think about data aggregation

When dealing with TSDB’s (Time Series Database) you’ll find they will start aggregating multiple data points into a single data point. This is known as the “roll up” effect.

ℹ️ NOTE

if you weren’t aware, a TSDB is made up of key/value pairs. The key is the timestamp and the value is the metric value at that point in time.

The problem with ‘rolling up’ data is that it smooths out your data points. Meaning you shift from a graph that has lots of spikes (i.e. a graph that shows every possible false positive), to a graph that covers up those false positive spikes.

On the plus side, rolling up data like this means you get to see the data at a higher level and so patterns of use start to emerge.

There have been examples in the past where important spikes in CPU/Memory were not captured due to the smoothing out of aggregated data and so it can be useful to look at your data closely (at first) and then in some instances force the disabling of rolling up data using Datadog’s .rollup() method.

Ultimately, you’ll need to find the balance that works best for you and your monitoring requirements.

ℹ️ NOTE

you can read more about this smoothing out process here, as well as the .rollup() method Datadog provides to allow you to control this behaviour.

Know your graphs

We won’t repeat the details here, but suffice to say, each graph in Datadog has a purpose and specialised use case. You should review Datadog’s articles on the various graphs they offer and the whys/when of using them:

UPDATE

I’ve since written a blog post about stastistics (aimed at beginners) and so by understanding the basics of statistics you’ll be able to understand more clearly what certain graphs represent and how.

You can read the post here: Statistics and Graphs: The Basics

Map your graphs

It can be useful to order your graphs (within a dashboard/timeboard) chronologically. For example, CDN -> LB -> Service. This can help you mentally model the request flow through your system, such that you know the request starts by hitting your CDN layer, it’s then routed inside of your infrastructure and hits a load balancer, finally that load balancer distributes the request to a specific service node.

It can equally be useful to collate multiple services (and their graphs) within a single overarching dashboard, because when there is a problem in the system you can follow the request flow from start to finish and see where a bottleneck (or anomaly) somewhere else in the chain is causing a side effect elsewhere in the chain.

An alternative approach is to have a dashboard that focuses on the key metrics for a service’s performance, and underneath that they’ll have graphs that monitor their dependencies. So when an engineer gets a call because of an issue that seems to be with their service, they’ll check the dashboard and might see there’s an issue upstream of them with one of their dependencies.

Some companies even take that approach a step further and formalize this process and subsequently define a standardized structure for dashboards (i.e. all dashboards are structurally the same). The benefit of that approach is that people on-call can start at the beginning of a request and then follow the dashboards like a thread until they reach a service that is the root cause of the problem being reported.

Choosing between a metric or log

In order to help individual teams identify whether they should collect data as a metric or as a log, one recommended approach is to ask the following questions:

Is it easier for your team to think about metrics or logs?
Is it more effective for the thing in question to be a log entry or metric?
What questions do you typically ask when debugging?

We can’t answer these questions for you, but we have generally found the following approach works reasonably well as a generic pattern (YMMV)…

Collect an exception as both a log and an error.
- The log helps add additional context not available in the metric.
- The metric helps with monitoring and triggering alarms.
Log only errors/exceptions.
- “No news is good news”.
- Control other log calls using log levels (so they can be enabled when necessary).
Include unique identifiers with your logs
- This helps to quickly figure out what the log is possibly associated with when looking from a centralized distributed log system which contains many logs aggregated from many distinct services.
Mostly everything else we’ll record as a metric so we can monitor pattern changes.

Other useful tips

Datadog tags are useful for splitting metrics by type (e.g. status codes).
Datadog events, are useful for capturing additional info (e.g. exception message).
99% of the time you want a Timeboard, not a Screenboard.
- Timeboards allow for tracking data points across multiple graphs at once.
Let people know where the dashboards are, Slack pinned, Runbooks etc
For latency use 95th percentile (standard deviation), not just the ‘mean average’.
- Because the mean can miss important slow requests.
Load balancer metrics can also be useful to monitor (especially if service is falling over).

Reference material

Load Testing Guidelines

Mon, 13 Nov 2017 00:00:00 +0000

Introduction

This post should serve as a guideline for running load tests.

It is designed to be a concise list of notes and tips.

For a list of load testing tools, including their reviews and benchmark results, I refer you to the “Tools” section at the end.

Use real datasets

Parse your access logs for real user requests, then replay those requests within your load test. Also be mindful of different time periods for more accurate distribution of transactions. Look for realistic worst case situations (there may be more than one).

Identify collateral damage

What users or upstreams will be affected by this additional load?

Are there any vulnerable dependencies that should be: notified of the load test or protected from it? † (see next section)

† not all services are completely isolated. Consider what happens when your services is using an external API service. If the API is doing its job right it should start rate limiting you (or denies you access for exceeding your thresholds). But that still doesn’t necessarily mean you have to be a bad web citizen and start hammering their service (on a side note: you should probably be considering caching).

Stub services if necessary

Rather than hit a real upstream, try creating a mock version. The benefit of doing this is that you can your mock service to allow for controlling the latency and/or throughput restraints (thus mimicking different failure scenarios and seeing how your service behaves under different conditions).

Distribute your traffic

When your service response is fairly big, it’s easy to hit a network capacity limit, so if you’re seeing that your service doesn’t scale with addition of new instances, this may be the root cause. To help avoid this, use distributed testing (here’s an example wrapper to simplify the process).

Automate for reproducible runs

Write simple scripts that let you coordinate a fresh load test run. Such as configuring mock services and defining specific strategies (e.g. running a ‘soak’ test vs. a shorter duration load test that mimicks a slow ramp-up in traffic pattern vs a load test that mimicks a ‘thundering herd’ approach).

Consider disabling authentication

It can be hard to include auth tokens/keys in load testing tools, and as such it is often easier to use a network secured backend and a custom code branch that either allows for auth bypass or has no authentication.

ℹ️ NOTE

although being able to load test with authentication is important as it could highlight important problem points in your architecture design.

Don’t immediately scale dependencies

If you have a dependency such as redis (or another type of data store), then leave it configured to the size it currently is. If you have a stage environment, it can be tempting to configure that stage resource to be identical to your production environment. But it would be better to first verify if that resource is even a problem point by analysing the results of an initial load test with all resources configured normally for that environment.

Once you’ve run your load test, if you find you’re having no issues, then sure you could consider increasing/scaling up the resource in question. But ultimately if it wasn’t a problem before, then it is unlikely to be an issue once it has even more compute/memory/network/etc.

Send traffic from different geographical regions

It can be beneficial to set-up load testing instances in multiple regions and availability zones. This is ideal for a highly dynamic application expected to be globally utilized, as it means you can accurately simulate traffic with different latency expectations. Although, if you have a simple/static application, maybe the use of CDN edge caching is enough to mitigate concern.

Decide if your tool should support reusing connections

A tool such as Siege doesn’t support reusing existing http connections. This can be interesting as far as identifying how your system behaves under those conditions. But reusing existing connections is mostly preferred so that throughput can be properly verified. Tools such as Vegeta and Wrk support connection reuse and have better performance/features.

Start testing a single instance, before moving onto clusters

We should ideally identify the threshold of a single instance before moving onto testing a cluster of instances. This is so that we can make appropriate tweaks to our application(s) based on the initial load testing results and should then help improve the initial cluster results as well.

Verify load using multiple strategies

There are multiple types of load testing strategies: constant, ramp-up, soak test (and more). Research and identify which is the most appropriate for your service under test. It may be that you’ll want to execute multiple strategies.

Reset your environment between test runs

Ensure your system is back to ‘normal’ (that means different things to different services) before starting another load test. Otherwise your test results will be skewed by the system being in a ‘hot’ state and could also mean putting your upstream services under DoS like scenarios.

Document the results

It is beneficial for all (i.e. your team and others) to be able to review the load test results. Ensure each test is documented in a consistent, known and obvious location. For example, store them in a /docs/load-tests folder inside your service code base and include any key fixes (e.g. changes made that resolved a problem in your service performance).

Tools

There are various load/stress testing tools available.

The following, for me, are the top three open-source choices:

For a review of these tools (and many more open-source options), please read:

Logging 101

Sun, 12 Nov 2017 00:00:00 +0000

Logs?

Applications should record information/events to help make debugging (and understanding) what a program is doing easier.

Typically this information is recorded into a log file.

But in some cases it is preferred to send this information to stdout because it then becomes the responsibility of the operating system/environment to determine where and how those logs are stored.

To quote 12factor:

A (service) never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage log files. Instead, each running process writes its event stream, unbuffered, to stdout.

Applications should be conscious of the volume and quality of the logs they emit. Logging isn’t free, and high volume logs aren’t necessarily more useful than fewer, thoughtful, log messages. Include important context with your log messages to be able to better understand the state of the system at that time.

You may discover that some log information is better off not logged but recorded as time series metrics so they can be provided and analysed by tools such as Datadog. The benefit of doing this is that you can more easily measure those metrics (and gain more insights) over time.

It can also, depending on the tools you use to analyse your log data (e.g. external log analysis system, such as Papertrail), be better to log data in a more structured format in order to provide better contextual information and to make filtering logs easier. For more information on structured logging, I recommend this article inspired from “The Art of Monitoring”.

Levels

When recording log data, there are various ‘levels’ you can utilise that indicate different severities.

Below are some common log levels along with a short description that describes when you would/should expect to use that particular level…

ℹ️ NOTE

I saw these descriptions in a tweet a long time ago, and saved them off, and then months later when I’ve come to write this post I have since lost the link to the original tweet. If someone knows what it is, could you ping me on twitter and I’ll get the author credited appropriately.

Fatal

Wake me up at 4am on a Sunday.

Error

Apologize to the user and raise a ticket.

Warn

Make a note in case it happens again.

Info

Everything's fine, just checking in.

Debug

Fill my hard-drive with stack traces.

Quality

There are many “logging best practice” articles, below is a short bullet list pulled from Splunk.

I recommend reading through their article for the full details.

Use clear key-value pairs
Create events that humans can read
Use timestamps for every event
Use unique identifiers
Log in text format
- e.g. log an image’s attributes, but not the binary data itself
Use developer-friendly formats
- e.g. json
Log more than just debugging events
Use categories/levels
- e.g. info, warn, error, debug
Identify the source
Keep multi-line events to a minimum

Fastly Varnish

Thu, 02 Nov 2017 00:00:00 +0000

UPDATE

As of October 2020 I joined the Fastly team to help improve Developer Relations 🎉

This means one of my many responsibilities will be to make this blog post redundant by ensuring the developer.fastly.com content is as relevant and valuable as possible.

But until that time, remember that this post was written (and updated), from the perspective of a long time customer, over many years and so the tone and personality of this blog post doesn’t represent the opinions of Fastly in any way.

If you have any questions, then feel free to reach out to me.

In this post I’m going to be explaining how the Fastly CDN works, with regards to their ‘programmatic edge’ feature (i.e. the ability to execute code on cache servers nearest to your users).

We will be digging into quite a few different areas of their implementation, such as their clustering solution, shielding, and various gotchas and caveats to their service offering.

Be warned: this post is a monster!
It’ll take a long time to digest this information…

Introduction

Fastly utilizes ‘free’ software (Varnish) and extends it to fit their purposes, but this extending of existing software can make things confusing when it comes to understanding what underlying features work and how they work.

In short: Varnish is an HTTP accelerator.\ More concretely it is a web application that acts like a HTTP reverse-proxy.

You place Varnish in front of your application servers (those that are serving HTTP content) and it will cache that content for you. If you want more information on what Varnish cache can do for you, then I recommend reading through their introduction article (and watching the video linked there as well).

Fastly is many things, but for most people they are a CDN provider who utilise a highly customised version of Varnish. This post is about Varnish and explaining a couple of specific features (such as hit-for-pass and serving stale) and how they work in relation to Fastly’s implementation of Varnish.

One stumbling block for Varnish is the fact that it only accelerates HTTP, not HTTPS. In order to handle HTTPS you would need a TLS/SSL termination process sitting in front of Varnish to convert HTTPS to HTTP. Alternatively you could use a termination process (such as nginx) behind Varnish to fetch the content from your origins over HTTPS and to return it as HTTP for Varnish to then process and cache.

Simple, right?

ℹ️ NOTE

Fastly helps both with the HTTPS problem, and also with scaling Varnish in general.

The reason for this post is because when dealing with Varnish and VCL it gets very confusing having to jump between official documentation for VCL and Fastly’s specific implementation of it. Even more so because the version of Varnish Fastly are using is now quite old and yet they’ve also implemented some features from more recent Varnish versions. Meaning you end up getting in a muddle about what should and should not be the expected behaviour (especially around the general request flow cycle).

Ultimately this is not a “VCL 101”. If you need help understanding anything mentioned in this post, then I recommend reading:

Fastly also has a couple of excellent articles on utilising the Vary HTTP header, which is highly recommended reading.

Lastly, as of May 2020, Fastly has started rolling out an updated developer portal which hopes to address some of the ‘documentation’ issues I’ve noted in this post.

Fastly reached out to me to review their updated developer portal (inc. the new layout of their reference and API documentation) and requested my feedback. I was happy to report that (other than a few minor comments) I found it to be a good start and I’m very much looking forward to many future improvements.

OK, let’s crack on…

Varnish Basics

Varnish is a ‘state machine’ and it switches between these states via calls to a return function (where you tell the return function which state to move to). The various states are:

recv: request is received and can be inspected/modified.
hash: generate a hash key from host/path and lookup key in cache.
hit: hash key was found in the cache.
miss: hash key was not found in the cache.
pass: content should be fetched from origin, regardless of if it exists in cache or not, and response will not be cached.
pipe: content should be fetched from origin, and no other VCL will be executed.
fetch: content has been fetched, we can now inspect/modify it before delivering it to the user.
deliver: content has been cached (or not, depending on what you return in vcl_fetch) and ready to be delivered to the user.

For each state there is a corresponding subroutine that is executed. It has the form vcl_<state>, and so there is a vcl_recv, vcl_hash, vcl_hit etc.

As an example, in vcl_recv to change state to “pass” you would execute return(pass). If you were in vcl_fetch and wanted to avoid caching the content (i.e. move to vcl_pass before moving to vcl_deliver), then you would execute return(pass) otherwise executing return(deliver) from vcl_fetch would cache the origin response and move you to vcl_deliver directly.

For a state such as vcl_miss, when that state function finishes executing it will automatically trigger a request to the origin/backend service to acquire the requested content. Once the content is requested, then we end up at vcl_fetch where we can then inspect the response from the origin.

This is why at the end of vcl_miss we change state by calling return(fetch). It looks like we’re telling Varnish to ‘fetch’ data but really we’re saying move to the next logical state which is actually vcl_fetch.

Lastly, as Varnish is a state machine, we have the ability to ‘restart’ a request (while also keeping any modifications you might have made to the request intact). To do that you would call return(restart).

^^ Varnish in action

States vs Actions

According to Fastly vcl_hash is the only exception to the rule of return(<state>) because the value you provide to the return function is not a state per se. The state function is called vcl_hash but you don’t execute return(hash). Instead you execute return(lookup).

Fastly suggests this is to help distinguish that we’re performing an action and not a state change (i.e. we’re going to lookup the requested resource within the cache).

Although vcl_miss’s return(fetch) is a bit ambiguous considering we could maybe argue it’s also performing an ‘action’ rather than a state change. Let’s also not forget operations such as return(restart) which looks to be triggering an ‘action’ rather than a state change (in that example you might otherwise expect to execute something like return(recv)).

Varnish Default VCL

When using the free version of Varnish, you’ll typically implement your own custom VCL logic (e.g. add code to vcl_recv or any of the other common VCL subroutines). But it’s important to be aware that if you don’t return an action (e.g. return(pass), or trigger any of the other available Varnish ‘states’), then Varnish will continue to execute its own built-in VCL logic which sits beneath your custom VCL.

You can view the ‘default’ (or ‘builtin’) logic for each version of Varnish via GitHub:

Varnish v2.1 (the version used by Fastly)
Varnish v3.0
Varnish v4.0
Varnish v5.0

ℹ️ NOTE

after v3 Varnish renamed the file from default.vcl to builtin.vcl.

But things are slightly different with Fastly’s Varnish implementation (which is based off Varnish version 2.1.5).

Specifically:

no return(pipe) in vcl_recv, they do pass there
some modifications to the synthetic in vcl_error
…and God knows what else.

Fastly Default VCL

On top of the built-in VCL the free version of Varnish uses, Fastly also includes its own ‘default’ VCL logic alongside your custom VCL.

When creating a new Fastly ‘service’, this default VCL is added automatically to your new service. You are then free to remove it completely and replace it with your own custom VCL if you like.

See the link below for what this default VCL looks like, but in there you’ll notice code comments such as:

#--FASTLY RECV BEGIN

...code here...

#--FASTLY RECV END

ℹ️ NOTE

those specific portions of the default code define critical behaviour that needs to be defined whenever you want to write your own custom VCL.

Fastly has some guidelines around the use (or removal) of their default VCL which you can learn more about here.

Below are some useful links to see Fastly’s default VCL:

ℹ️ NOTE

Fastly also has what they call a ‘master’ VCL which runs outside of what we (as customers) can see, and this VCL is used to help Fastly scale varnish (e.g. handle things like their custom clustering solution).

fastly’s master VCL be all like ^^

Custom VCL

When adding your own custom VCL code you’ll need to ensure that you add Fastly’s critical default behaviours, otherwise things might not work as expected.

The way you add their defaults to your own custom VCL code is to add a specific type of code comment, for example:

sub vcl_recv {
  #FASTLY recv
}

See their documentation for more details, but ultimately these code comments are ‘macros’ that get expanded into the actual default VCL code at compile time.

It can be useful to know what the default VCL code does (see links in previous section) because it might affect where you place these macros within your own custom code (e.g. do you place it in the middle of your custom sub routines or at the start or the end).

This is important because, for example, the default behaviours Fastly defines for vcl_recv is to set a backend for your service. Your custom VCL can of course override that backend, but where you define your custom code that does that overriding might not function correctly if placed in the wrong place.

Here is an example of why it’s important to know what is happening inside of these macros: we had a conditional comment that looked something like the following…

if (req.restarts == 0) {
  ...set backend...
}

…later on in our VCL we would trigger a request restart (e.g. return(restart)), but now req.restarts would be equal to 1 and not zero and so when the request restarted we wouldn’t set the backend.

Now, this of course is a bug in our code and has nothing to do with the default Fastly VCL (yet). But what’s important to now be aware of is that our requests didn’t just ‘fail’ but were sent to a different origin altogether (which was extremely confusing to debug at the time).

Turns out what was happening was that the default Fastly VCL was selecting a default backend for us, and this selection was based on the age of the backend!

To quote Fastly directly…

We choose the first backend that was created that doesn’t have a conditional on it. If all of your backends have conditionals on them, I believe we then just use the first backend that was created. If a backend has a conditional on it, we assume it isn’t a default. That backend is only set under the conditions defined, so then we look for the oldest backend defined that doesn’t have a conditional to make it the default.

Be Careful!

We experienced a problem that broke our production site and it was related to implementing our own vcl_hash subroutine.

The problem wasn’t as obvious as you might think. We didn’t implement vcl_hash to change the hashing algorithm, but instead we wanted to add some debug log calls into it.

We looked at the VCL that Fastly generated before we added our own vcl_hash subroutine and that VCL looked like the following…

sub vcl_hash {
#--FASTLY HASH BEGIN
  #if unspecified fall back to normal
  {
    set req.hash += req.url;
    set req.hash += req.http.host;
    set req.hash += "#####GENERATION#####";
    return (hash);
  }
#--FASTLY HASH END
}

We thought “OK, that’s what Fastly is generating, so we’ll just let them continue generating that code, nothing special we need to do” …wrong!

So we added the following code to our own VCL…

sub vcl_hash {
  #FASTLY hash

  call debug_info;

  return(hash)
}

The expectation was that the #FASTLY hash macro would still include all the code from inbetween #--FASTLY HASH BEGIN and #--FASTLY HASH END (see the earlier code snippet).

What actually ended up happening was that the Fastly macro dynamically changed itself to not include critical behaviours

Notice the set req.hash += req.url; and set req.hash += req.http.host; that they originally were generating? Yup. They were no longer included. This caused the system caching to blow up.

The code that was being generated now looked like the following…

#--FASTLY HASH BEGIN
# support purge all
set req.hash += req.vcl.generation;
#--FASTLY HASH END

So to fix the problem we had to put those missing settings manually back into our own vcl_hash subroutine…

sub vcl_hash {
  #FASTLY hash

  set req.hash += req.url;
  set req.hash += req.http.host;

  call debug_info;

  return(hash);
}

Interestingly Fastly’s default VCL doesn’t require us to also set set req.hash += "#####GENERATION#####";, so they happily keep that part within their generated code 🤦

Example Boilerplate

Before we move on, while we’re discussing custom VCL, I’d like to share with you some VCL boilerplate I typically start out all new projects with (which I then modify to suit the project’s requirements, but ultimately this VCL consists of all the standard stuff you’d typically would need):

#
# detailed blog post on fastly's implementation details:
# https://www.integralist.co.uk/posts/fastly-varnish/
#
# fastly custom vcl boilerplate:
# https://docs.fastly.com/vcl/custom-vcl/creating-custom-vcl/#fastlys-vcl-boilerplate
#
# states defined in this file:
# vcl_recv
# vcl_error
# vcl_hash
# vcl_pass
# vcl_miss
# vcl_hit
# vcl_fetch
# vcl_deliver
#

table deny_list {
  "/bad-thing-1": "true",
  "/bad-thing-2": "true",
}

sub set_backend {
  set req.backend = F_httpbin;

  if (req.http.Host == "example-stage.acme.com/") {
   set req.backend = F_httpbin_stage;
  }
}

sub vcl_recv {
  #FASTLY recv

  call set_backend;

  # configure purges to require api authentication:
  # https://docs.fastly.com/en/guides/authenticating-api-purge-requests
  #
  if (req.method == "FASTLYPURGE") {
      set req.http.Fastly-Purge-Requires-Auth = "1";
  }

  # force HTTP to HTTPS
  #
  # related: req.http.Fastly-SSL
  # https://docs.fastly.com/en/guides/tls-termination
  #
  if (req.protocol != "https") {
    error 601 "Force SSL";
  }

  # fastly 'tables' are different to 'edge dictionaries':
  # https://docs.fastly.com/en/guides/about-edge-dictionaries
  #
  if (table.lookup(deny_list, req.url.path)) {
    error 600 "Not found";
  }

  # don't bother doing a cache lookup for a request type that isn't cacheable
  if (req.method !~ "(GET|HEAD|FASTLYPURGE)") {
    return(pass);
  }

  if (req.restarts == 0) {
    # nagios/monitoring cache bypass
    #
    if (req.url ~ "123") {
      set req.http.X-Monitoring = "true";
      return(pass);
    }
  }

  return(lookup);
}

sub vcl_error {
  #FASTLY error

  # fastly synthetic error responses:
  # https://docs.fastly.com/en/guides/creating-error-pages-with-custom-responses
  #
  if (obj.status == 600) {
    set obj.status = 404;

    synthetic {"
      <!doctype html>
      <html>
        <head>
          <meta charset="utf-8">
          <title>Error</title>
        </head>
        <body>
          <h1>404 Not Found (varnish)</h1>
        </body>
      </html>
      "};

    return(deliver);
  }

  # fastly HTTP to HTTPS 301 redirect:
  # https://docs.fastly.com/en/guides/generating-http-redirects-at-the-edge
  #
  # example:
  # curl -sD - http://example.acme.com/
  # curl -H Fastly-Debug:1 -sLD - -o /dev/null http://example.acme.com/?cachebust=$(uuidgen)
  #
  if (obj.status == 601 && obj.response == "Force SSL") {
    set obj.status = 301;
    set obj.response = "Moved Permanently";
    set obj.http.Location = "https://" req.http.host req.url;
    synthetic {""};
    return (deliver);
  }
}

sub vcl_hash {
  #FASTLY hash

  set req.hash += req.url;
  set req.hash += req.http.host;

  # call debug_info_hash;

  return(hash);
}

sub vcl_pass {
  #FASTLY pass
}

sub vcl_miss {
  #FASTLY miss

  return(fetch);
}

sub vcl_hit {
  #FASTLY hit
}

sub vcl_fetch {
  #FASTLY fetch

  # fastly caching directive:
  # https://docs.fastly.com/en/guides/cache-control-tutorial
  #
  # example:
  # define stale behaviour if none provided by origin
  #
  if (beresp.http.Surrogate-Control !~ "(stale-while-revalidate|stale-if-error)") {
    set beresp.stale_if_error = 31536000s; // 1 year
    set beresp.stale_while_revalidate = 60s; // 1 minute
  }

  # fastly stale-if-error:
  # https://docs.fastly.com/en/guides/serving-stale-content
  #
  if (beresp.status >= 500 && beresp.status < 600) {
    if (stale.exists) {
      return(deliver_stale);
    }
  }

  # hit-for-pass:
  # https://www.integralist.co.uk/posts/fastly-varnish/#hit-for-pass
  #
  if (beresp.http.Cache-Control ~ "private") {
    return(pass);
  }

  return(deliver);
}

sub vcl_deliver {
  #FASTLY deliver

  # fastly internal state information:
  # https://docs.fastly.com/en/guides/useful-variables-to-log
  #
  set resp.http.Fastly-State = fastly_info.state;

  if (req.http.X-Monitoring == "true") {
    set resp.http.X-Monitoring = req.http.X-Monitoring;
  }

  return(deliver);
}

UPDATE 2020.02.25: Fastly have published a blog post that details what they consider to be VCL anti-patterns and offer solutions/alternative patterns: https://www.fastly.com/blog/maintainable-vcl

Fastly TTLs

When Fastly caches your content, it of course only caches it for a set period of time (known as the content’s “Time To Live”, or TTL). Fastly has some rules about how it determines a TTL for your content.

Their ‘master’ VCL sets a TTL of 120s (this comes from Varnish rather than Fastly) when no other VCL TTL has been defined and if no cache headers were sent by the origin.

Fastly does a similar thing with its own default VCL which it uses when you create a new service. It looks like the following and increases the default to 3600s (1hr):

if (beresp.http.Expires || 
    beresp.http.Surrogate-Control ~ "max-age" || 
    beresp.http.Cache-Control ~"(s-maxage|max-age)") {
  # keep the ttl here
} else {
  # apply the default ttl
  set beresp.ttl = 3600s;
}

ℹ️ NOTE

3600 isn’t long enough to persist your cached content to disk, it will exist in-memory only. See their documentation on “Why serving stale content may not work as expected” for more information.

You can override this VCL with your own custom VCL, but it’s also worth being aware of the priority ordering Fastly gives when presented with multiple ways to determine your content’s cache TTL…

beresp.ttl = 10s: caches for 10s
Surrogate-Control: max-age=300 caches for 5 minutes
Cache-Control: max-age=10 caches for 10s
Expires: Fri, 28 June 2008 15:00:00 GMT caches until this date has expired

As we can see from the above list, setting a TTL via VCL takes ultimate priority even if caching headers are provided by the origin server.

DNS TTL Caching

There are two fundamental pieces of information I’m about to describe…

Fastly only uses the first IP returned from a DNS lookup, and will cache it until the DNS’ TTL expires. A failed DNS lookup for a service without a Fastly ‘health check’ (see docs) would lead to 600 seconds (10 minutes!) of stale IP use before attempting to requery the DNS.\ An example of why this is bad: we had a 60s DNS TTL to align with that of AWS load balancers, but we discovered we were getting Fastly errors for ten minutes instead of just 1 minute.
Fastly does offer a ‘High Availability’ feature (on request) which allows for traffic distribution across multiple IPs returned from a DNS lookup, rather than just using one until the next DNS lookup.

Caching Priority List

Fastly has some rules about the various caching response headers it respects and in what order this behaviour is applied. The following is a summary of these rules:

Surrogate-Control determines proxy caching behaviour (takes priority over Cache-Control) †.
Cache-Control determines client caching behaviour.
Cache-Control determines both client/proxy caching behaviour if no Surrogate-Control †.
Cache-Control determines both client/proxy caching behaviour if it includes both max-age and s-maxage.
Expires determines both client/proxy caching behaviour if no Cache-Control or Surrogate-Control headers.
Expires ignored if Cache-Control is also set (recommended to avoid Expires).
Pragma is a legacy cache header only recommended if you need to support older HTTP/1.0 protocol.

† except when Cache-Control contains private.

Next in line is Surrogate-Control (see my post on HTTP caching for more information on this cache header), which takes priority over Cache-Control. The Cache-Control header itself takes priority over Expires.

ℹ️ NOTE

if you ever want to debug Fastly and your custom VCL then it’s recommended you create a ‘reduced test case’ using their Fastly Fiddle tool. Be aware this tool shares code publicly so don’t put secret codes or logic into it! Don’t forget to add return statements to the functions in the Fiddle UI, otherwise the default Fastly boilerplate VCL will be executed and that can cause confusion if your service typically doesn’t use it! e.g. httpbin origin was sending back a 500 and vcl_fetch was triggering a restart and I didn’t know why. I discovered I had to add return(deliver) in vcl_fetch to prevent Fastly boilerplate from executing!

Fastly Default Cached Status Codes

The CDN (Fastly) doesn’t cache all responses. It will not cache any responses with a status code in the 5xx range, and it will only cache a tiny subset of responses with a status code in the 4xx and 3xx range.

The status codes it will cache by default are:

200 OK
203 Non-Authoritative Information
300 Multiple Choices
301 Moved Permanently
302 Moved Temporarily
404 Not Found
410 Gone

ℹ️ NOTE

in VCL you can allow any response status code to be cached by executing set beresp.cacheable = true; within vcl_fetch (you can also change the status code if you like to look like it was a different code with set beresp.status = <new_status_code>;).

Fastly Request Flow Diagram

There are various request flow diagrams for Varnish (example) and generally they separate the request flow into two sections: request and backend.

So handling the request, looking up the hash key in the cache, getting a hit or miss, or opening a pipe to the origin are all considered part of the “request” section. Whereas fetching of the content is considered part of the “backend” section.

The purpose of the distinction is because Varnish likes to handle backend fetches asynchronously. This means Varnish can serve stale data while a new version of the cached object is being fetched. This means less request queuing when the backend is slow.

But the issue with these diagrams is that they’re not all the same. Changes between Varnish versions and also the difference in Fastly’s implementation make identifying the right request flow tricky.

Below is a diagram of Fastly’s VCL request flow (including its WAF and Clustering logic). This is a great reference for confirming how your VCL logic is expected to behave.

Fastly-Debug

Fastly will display extra information about a request/response flow within its HTTP response headers if the request was issued with a Fastly-Debug HTTP request header (set with a value of 1).

Some of the extra headers you’ll find in the response are:

Fastly-Debug-Digest
Fastly-Debug-Path
Fastly-Debug-TTL
Surrogate-Control
Surrogate-Key
X-Cache-Hits
X-Cache
X-Served-By (reports fetching node, see Clustering)

ℹ️ NOTE

the Surrogate-* response headers are typically set by an origin server and are otherwise stripped by Fastly.

Some of the following information will make reference to ‘delivery’ and ‘fetching’ nodes. It’s probably best you read ahead to the section on clustering to understand what these concepts mean in the scope of Fastly’s system design. Once you understand them, come back here and the next few sentences will make more sense.

The quick summary is this:

delivery node: the cache server your request is first routed to (and which sends the response back to you).
fetching node: the cache server that actually makes a request to your origin, before returning the origin response back to the delivery node.

When using Fastly-Debug:1 to inspect debug response headers, we might want to look at fastly-state, fastly-debug-path and fastly-debug-ttl. These would have values such as…

< fastly-state: HIT-STALE
< fastly-debug-path: (D cache-lhr6346-LHR 1563794040) (F cache-lhr6324-LHR 1563794019)
< fastly-debug-ttl: (H cache-lhr6346-LHR -10.999 31536000.000 20)

The fastly-debug-path suggests we delivered from the delivery node lhr6346, while we fetched from the fetching node lhr6324.

The fastly-debug-ttl header suggests we got a HIT (H) from the delivery node lhr6346, but this is a misleading header and one you need to be careful with.

ℹ️ NOTE

it may take a few requests to see numbers populating the Fastly-Debug-TTL, as the request needs to either land on the fetching node, or a delivery node where the content exists in temporary memory. If you see - it might be because you arrived at a delivery node that doesn’t have it in-memory.

Why this header is misleading is actually quite a hard thing to explain, and Fastly has discussed the reasoning for the confusion in at least a couple different talks I’ve seen (one being: https://vimeo.com/showcase/6623864/video/376921467 which I highly recommend btw).

In essence, the HIT state is recorded at the fetching node. When the response makes its way back to the delivery node, it will then set the fastly-debug-ttl. But this doesn’t mean that the cache HIT happened at the delivery node, only that the header was set there.

The reason this is a concern is that you don’t necessarily know if the request did indeed go to the fetching node or whether the stale content actually came from the delivery node’s in-memory cache. The only way to be sure is to check the fastly-state response header.

The fastly-state header is ultimately what I use to verify where something has happened. If I see a HIT (or HIT-STALE), then I know I got a cache HIT from the delivery node (e.g. myself or someone else has already requested the resource via this delivery node).

ℹ️ NOTE

a reported HIT can in some cases be because the first cache server your request was routed to was the primary node (again, see clustering for details of what a ‘primary’ node is in relation to a ‘fetching’ node).

If I see instead a HIT-CLUSTER (or HIT-STALE-CLUSTER) it means I’m the first person to reach this delivery node and request this resource, and so there was nothing cached and thus we went to the fetching node and got a cache HIT there.

Another confusing aspect to fastly-debug-ttl is that with regards to stale-while-revalidate you could end up seeing a - in the section where you might otherwise expect to see the grace period of the object (i.e. how long can it be served stale for while revalidating). This can occur when the origin server hasn’t sent back either an ETag header or Last-Modified header. Fastly still serves stale content if the stale-while-revalidate TTL is still valid but the output of the fastly-debug-ttl can be confusing and/or misleading.

Something else to note, while I write this August 2019 update is that the fastly-debug-ttl only every displays the ‘grace’ value when it comes to stale-if-error, meaning if you’re trying to check if you’re serving stale-while-revalidate by looking at the grace period you might get confused when you see the stale-if-error grace period (or worse a - value), this is because the fastly-debug-ttl header isn’t as granular as it should be. Fastly have indicated that they intend on making updates to this header in the future in order for the values to be much clearer.

Lastly, when dealing with shielding the fastly-debug-ttl can be misleading in another sense which is: imagine the fetching node got a MISS (so it fetched content from origin and returned it to the delivery node). The delivery node will cache the response from the fetching node (including the MISS reported by fastly-debug-ttl) and so even if another request reaches the name delivery node, it will report a MISS, HIT combination.

thanks Fastly, for this totally not confusing setup ^^

304 Not Modified

Although not specifically mentioned in the above diagram it’s worth noting that Fastly doesn’t execute vcl_fetch when it receives a 304 Not Modified from origin, but it will use any Cache-Control or Surrogate-Control values defined on that response to determine how long the stale object should now be kept in cache.

If no caching headers are sent with the 304 Not Modified response, then the stale object’s TTL is refreshed. This means its age is set back to zero and the original max-age TTL is enforced.

Ultimately this means if you were hoping to execute logic defined within vcl_fetch whenever a 304 Not Modified was returned (e.g. dynamically modify the stale/cached object’s TTL), then that isn’t possible.

UPDATE 2019.08.10

Fastly reached out to me to let me know that this diagram is now incorrect.

Specifically, the request flow for a hit-for-pass (see below for details) was:

RECV, HASH, HIT, PASS, DELIVER

Where vcl_hit would return(pass) once it had identified the cached object as being a hit-for-pass object.

It is now:

RECV, HASH, PASS, DELIVER

Where after the object is returned from vcl_hash’s lookup, it’s immediately identified as being a HFP (hit-for-pass) and thus triggers vcl_pass as the next state, and finally vcl_deliver.

What this ultimately means is there is some redundant code later on in this blog post where I make reference to serving stale content. Specifically, I mention that vcl_hit required some custom VCL for checking the cached object’s cacheable attribute for the purpose of identifying whether it’s a hit-for-pass object or not.

ℹ️ NOTE

this vcl_hit code logic is still part of the free Varnish software, but it has been made redundant by Fastly’s version.

Error Handling

In Varnish you can trigger an error using the error directive, like so:

error 900 "Not found";

ℹ️ NOTE

it’s common to use the made-up 9xx range for these error triggers (900, 901, 902 etc).

Once executed, Varnish will switch to the vcl_error state, where you can construct a synthetic error to be returned (or do some other action like set a header and restart the request).

if (obj.status == 900) {
  set obj.status = 500;
  set obj.http.Content-Type = "text/html";
  synthetic {"<h1>Hmmm. Something went wrong.</h1>"};
  return(deliver);
}

In the above example we construct a synthetic error response where the status code is a 500 Internal Server Error, we set the content-type to HTML and then we use the synthetic directive to manually construct some HTML to be the ‘body’ of our response. Finally we execute return(deliver) to jump over to the vcl_deliver state.

If you want to send a synthetic JSON response (maybe your Fastly service is fronting an API that returns JSON), then this is possible:

if (obj.status == 401) {
  set obj.http.Content-Type = "application/json";
  set obj.http.WWW-Authenticate = "Basic realm=Secured";

  synthetic "{" +
    "%22error%22: %22401 Unauthorized%22" +
  "}";

  return(deliver);
}

We discuss JSON generation in more detail later.

Unexpected State Change

Now, I wanted to talk briefly about error handling because there are situations where an error can occur, and it can cause Varnish to change to an unexpected state. I’ll give a real-life example of this…

We noticed that we were getting a raw 503 Backend Unavailable error from Varnish displayed to our customers. This is odd? We have VCL code in vcl_fetch (the state that you move to once the response from the origin has been received by Fastly/Varnish) which checks the response status code for a 5xx and handles the error there. Why didn’t that code run?

Well, it turns out that vcl_fetch is only executed if the backend/origin was considered ‘available’ (i.e. Fastly could make a request to it). In this scenario what happened was that our backend was available but there was a network issue with one of Fastly’s POPs which meant it was unable to route certain traffic, resulting in the backend appearing as ‘unavailable’.

So what happens in those scenarios? In this case Varnish won’t execute vcl_fetch because of course no request was ever made (how could Varnish make a request if it thinks the backend is unavailable), so instead Varnish jumps from vcl_miss (where the request to the backend would be initiated from) to vcl_error.

This means in order to handle that very specific error scenario, we’d need to have similar code for checking the status code (and trying to serve stale, see later in this article for more information on that) within vcl_error.

UPDATE 2019.11.07

Fastly’s Fiddle tool now shows a compiler error that suggests 8xx-9xx are codes used internally by Fastly and that we should use the 6xx range instead.

According to Fastly the 900 code is often a dangeous code to use in custom VCL. In 8xx territory, they have a special case attached to 801, which is used for redirects to TLS in the case of requests coming in on HTTP. So if you manually trigger an 801, weird stuff happens.

6xx and 7xx are clean. 600, 601 and 750 are by far the most popular codes used in customer configs apparently. In the standards range, we have a special case attached to 550, for some reason, lost in the mists of time, but otherwise errors in the standards range are interpreted as specified in the IETF spec

State Variables

Each Varnish ‘state’ has a set of built-in variables you can use.

Below is a list of available variables and which states they’re available to:

Based on Varnish 3.0 (which is the only explicit documentation I could find on this), although you can see in various request flow diagrams for different Varnish versions the variables listed next to each state. But this was the first explicit list I found. Fastly themselves recommend this Varnish reference but that doesn’t indicate which variables are read vs write.

UPDATE 2020.12.11

The Fastly ‘Developer Hub’ now has an official reference 🎉

developer.fastly.com/reference/vcl/variables/

Here’s a quick key for the various states:

R: recv
F: fetch
P: pass
M: miss
H: hit
E: error
D: deliver
I: pipe
#: hash

	R	F	P	M	H	E	D	I	#
`req.*`	R/W	R/W	R/W	R/W	R/W	R/W	R/W	R/W	R/W
`bereq.*`	R/W	R/W	R/W	R/W
`obj.hits`	R	R
`obj.ttl`	R/W	R/W
`obj.grace`	R/W
`obj.*`	R	R/W
`beresp.*`	R/W
`resp.*`	R/W	R/W

For the values assigned to each variable:
R/W is “Read and Write”,
and R is “Read”
and W is “Write”

It’s important to realise that the above matrix is based on Varnish and not Fastly’s version of Varnish. But there’s only one difference between them, which is the response object resp isn’t available within vcl_error when using Fastly.

When you’re dealing with vcl_recv you pretty much only ever interact with the req object. You generally will want to manipulate the incoming request before doing anything else.

ℹ️ NOTE

the only other reason for setting data on the req object is when you want to keep track of things (because, as we can see from the above table matrix, the req object is available to R/W from all available states).

Once a lookup in the cache is complete (i.e. vcl_hash) we’ll end up in either vcl_miss or vcl_hit. If you end up in vcl_hit, then generally you’ll look at and work with the obj object (this obj is what is pulled from the cache - so you’ll check properties such as obj.cacheable for dealing with things like ‘hit-for-pass’).

If you were to end up at vcl_miss instead, then you’ll probably want to manipulate the bereq object and not the req object because manipulating the req object doesn’t affect the request that will shortly be made to the origin. If you decide at this last moment you want to send an additional header to the origin, then you would set that header on the bereq and that would mean the request to origin would include that header.

ℹ️ NOTE

this is where understanding the various state variables can be useful, as you might want to modify the req object for the sake of ‘persisting’ a change to another state, where as bereq modification will only live for the lifetime of the vcl_miss subroutine.

Once a request is made, the content is copied into the beresp variable and made available within the vcl_fetch state. You would likely want to modify this object in order to change its ttl or cache headers because this is the last chance you have to do that before the content is stored in the cache.

Finally, the beresp object is copied into resp and that is what’s made available within the vcl_deliver state. This is the last chance you have for manipulating the response that the client will receive. Changes you make to this object doesn’t affect what was stored in the cache (because that time, vcl_fetch, has already passed).

Anonymous Objects

There are two scenarios in which a pass occurs. One is that you return(pass) from vcl_recv explicitly, and the other is that vcl_recv executes a return(lookup) (which is the default behaviour) and the lookup results in a hit-for-pass object.

In both cases, an anonymous object is created, and the next customer-accessible VCL hook that will run is vcl_pass. Because the object is anonymous, any changes you make to it in vcl_fetch are not persisted beyond that one client request.

The only difference between the behaviour of a return(pass) from vcl_recv and a return(pass) resulting from a hit-for-pass in the cache, is that req.digest will be set. Fastly’s internal varnish engineering team state that req.digest is not an identifier for the object but rather a property of the request, which has been set simply because the request went through the hash process.

An early return(pass) from vcl_recv doesn’t go through vcl_hash and so no hash (req.digest) is added to the anonymous object. If there is a hash (req.digest) available on the object inside of vcl_pass, it doesn’t mean you retain a reference to the cache object.

Forcing Request to Origin

While we’re discussing state variables, I recently (2020.03.04) discovered that the req object has a set of properties exposed that will enable you to force a request through to the origin while allowing the response to be cached.

These properties are:

req.hash_always_miss
req.hash_ignore_busy

You would need to set them to true in vcl_recv:

set req.hash_always_miss = true;
set req.hash_ignore_busy = true;

It’s important to realize that getting a request through to the origin is a different scenario to using return(pass) in either vcl_recv and vcl_fetch (as might have been your first thought).

In the case of vcl_recv a return(pass) would cause the request to not only bypass a cache lookup (thus going to origin to fetch the content), but also the primary/fetching node wouldn’t have executed vcl_fetch! The fetching of content would have happened on the delivery node and so another client (if they reached a different delivery node, might end up at the fetching node and find no cached content there).

In the case of vcl_fetch a return(pass) would mean the response isn’t cached. So you can see how both the vcl_recv and vcl_fetch states executing return(pass) isn’t the same thing as letting a request bypass the cache but still having the origin response be cached!

When using hash_always_miss we’re causing the cache lookup to think it got a miss (rather than ‘passing’ it altogether). So this means features such as request collapsing are still intact.

Where as the use of hash_ignore_busy disables request collapsing and so this will result in a ‘last cached wins’ scenario (e.g. if you have two requests now going simultaneously to origin, whichever responds first will be cached but then the one that responds last will overwrite the first cached response).

request collapsing in action ^^

Clustering

Now that we know there are state variables available, and we understand generally when and why we would use them, let’s now consider the problem of ‘clustering’. Because if you don’t understand Fastly’s clustering design, then you’ll end up in a situation where data you’re setting on these variables are being lost and you won’t know why.

To help make clustering easier to understand I’ve created a diagram (below) of some Varnish internal ‘states’. In this diagram are two sections ‘cluster edge’ and ‘cluster shield’. Both represent individual cache servers running Varnish, but for the sake of simplicity I’ve omitted most of the Varnish states from the ‘cluster shield’ node.

ℹ️ NOTE

this diagram does not cover all the states available to Fastly’s implementation of Varnish. There is also vcl_log which is executed after vcl_deliver. There is also vcl_error which can be triggered from multiple other state functions.

The directional lines drawn on the diagram represent various request flows you might see in a typical Varnish implementation (definitely in my case at any rate). Of course these aren’t the only request flows you might see; there are numerous ways for a request to flow through the Varnish state machine.

Before we jump into some request flow examples (and then further onto the discussion of clustering), let’s take a moment to define some terminology that will help us throughout the rest of this post…

Terminology

The terminology I will use from here on, will be as follows:

delivery node (i.e. the ‘cluster edge’)
fetching node (i.e. the ‘cluster shield’)
primary node (i.e. fetching node for a specific object)

To explain the terminology:

When a cache server within a Fastly POP receives a request for an object, that server is ‘acting’ as the delivery node (as it will be the server responsible for returning a response to the client).

If the delivery node has an empty cache, then the request won’t immediately reach the origin. Instead the hash key for the requested resource/object will be used to identify a “primary” node, which is the node that will always (†) ‘act’ as a fetching node (i.e. it’s the server that will be responsible for handling the request for that specific resource/object).

† well, almost always. Except when the delivery node that ends up being selected is the “primary” node. Then the primary node is no longer acting as a fetching node, as it’s forced to act as a delivery node instead. Requiring a “secondary” node to act as a backup for those scenarios.

In some cases you’ll notice a “fetching node” being refered to as a “cluster node”. this can help to understand some of Fastly’s APIs (e.g. is_cluster).

Delivery node request flow

Let’s now consider a simple example request flow using that diagram. For this example, a node has been selected within the POP and it will start to process the request (so the server node is ‘acting’ as the delivery node).

client issues a request (and delivery node identified).
request is received and vcl_recv is called.
imagine the vcl_recv logic triggers the return(pass) directive.
the state will change to vcl_pass (skipping cache lookup).
after vcl_pass has completed, the request flows through to origin.
the origin responds and the state changes to vcl_fetch.
modifications to the response object will be cached.
after vcl_fetch has completed, response content is stored in the cache.
the state changes to vcl_deliver (modifications here are sent to client).
response is sent back to the client.

That’s just one example route that could be taken, and I conveniently chose that example because it meant only requiring one cache server to serve the response to a client (the fetching node, and the concept of clustering, didn’t come into effect in that example).

Example without Clustering

What’s important to understand though is the fact that Fastly’s infrastructure does mean (in specific scenarios) a request could indeed be handled by two separate cache servers. The initial cache server (the delivery node), and then potentially a second cache server (the fetching node).

When that happens, it is referred to as “clustering” but to more easily understand clustering, let’s consider what happens if it didn’t exist…

Without clustering what happens is this: the cache server that gets the request will generate a ‘hash key’ based upon the requested URL host and path. That hash key is then used to identify the resource in the cache that resides on the server.

If the cache doesn’t contain the requested resource, then the cache server will allow the request to flow through to the origin server which will respond with the appropriate content and the cache will be populated with that content for the next time that resource is requested.

This all works, but here’s the problem…

A Fastly POP contains 64 individual caching servers. All of them have a unique cache. That means every single cache server would at some point have to go back to the origin for the same resource that was already cached at a different cache server. That’s not an efficient system design.

To solve that problem, Fastly came up with the concept of Clustering.

Clustering is the coordination of two nodes within a POP to fulfill a request. The following image (and explanation) should help to clarify why this design is much better.

With clustering, the hash key for a requested resource is used, firstly to lookup the content in the cache of the current server (e.g. the delivery node), but also to identify a specific node that will be used for fetching the content from origin (and is referred to as the “primary” node, or fetching node) for when the content cannot be found in the delivery node’s cache.

This design means that multiple requests to different delivery nodes can all go to the same “primary” cache node (e.g. the fetching node) to fetch the content from the origin if they themselves didn’t have the requested resource cached.

This way, if we had ten requests for the same resource, and each request ended up at a different delivery node (and each delivery node failed to locate the resource in their cache) then only one of those ten requests would need to go to the origin (via the fetching node).

When the fetching node gets the content from the origin, it sticks the response in its cache and the response is sent back to the delivery node which also caches the response but only in memory (not on disk!).

But how does the fetching node stop all ten requests from each delivery node from reaching the origin, I hear you ask? Well, this is what’s referred to as “request collapsing”. It’s the process of blocking in-flight requests (for the same resource) until at least one request for that resource has completed.

In most cases the cache server that initially receives the request will not be the “primary”. But not always. Sometimes the cache server that initially handles the incoming client request IS the primary (e.g. what was the fetching node)!

Why is that you ask? It’s because the server that the request is initially sent to is picked at random. So what happens in that scenario?

This is where Fastly introduce the concept of a “secondary” node. It exists much like the “primary” does, to act as a cache layer before a request reaches the origin and exists for those specific times where the primary node is forced to act as the delivery node (e.g. just because a primary node is acting as a delivery shouldn’t mean it goes directly to the origin; the “secondary” helps to prevent that).

ℹ️ NOTE

for the most part you don’t need to worry too much about primary and secondary nodes, and only really need to know about the general concept of clustering consisting of a delivery node and a fetching node.

Fetching node request flow

Now go back to the earlier ‘request flow’ diagram. Try to imagine a request reaching a delivery node, and the node having an empty cache.

You’d find the state changes would be vcl_recv, vcl_hash, vcl_miss (the red one, which is an internal fastly operation that checks to see if A. clustering is enabled, which it is by default and also B. is the current server a fetching node).

Also, because any node within the cluster can be selected as the delivery node, it means there could be an in-memory copy of the cached content existing there. In that scenario the request would yield a cache HIT and thus only that one cache server is handling the request (e.g. no need to then go to the primary/fetching node).

Why are there different states running on different nodes?

One other thing to be aware of is that when a request flows from a delivery node to a fetching node (because the content isn’t found in the cache on the delivery node) the request will have to flow back to the delivery node in order for the client to receive a response!

Think about it, if the request is proxied from a delivery node to a fetching node, then it isn’t possible for the fetching node to respond to the client because the caller wasn’t the client but the delivery node.

This is why for most standard request flows you’ll find that the vcl_recv, vcl_hash, vcl_pass, vcl_deliver states are all handled by the delivery node, while vcl_miss, vcl_hit and vcl_fetch are documented as being executed on the fetching node.

This is because when the delivery node’s cache lookup fails to find any cached content, the request is proxied to the fetching node where it will again attempt a cache lookup. From that point (let’s say no content was cached there either) the fetching node will execute a vcl_miss, then vcl_fetch and ultimately it’ll have to stop there and return the response to the delivery node so it can change to the vcl_deliver state in order to send the response to the client.

Now as far as Fastly’s documentation is concerned (as of 2019.11.08) they A.) don’t document the fact that vcl_hash runs on the delivery node, and B.) is incorrect with regards to vcl_pass, which they claim runs on the fetching node (which it doesn’t, for the most part, as it is designed to break clustering).

When I pointed Fastly to their documentation, their response was:

“if you pass in recv, you’re saying no to cache lookup, and no to request collapsing, so there’s no reason to cluster the request”

Which was then followed with…

“big sigh, ok, I’m filing an issue”

You definitely can end up in vcl_pass on a fetching node, but admittedly it happens less frequently because you would have to return(pass) from either vcl_hit or vcl_miss (and that’s not a typical state change flow for most people).

^^ whenever I learn something new about Fastly

Breaking clustering

If we’re on a delivery node and we have a state that executes a return(restart) directive, then we actually break ‘clustering’.

This means the request will go back to the delivery node and that node will handle the full request cycle. This is done for reasons of performance, such as finding stale content in the vcl_deliver state on the delivery node and wanting to serve that stale content †

† we’ll learn about serving stale later.

Now go back to the earlier ‘request flow’ diagram. Try to imagine a request reaching a delivery node, and the node having an empty cache, but also that clustering has been broken (due to a return(restart) being triggered in vcl_deliver on the delivery node).

In this case, clustering is enabled and we’re not on the fetching node. So the request flows from the delivery node to the fetching node where that server’s cache is checked and either the vcl_hit or vcl_miss states will be triggered. But ultimately the request will flow back to the delivery node, where we’ll reach its vcl_deliver state.

From there the vcl_deliver triggers a return(restart), which as we now know breaks clustering, and so we end up back in vcl_recv of the delivery node.

At this point the request will reach vcl_hash and we might find we again reach vcl_miss (the ‘internal’ red one), but this time clustering is broken so we now reach your own code’s vcl_miss of the delivery node and thus the delivery node will proxy the request onto the origin and then execute the vcl_fetch state (instead of those steps happening on the fetching node).

Avoid breaking clustering

So what if breaking clustering is something you want to avoid? Let me start by saying I personally have never found a reason to avoid breaking clustering when executing a return(restart) but that doesn’t mean good reasons don’t exist.

In order to avoid breaking the clustering process you’ll need to utilize the Fastly-Force-Shield: 1 request header. This header will re-enable clustering so that we again use multiple server nodes within a POP when executing the different VCL subroutine states.

Legacy terminology

Notice the naming of this header (Fastly-Force-Shield), it uses what’s considered by Fastly now to be legacy terminology.

To explain: Fastly has historically used the term “shield” to describe the fetching node, as per the concept we now refer to as “clustering”. But it wasn’t always called clustering.

Long ago clustering was referred to as “shielding”, but in more recent times Fastly designed an extension to clustering which became a new feature also called Shielding. Fastly changed the old “shielding” terminology to “clustering”, thus allowing the new Shielding feature to be more easily distinguished (even though the underlying concepts are closely related).

The problem of persisting state

OK, so two more really important things to be aware of at this point:

Data added to the req object cannot persist across boundaries (except for when initially moving from the edge to the cluster).
Data added to the req object can persist a restart, but not when they are added from the cluster environment.

For number 1. that means: req data you set in vcl_recv and vcl_hash will be available in states like vcl_pass and vcl_miss.

For number 2. that means: if you were in vcl_deliver and you set a value on req and then triggered a restart, the value would be available in vcl_recv. Yet, if you were in vcl_miss for example and you set req.http.X-Foo and let’s say in vcl_fetch you look at the response from the origin and see the origin sent you back a 5xx status, you might decide you want to restart the request and try again. But if you were expecting X-Foo to be set on the req object when the code in vcl_recv was re-executed, you’d be wrong. That’s because the header was set on the req object while it was in a state that is executed on a fetching node; and so the req data set there doesn’t persist a restart.

Summary: modifications on the fetching node don’t persist to the delivery node.

If you’re starting to think: “this makes things tricky”, you’d be right :-)

The earlier diagram visualizes the approach for how a request inside of a POP will reach a specific cache node (i.e. “clustering”) but it doesn’t cover how “shielding” works, which effectively is a nested clustering process.

Let’s dig into Shielding next…

Shielding

Shielding is a designated POP that a request will flow through before reaching your origin.

As mentioned earlier, clustering is the coordination of two nodes within a POP, and this ‘clustering’ happens within every POP. The shield POP is no different from any other POP in its fundamental behaviour.

ℹ️ NOTE

when people start talking about shielding and its “Shield POP”, you’ll usually find the terminology of “POP” changes to “Edge POP” as a way to help distinguish that there are two separate POPs involved in the discussion. I personally don’t do that. I just call an Edge POP a POP and when talking about shielding I’ll say “Shield POP”.

The purpose of shielding is to give extra protection to your origin, because multiple users will arrive at different POPs (due to their locality) and so a POP in the UK might not have a cached object, while a POP in the USA might have a cached version of the resource. To help prevent the UK user from making a request back to the origin, we can select a POP nearest the origin to act as a single point of access.

Now when the UK request comes through and there is no cached content within that POP, the request wont go to origin, it’ll go to the shield POP which will hopefully have the content cached (if not then the shield POP sends the request onto the origin).

But ultimately the content either already cached at the shield (or is about to be cached at the shield if it had no cache) will be bubbled back to the UK POP where the content will be cached there as well.

ℹ️ NOTE

there isn’t any real latency concern with using shielding because Fastly optimizes the network BGP routing between POPs. The only thing to ensure is that your shield POP is located next to your origin because Fastly can’t optimize the connection from the shield POP to the origin (it can only optimize traffic within its own network).

This also gives us extra nomenclature to distinguish POPs. Before we learnt about shielding, we just knew there were ‘POPs’ but now we know that with shielding enabled we have ‘edge’ POPs and a singular ‘shield’ POP for a particular Fastly service.

I’ll link here to a Fastly-Fiddle (created by a Fastly engineer) that demonstrates the clustering/shielding flow. If that fiddle no longer exists by the time you read this then I’ve made a copy of it in a gist. It’s interesting to see how the various APIs for identifying a server node come together.

Lastly we should be aware that if for some reason there is a network issue in the region of your selected ‘shield’ POP, then Fastly will first (from the ‘edge’ POP) check the health of the shield POP before forwarding the request there. If the shield is unhealthy (e.g. maybe there is a network outage in its region), then the request will be forwarded directly to your origin.

Caveats of Fastly’s Shielding

First thing to note is that if your service has a restart statement, then be warned that this doesn’t just disable clustering but also shielding. To re-enable clustering you can use Fastly-Force-Shield but there is no way to re-enable shielding unless you manually copy/paste the relevant logic from Fastly’s generated VCL.

Be careful with changes you make to a request as they could result in the lookup hash to change between the edge POP nodes and shield POP nodes (so it’s likely best you make changes to the bereq object in vcl_miss rather than the req object within vcl_recv). Also the shield POP will be using the same Domain/Host UI configuration and so if you change the Host header, then proxying the request from the edge POP to the shield POP would result in a breakage as the shield POP won’t recognize the Host of the incoming request and thus will not know which ‘service’ to direct the request onto.

Also, be aware that the “backend” will change when shielding is enabled. Traditionally (i.e. without shielding) you defined your backend with a specific value (e.g. an S3 bucket or a domain such as https://app.domain.com) and it would stay set to that value unless you yourself implemented custom vcl logic to change its value.

But with shielding enabled, the delivery node will dynamically change the backend to be a shield node value (as it’s effectively always going to pass through that node if there is no cached content found). Once on the delivery node within the shield POP, its “backend” value is set to whatever your actual origin is (e.g. an S3 bucket).

So, if you dynamically set your backend in VCL, be sure to A.) set it before the fastly recv macro is executed, and B.) be sure to only set it on the shield POP otherwise your request will go from the edge POP direct to your origin and not your shield POP.

ℹ️ NOTE

be careful with ‘shared code’ (e.g. VCL code you reuse across multiple services) because if you add a conditional such as if (req.backend.is_shield) { /* execute code on edge POP */ } then this is fine when executing this code on a service with shielding enabled, but it won’t work as intended on a service that doesn’t use shielding! Because a service without shielding will not have the backend set to a shield, so that conditional will fail to match.

It’s probably best to only modify your backends dynamically whilst your VCL is executing on the shield (e.g. if (!req.backend.is_shield), maybe abstract in a variable declare local var.shield_node BOOL;) and to also only restart a request in vcl_deliver when executing on a node within the shield POP.

You might also need to modify vcl_hash so that the generated hash is consistent with the edge POP’s delivery node if your shield POP nodes happen to modify the request! Remember that modifying either the host or the path will cause a different cache key to be generated and so modifying that in either the edge POP or the shield POP means modifying the relevant vcl_hash subroutine so the hashes are consistent between them.

sub vcl_hash {
  # we do this because we want the cache key to be identical at the edge and
  # at the shield. Because the shield rewrites req.url (but not the edge), we
  # need align vcl_hash by using the original Host and URL.
  set req.hash += req.http.X-Original-URL;
  set req.hash += req.http.X-Original-Host;

  #FASTLY hash

  return(hash);
}

ℹ️ NOTE

alternatively you could move rewriting of the URL to a state after the hash lookup, such as vcl_miss (e.g. modifying the bereq object).

Be careful with non-idempotent changes. For example, things like the Vary header being modified on the shield POP and then again on the edge POP, as this could result in the edge POP getting a poor HIT ratio due to the fact that the shield has appended a header and then that header is appended again as part of the edge POP execution.

Consideration needs to be given to the use of req.backend.is_shield when you get a 5xx (or any other uncacheable error code) back from the origin to the shield POP, because it means an ‘pass’ state will be triggered. When the response reaches the delivery node within the edge POP, the pass state will cause clustering to be disabled and so the origin will no longer be the shield POP (as it was at the start of the request flow).

Lastly, when enabling shielding, make sure to deploy your VCL code changes first before enabling shielding. This way you avoid a race condition whereby a shield has old VCL (i.e. no conditional checks for either Fastly-FF or req.backend.is_shield) and thus tries to do something that should only happen on the edge cache node.

Debugging Shielding

If you want to track extra information when using shielding, then use (in combination with either req.backend.is_origin or !req.backend.is_shield) the values from server.datacenter and server.hostname which can help you identify the POP as your shielding POP (remember there is only one POP that is designated as your shield, so this can come in handy).

ℹ️ NOTE

remember that, although a statistically small chance, the edge POP that is reached by a client request could be the shield POP so your mechanism for checking if something is a shield needs to account for that scenario.

Additionally, there is this Fastly-Fiddle which clarifies the req.backend.is_cluster API which actually is different to similarly named APIs such as req.backend.is_origin and req.backend.is_shield, so let’s dig into that quickly…

`is_cluster` vs `is_origin` vs `is_shield`

There are a few properties hanging off the req.backend object in VCL…

is_cluster: indicates when the request has come from a clustering node (e.g. fetching node).
is_origin: indicates if the request will be proxied to an origin server (e.g. your own backend application).
is_shield: indicates if the request will be proxied to a shield POP (which happens when shielding is enabled).

If you try to access is_origin from within the vcl_recv state subroutine, for example, it will be cause a compiler error. This is because that API is only available to fetching nodes (and specifically only states that would result in a request being proxied, meaning although vcl_hit runs on a fetching node, that state would not have access to is_origin).

So depending on what you’re trying to verify, it might be preferable to use the negated is_shield approach for checking if the request is going to be proxied to origin or a shield pop node.

Undocumented APIs

req.http.Fastly-FF
fastly_info.is_cluster_edge
fastly_info.is_cluster_shield
fastly.ff.visits_this_service

`req.http.Fastly-FF`

The first API we’ll look at is: req.http.Fastly-FF which indicates if a request has come from a Fastly server.

It’s worth mentioning that it’s not really safe to use req.http.Fastly-FF because it can be set by a client making the request, and so there is no guarantee of its accuracy.

Also, the use of req.http.Fastly-FF can become complicated if you have multiple Fastly services chained one after the other because it means Fastly-FF could be set by a Fastly service not owned by you (i.e. the reported value is misleading).

`fastly_info.is_cluster_edge` and `fastly_info.is_cluster_shield`

With regards to is_cluster there are also some additional undocumented APIs we can use:

fastly_info.is_cluster_edge: true if the current vcl_ state subroutine is running on a delivery node.
fastly_info.is_cluster_shield: true if the current vcl_ state subroutine is running on a fetching node.

It’s important to realize that is_cluster_edge will only ever report true from vcl_deliver, as (just like req.backend.is_cluster) we have to come from a clustering/fetching node first. The vcl_recv state can’t know if it’s going to go into clustering at that stage of the request hence it reports as false there.

With vcl_fetch it knows it has come from the delivery node and thus we’ve gone into ‘clustering’ mode, hence it can report is_cluster_shield as true.

So as a more fleshed out example, if we tried to log all three cluster APIs in all VCL subroutines (and imagine we have clustering enabled with Fastly, which is the default behaviour), then we would find the following results…

vcl_recv:
- req.backend.is_cluster: no
- fastly_info.is_cluster_edge: no
- fastly_info.is_cluster_shield: no
vcl_hash:
- req.backend.is_cluster: no
- fastly_info.is_cluster_edge: no
- fastly_info.is_cluster_shield: no
vcl_miss:
- req.backend.is_cluster: no
- fastly_info.is_cluster_edge: no
- fastly_info.is_cluster_shield: yes
vcl_fetch:
- req.backend.is_cluster: no
- fastly_info.is_cluster_edge: no
- fastly_info.is_cluster_shield: yes
vcl_deliver:
- req.backend.is_cluster: yes
- fastly_info.is_cluster_edge: yes
- fastly_info.is_cluster_shield: no

`fastly.ff.visits_this_service`

The last undocumented API we’ll look at will be fastly.ff.visits_this_service which indicates for each server node how many times it has seen the request currently being handled. This helps us to execute a piece of code only once (maybe authentication needs to happen at the edge only once).

Let’s see what this looks like in a clustering scenario like shown a moment ago…

vcl_recv:
- fastly.ff.visits_this_service: 0 (we’re on a delivery node and we’ve never seen this request before)
vcl_hash:
- fastly.ff.visits_this_service: 0 (we’re still on the same delivery node so the reported value is the same)
vcl_miss:
- fastly.ff.visits_this_service: 1 (we’ve jumped to the fetching node so we know it’s been seen once before somewhere)
vcl_fetch:
- fastly.ff.visits_this_service: 1 (we’re on the same fetching node so the value is reported the same)
vcl_deliver:
- fastly.ff.visits_this_service: 0 (we’re back onto the original delivery node so the value is reported as 0 again).

Even if you restart the request, the value of the variable will go back to zero on the delivery node. See this fiddle for an example.

ℹ️ NOTE

when you introduce shielding you’ll find that the value increases to 2 when we reach vcl_recv on the delivery node inside the shield POP.

The following code snippet demonstrates how to run code either on the edge POP or the shield POP whilst also protecting against the scenario where the code is copy/pasted into a service that doesn’t have shielding enabled (see also: Caveats of Fastly’s Shielding).

if (req.backend.is_shield || fastly.ff.visits_this_service < 2) {
  # edge POP
}

if (!req.backend.is_shield && fastly.ff.visits_this_service >= 2) {
  # shield POP
}

# one liner for variable assignment
if(req.backend.is_shield, "edge", if(fastly.ff.visits_this_service < 2, "edge", "shield"))

ℹ️ NOTE

be careful with shielding because if the shield POP got a 5xx from the origin then it’ll trigger an internal PASS state and so once we’re back at the delivery node inside the edge POP, if we try to check if the backend is the ‘shield’ using req.backend.is_shield then it won’t match because the origin for the edge POP will no longer be the shield (clustering has been disabled because of the ‘pass state’).

Let’s now consider a requirement for creating a breadcrumb trail of the various states a request moves through (we’ll do this by defining a X-VCL-Route HTTP header). Implementing this feature is great for debugging purposes as it’ll validate your understanding of how your requests are flowing through the varnish state machine.

First I’m going to show you the basic outline of the various vcl state subroutines and a set of function calls. Next I’ll show you the code for those function calls and talk through what they’re doing.

ℹ️ NOTE

the code examples will be simplified for sake of brevity and to highlight the calls related to the breadcrumb trail functionality.

In essence though, we’re using the understanding we have for the delivery node and fetching node boundaries and ensuring that while on the fetching node we track information in Varnish objects (e.g. things like req, obj, beresp etc) that are able to persist crossing those boundaries.

This means when going from vcl_fetch to vcl_deliver we can’t track information on the req object because it’ll be lost when we move over to vcl_deliver and so we track it in the beresp object that vcl_fetch has access to. We do this because we know that beresp is copied over to vcl_deliver and exposed inside of vcl_deliver as the resp object.

Similarly for vcl_error, we track information in its obj reference because when we move over to vcl_deliver the obj reference will be copied over as the resp object.

From vcl_deliver we’re then free to copy the tracked information from the X-VCL-Route header (stored in the resp object) into the req object (in case we need to restart the request and keep tracking information after a restart).

Remember that vcl_pass and vcl_miss both run on the fetching node and so again tracking information on the req object won’t persist when moving over to vcl_deliver. That’s why although we track information on the req object within those state subroutines, we in fact (once we’ve reached vcl_fetch) copy those tracked values into the beresp object available to vcl_fetch for the reasons we described earlier.

OK, enough talking, let’s see the code…

include "debug_info"

sub vcl_recv {
  call debug_info_recv;

  #FASTLY recv

  return(lookup);
}

sub vcl_hash {
  #FASTLY hash

  set req.hash += req.url;
  set req.hash += req.http.host;

  call debug_info_hash;

  return(hash);
}


sub vcl_miss {
  #FASTLY miss

  call debug_info_miss;

  return(fetch);
}

sub vcl_pass {
  #FASTLY pass

  call debug_info_pass;
}

sub vcl_fetch {
  #FASTLY fetch

  call debug_info_fetch;

  return(deliver);
}

sub vcl_error {
  #FASTLY error

  call debug_info_error;

  return(deliver);
}

sub vcl_deliver {
  call debug_info_deliver;

  ...other code...

  call debug_info_send;

  #FASTLY deliver

  return(deliver);
}

The thing to look out for (in the above code) are the function calls that start with debug_info_ (e.g. debug_info_recv, debug_info_hash …etc).

These functions are all defined within a VCL file called debug_info.vcl, which we import at the top of the example code (i.e. include "debug_info").

Once that file is imported we will have access to the various debug_info_ functions that the VCL state subroutines are calling.

The other thing you’ll notice is that we have two separate function calls within vcl_deliver. We have the standard debug_info_deliver (as described a moment ago), but we also have debug_info_send.

The debug_info_send isn’t strictly necessary (e.g. the code within it could be moved inside of debug_info_deliver) but I wanted to distinguish between functions that collected data and this function that was responsible for sending the collected data back within the response.

Let’s now take a look at that debug_info VCL and then we’ll explain what the code is doing…

# This file sets a X-VCL-Route response header.
#
# X-VCL-Route has a baseline format of:
#
# pop: <value>, node: <value>, state: <value>, host: <value>, path: <value>
#
# pop  = the POP where the request is currently passing through.
# node  = whether the cache node is a delivery node or fetching node
# state = internal fastly variable that reports state flow (req waited for collapsing or clustered).
# host  = the full host name, without the path or query parameters.
# path   = the full path, including query parameters.
#
# Additional to this baseline we include information relevant to the subroutine state.

sub debug_info_recv {
  declare local var.context STRING;
  set var.context = "";

  if (req.restarts > 0) {
    set var.context = req.http.X-VCL-Route + ", ";
  }

  set req.http.X-VCL-Route = var.context + "VCL_RECV(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_edge, " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url +
    ")";
}

sub debug_info_hash {
  set req.http.X-VCL-Route = req.http.X-VCL-Route + ", VCL_HASH(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_edge, " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url +
    ")";
}

sub debug_info_miss {
  set req.http.X-PreFetch-Miss = ", VCL_MISS(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    " [" server.datacenter ", " server.hostname "], " +
    "node: cluster_" + if(fastly_info.is_cluster_shield, "shield", "edge") + ", " +
    "state: " + fastly_info.state + ", " +
    "host: " + bereq.http.host + ", " +
    "path: " + bereq.url +
    ")";
}

sub debug_info_pass {
  set req.http.X-PreFetch-Pass = ", " if(fastly_info.state ~ "^HITPASS", "VCL_HIT", "VCL_PASS") "(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_" + if(fastly_info.is_cluster_shield, "shield", "edge") + ", " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url + ", " +
    ")";
}

sub debug_info_fetch {
  set beresp.http.X-Track-VCL-Route = req.http.X-VCL-Route;
  set beresp.http.X-PreFetch-Pass = req.http.X-PreFetch-Pass;
  set beresp.http.X-PreFetch-Miss = req.http.X-PreFetch-Miss;
  set beresp.http.X-PostFetch = ", VCL_FETCH(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_" + if(fastly_info.is_cluster_shield, "shield", "edge") + ", " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url + ", " +
    "status: " + beresp.status + ", " +
    "stale: " + if(stale.exists, "exists", "none") + ", " +
    if(beresp.http.Cache-Control ~ "private", "cache_control: private, return: pass", "return: deliver") +
    ")";
}

sub debug_info_error {
  declare local var.error_page BOOL;
  set var.error_page = false;

  set obj.http.X-VCL-Route = req.http.X-VCL-Route + ", VCL_ERROR(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_" + if(fastly_info.is_cluster_shield, "shield", "edge") + ", " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url + ", " +
    "status: " + obj.status + ", " +
    "stale: " + if(stale.exists, "exists", "none") + ", " +
    ")";
}

sub debug_info_deliver {
  # only track the previous route flow if we've come from vcl_fetch
  # otherwise we'll end up displaying the uncached request flow as
  # part of this cache hit request flow (which would be confusing).
  if (resp.http.X-Track-VCL-Route && fastly_info.state ~ "^(MISS|PASS)") {
    set req.http.X-VCL-Route = resp.http.X-Track-VCL-Route;

    if (resp.http.X-PreFetch-Pass) {
      set req.http.X-VCL-Route = req.http.X-VCL-Route + resp.http.X-PreFetch-Pass;
    }

    if (resp.http.X-PreFetch-Miss) {
      set req.http.X-VCL-Route = req.http.X-VCL-Route + resp.http.X-PreFetch-Miss;
    }

    if (resp.http.X-PostFetch) {
      set req.http.X-VCL-Route = req.http.X-VCL-Route + resp.http.X-PostFetch;
    }
  } elseif (fastly_info.state ~ "^ERROR") {
    # otherwise track in the request object any request flow information that 
    # has occurred from an error request flow 
    # which should include either the original vcl_fetch flow or the vcl_hit flow.
    set req.http.X-VCL-Route = resp.http.X-VCL-Route;
  } elseif (fastly_info.state ~ "^HIT($|-)") {
    # otherwise track the initial vcl_hit request flow.
    set req.http.X-VCL-Route = req.http.X-VCL-Route + ", VCL_HIT(" +
      "pop: " + if(req.backend.is_shield, "edge", 
      if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
      + " [" + server.datacenter + ", " + server.hostname + "], " +
      "node: cluster_shield, " +
      "state: " + fastly_info.state + ", " +
      "host: " + req.http.host + ", " +
      "path: " + req.url + ", " +
      "status: " + resp.status + ", " +
      "cacheable: true, " +
      "return: deliver" +
      ")";
  }

  # used to extend the baseline X-VCL-Route (set below)
  declare local var.context STRING;
  set var.context = "";

  # there is one state subroutine that we have no way of tracking information
  # for: vcl_hit # this is because the only object we have available with R/W access
  # is the req object # and modifications to the req object don't persiste from
  # cluster node to edge node (e.g. vcl_deliver) # this means we need to utilise
  # fastly's internal state to see if we came from vcl_hit.
  #
  # we also use this internal state variable to help identify other states 
  # progressions such as
  # STALE (stale content found in case of an error) and ERROR 
  # (we've arrived to vcl_deliver from vcl_error).
  #
  # Documentation (fastly_info.state):
  # https://support.fastly.com/hc/en-us/community/posts/360040168391/comments/360004718351
  if (fastly_info.state ~ "^HITPASS") {
    set var.context = ", cacheable: uncacheable, return: pass";
  } elseif (fastly_info.state ~ "^ERROR") {
    set var.context = ", custom_error_page: " + resp.http.CustomErrorPage;
  }

  set req.http.X-VCL-Route = req.http.X-VCL-Route + ", VCL_DELIVER(" +
    "pop: " + if(req.backend.is_shield, "edge", 
    if(fastly.ff.visits_this_service < 2, "edge", "shield")) 
    + " [" + server.datacenter + ", " + server.hostname + "], " +
    "node: cluster_edge, " +
    "state: " + fastly_info.state + ", " +
    "host: " + req.http.host + ", " +
    "path: " + req.url + ", " +
    "status: " + resp.status + ", " +
    "stale: " + if(stale.exists, "exists", "none") +
    var.context +
    ")";
}

# this subroutine must be placed BEFORE the vcl_deliver macro `#FASTLY deliver`
# otherwise the setting of Fastly-Debug within this subroutine will have no effect.
sub debug_info_send {
  unset resp.http.X-Track-VCL-Route;
  unset resp.http.X-VCL-Route;
  unset resp.http.X-PreFetch-Miss;
  unset resp.http.X-PreFetch-Pass;
  unset resp.http.X-PostFetch;

  if (req.http.X-Debug) {
    # ensure that when Fastly's own VCL executes it will be able to identify
    # the Fastly-Debug request header as enabled.
    set req.http.Fastly-Debug = "true";

    # other useful debug information
    set resp.http.Fastly-State = fastly_info.state;
    set resp.http.X-VCL-Route = req.http.X-VCL-Route;
  }
}

So I’ve already summarized the basic approach (i.e. ensure we set tracking information in Varnish objects that we know persist the boundary changes), but there are some interesting bits to take away from this code still…

Within debug_info_recv, because a request can be restarted, the breadcrumb trail may need to repeat the VCL states. So for example if a request was restarted we would want to see VCL_RECV(...) twice in the output to indicate that this was the flow of the request. To account for that I use a var.context variable to prepend the information from before the request to the new X-VCL-Route header being created now the request has restarted.

Next you’ll notice that we have a strange expression which we’re assigning to a pop field. This field represents whether the request is being executed at an ‘edge’ POP or a ‘shield’ POP (in the case that the Fastly service has shielding enabled)…

if(req.backend.is_shield, "edge", 
if(fastly.ff.visits_this_service < 2, "edge", "shield"))

We use the req.backend.is_shield API to tell us if the server is going to send the request to a shield POP or not, but unfortunately it’s not as straight forward as just saying “ok is_shield is false so we must be on a node within the shield POP already”.

The reason being, if the Fastly service doesn’t have shielding enabled then req.backend.is_shield will always return false! So that’s why we have that nested conditional check…

if(fastly.ff.visits_this_service < 2, "edge", "shield")

Instead of just printing shield when req.backend.is_shield is false we will now first check fastly.ff.visits_this_service (which we discussed earlier). This says “ok, if the returned value is less than two we know that we must be running within an edge POP, because if shielding was enabled and we were on a shield POP currently then the number of ‘views’ would be three or more at this point”.

Although I don’t demonstrate it here in this example code, debug_info_hash might need an extra conditional header check added to it to prevent tracking the vcl_hash execution. The reason you might want to do that is because vcl_hash is always executed (even though it can in some scenarios be a no-op!).

For example, in my own VCL code at work we have a process whereby we’ll restart a request if we have an error come back from the origin. The reason we restart the request is so we can serve a custom error page synthetically from the edge server. In order to do that we set a header in vcl_deliver before we restart (e.g. X-Error) and then we check it from within vcl_recv only if the req.restarts == 1. From that point in vcl_recv if X-Error is set we’ll trigger vcl_error by calling the error directive, and within vcl_error we’ll construct our synthetic custom error page.

Now the problem we have is that in the final X-VCL-Route sent back in the response we’ll see vcl_hash in the flow just after the second vcl_recv (which happens because of the restart in order to handle the custom error page logic), and that just …looks weird because it suggests to a viewer that vcl_hash had some sort of purpose for running, but it doesn’t!

So to avoid recording the vcl_hash execution I wrap the tracking code in a check for the X-Error header (e.g. if (!req.http.X-BF-Error) { track the execution }).

In debug_info_fetch you’ll see that I don’t assign to a X-VCL-Route header on the beresp object but a new X-Track-VCL-Route header. The reason for this is because the beresp object is going to be placed into the Varnish cache system once vcl_fetch has finished executing and we need to be careful about setting X-VCL-Route there.

The reason for the concern is due to new client requests. Imagine we set the tracking information onto the actual X-VCL-Route header on the beresp (which will be cached).

When a new request is received for that same resource we’ll go to vcl_hit because the resource is found in the cache and from there we’ll end up at vcl_deliver which would normally just check for a X-VCL-Route on the resp object it has been passed.

But in this scenario the resp object is the object pulled from the Varnish cache and so we might end up including the uncached request flow in the final X-VCL-Route header sent back for this secondary request!

That would be wrong because the secondary request should only report a recv > hash > hit > deliver and not recv > hash > miss > fetch > recv > hash > hit > deliver (which is what would happen otherwise!). See how easy it is for things to get dangerous and confusing!

With that in mind if we look at debug_info_deliver we’ll see that we only track the ‘uncached’ request flow if we can tell that we actually came from a cache MISS state…

if (resp.http.X-Track-VCL-Route && fastly_info.state ~ "^MISS") 
{ track the data that was stored in X-Track VCL-Route }

If you want more information on fastly_info.state see this community comment.

You’ll also see in that same conditional block we then reconstruct a X-VCL-Route from the various temporary headers, such as X-PreFetch-Pass, X-PreFetch-Miss and X-PostFetch.

Remember that in vcl_hit we have no object that we can track information on that will be persisted to the delivery node where vcl_deliver is executed. This is because vcl_hit can go straight to vcl_deliver and there’s no inbetween state like vcl_fetch where we can borrow a Varnish object for tracking purposes like we do with vcl_miss and vcl_pass.

So to solve that problem you’ll see that we check to see if we came from a cache HIT state…

fastly_info.state ~ "^HIT($|-)"

ℹ️ NOTE

this will catch multiple types of hit flows, such as HIT or HIT-CLUSTER or HIT-STALE-CLUSTER.

Finally, in debug_info_send you’ll see that we make sure to unset any temporary response headers (e.g. X-Track-VCL-Route) so it doesn’t confuse people scanning the response headers we send back. We also don’t send this debug information back to the client unless they ask for it, which they can do by providing a req.http.X-Debug request header.

One interesting thing to note there is that in order to get Fastly to send back its own debug information the user needs to provide a Fastly-Debug request header, but to avoid having to rely on that knowledge we provide our own X-Debug (which we tell our engineers about) so they only have to remember one simpler request header to enable not only our own debugging information but Fastly’s as well.

In order then for the Fastly debug information to be included we have to do two things:

manually set req.http.Fastly-Debug = "true"; in our VCL code.
ensure our debug_info_send is executed before the #FASTLY deliver macro (which is where they execute their Fastly-Debug logic).

OK, let’s go back and revisit vcl_error…

The vcl_error subroutine is a tricky one because it exists on both the delivery node and the fetching node.

Meaning if you execute error 401 from vcl_recv, then vcl_error will execute in the context of the delivery node; whereas if you executed an error from a fetching node state like vcl_fetch, then vcl_error would execute in the context of the fetching node.

Meaning, how you transition information between vcl_error and vcl_deliver could depend on whether you’re on a delivery node or a fetching node.

This is why when on what are typically considered the fetching nodes, I don’t hardcode them a such. I use the following conditional check…

"node: cluster_" + if(fastly_info.is_cluster_shield, "shield", "edge") + ", " +

…this check means if the request is restarted, which we know this causes Fastly’s clustering behaviour to stop and for the request to flow through the same delivery node for all states, then we can change the reported node to be the delivery node instead of a fetching node for states such as MISS/PASS/FETCH.

ℹ️ NOTE

this conditional check could be improved by also checking for the existence of the Fastly-Force-Shield header (which we talked about earlier). This is because if someone uses that header then it would force Fastly’s clustering behaviour to not be stopped.

To help explain this I’m going to give another real example, where I wanted to lookup some content in our cache and if it didn’t exist I wanted to restart the request and use a different origin server to serve the content.

To do this I expected the route to go from vcl_recv, to vcl_hash and the lookup to fail so we would end up in vcl_miss. Now from vcl_miss I could have triggered a restart, but anything I set on the req object at that point (such as any breadcrumb data appended to X-VCL-Route) would have been lost as we transitioned from the fetching node back to the delivery node (where vcl_recv is).

I needed a way to persist the “miss” breadcrumb, so instead of returning a restart from vcl_miss I would trigger a custom error such as error 901 and inside of vcl_error I would have the following logic:

set obj.http.X-VCL-Route = req.http.X-VCL-Route;

if (fastly_info.state ~ "^MISS") {
  set obj.http.X-VCL-Route = obj.http.X-VCL-Route ",VCL_MISS(" req.http.host req.url ")";
}

set obj.http.X-VCL-Route = obj.http.X-VCL-Route ",VCL_ERROR";

if (obj.status == 901) {
  set obj.http.X-VCL-Route = obj.http.X-VCL-Route ",VCL_ERROR(status: 908, return: deliver)";

  return(deliver);
}

When we trigger an error an object is created for us. On that object I set our X-VCL-Route and assign it whatever was inside req.http.X-VCL-Route at that time †

† which would include vcl_recv, vcl_hash and vcl_miss. Remember the req object does persist across the delivery node/fetching node boundaries, but only when going from vcl_recv. After that, anything set on req is lost when crossing boundaries.

Now we could have arrived at vcl_error from vcl_recv (e.g. if in our vcl_recv we had logic for checking Basic Authentication and none was found on the incoming request we could decide from vcl_recv to execute error 401) or we could have arrived at vcl_error from vcl_miss (as per our earlier example). So we need to check the internal Fastly state to identify this, hence checking fastly_info.state ~ "^MISS".

After that we append to the obj object’s X-VCL-Route header our current state (i.e. so we know we came into vcl_error). Finally we look at the status on the obj object and see it’s a 901 custom status code and so we append that information in order to know what happened.

But you’ll notice we don’t restart the request from vcl_error, because if we did come from vcl_miss the data in obj would be lost because ultimately it was set on a fetching node (as vcl_error would be running on the fetching node when coming from vcl_miss).

Instead we return(deliver), because all that data assigned to obj is guaranteed to be copied into resp for us to reference when transitioning to vcl_deliver on the delivery node.

Once we’re at vcl_deliver we continue to set breadcrumb tracking onto req.http.X-VCL-Route as we know that will persist a restart (as we’re still going to be executing code on the delivery node).

Phew! Well, that was NOT an easy task, and if you struggled to follow along. That’s OK because it’s hard to learn about all this stuff at once. Just keep coming back to this post as a reference point if you need to.

Thankfully by going through this process there will be very little about Varnish’s request flow that you won’t now understand or have the ability to work around in future if the right problem scenario presents itself.

Header Overflow Errors

One final thing to note about building a ‘breadcrumb trail’ like I’ve detailed above is that you’ll need to be careful about how many times you restart your request flow.

I’ve seen us restart our request up to three times (for multiple-backend failover resiliency) and it has resulted in a raw Varnish error of Header overflow to be returned in the response (the client won’t even record any network data at all, it’s as if the client compleletely flatlines).

Googling around I discovered some Fastly documentation which makes reference to a Header count…

Exceeding the limit results in a Header overflow error. A small portion of this limit is reserved for internal Fastly use, making the practical limit closer to 85.

To me this was quite an ambiguous statement and have the value set to 96 didn’t help elucidate the situation at all either. I had no idea what this really meant.

Was Fastly saying the overall request header size of X-VCL-Route was too large (hence a 400 Bad Request would be the typical response expected for that type of error) or did it mean that I could only set that specific header 96 times and I was reaching that limit due to the number of restarts (surely not?)

Even if I restarted the request flow three times, there’s only a total of eight VCL states and they don’t all get executed in a single request flow, so that’s only a maximum of 24 times I would have called set on the X-VCL-Route header (way under the 96 limit).

Turns out it’s a bit more a broad range of possible problem causes than just that.

The official Fastly response to my question about what this documentation means was as follows…

header count = 96 specifically means 96 headers can be present for an object (this includes the request and response headers). The header overflow can be caused by exceeding any of the limits defined in the documentation throughout the whole VCL process.

So it’s more likely that we were hitting the http header size limit, but it was being transformed into this raw Varnish error instead.

At any rate this is something to keep in mind and be mindful of.

Hit for Pass

You may notice in Varnish’s built-in vcl_fetch the following logic:

sub vcl_fetch {
    if (!beresp.cacheable) {
        return (pass);
    }
    if (beresp.http.Set-Cookie) {
        return (pass);
    }
    return (deliver);
}

Now typically when you return(pass) you do that in vcl_recv to indicate to Varnish you do not want to lookup the content in the cache and to skip ahead to fetching the content from the origin. But when you return(pass) from vcl_fetch it causes a slightly different behaviour. Effectively we’re telling Varnish we don’t want to cache the content we’ve received from the origin.

In the case of the VCL logic above, we’re not caching this content because we can see there is a cookie set (indicating possibly unique user content).

You’ll probably also notice in some organisation’s own custom vcl_fetch the following additional logic:

if (beresp.http.Cache-Control ~ "private") {
  return(pass);
}

This content isn’t cached simply because the backend has indicated (via the Cache-Control header) that the content is private and so should not be cached.

But you’ll find that even though you’ve executed a return(pass) operation, Varnish will still create an object and cache it.

The object it creates is called a “hit-for-pass” (if you look back at the Fastly request flow diagram above you’ll see it referenced) and it is given a ttl of 120s (i.e. it’ll be cached for 120 seconds).

ℹ️ NOTE

the ttl can be changed using vcl but it should be kept small. Varnish implements a type known as a ‘duration’ and takes many forms: ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), y (years). For example, beresp.ttl = 1h.

The reason Varnish creates an object and caches it is because if it didn’t, when return(pass) is executed and the content subsequently is not cached, then if another request is made for that same resource, we would find “request collapsing” causes a performance issue for users. What is ‘request collapsing’ I hear you ask? Well…

Request Collapsing

Request collapsing is where Varnish blocks requests for what looks to be the same uncached resource. It does this in order to prevent overloading your origin. So for example, if there are ten requests for an uncached resource, it’ll allow one request through to origin and block the other nine until the origin has responded and the content has been cached. The nine requests would then get the content from the cache.

But in the case of uncachable content (e.g. content that uses cookies typically will contain content that is unique to the user requesting the resource) users are blocked waiting for an existing request for that resource to complete, only to find that as the resource is uncacheable the request needs to be made again (this cycle would repeat for each user requesting this unique/uncacheable resource).

i.e. out of 10 requests for a resource that is ultimately uncacheable, the first request would go through while nine are blocked waiting for the first request to complete. The first request completes and oh-no, it can’t be cached, nevermind let’s unblock the second request (while the other eight are blocked). When the second request comes back we again realise it can’t be cached and the cycle repeats until all ten requests have completed.

As you can imagine, this is very bad because the requests for this uncachable content has resulted in sequential processing.

So when we “pass” inside of vcl_fetch Varnish prevents this bad sequential processing. It does this by creating a “hit-for-pass” object which has a short ttl of 120s, and so for the next 120s any requests for this same resource will not result in request collapsing (i.e. user requests to origin will not be blocked waiting for an already “in-flight” origin request to complete). Meaning, all requests will be sent straight through to the origin.

In order for this processing to work, we need vcl_hit to check for a “hit-for-pass” object. If we check Varnish’s built-in logic we can see:

sub vcl_hit {
    if (!obj.cacheable) {
        return (pass);
    }
    return (deliver);
}

What this does is it checks whether the object we found in the cache has the attribute cacheable set to ‘false’, and if it does we’ll not deliver that cached object to the user but instead skip ahead to fetching the resource again from origin.

By default this cacheable attribute is set to ‘true’, but when Varnish executes return(pass) from inside of vcl_fetch it caches the “hit-for-pass” object with the cacheable attribute set to ‘false’.

The reason the ttl for a “hit-for-pass” object is supposed to be short is because, for that period of time, your origin is susceptible to multiple requests. So you don’t want your origin to become overloaded by lots of traffic for uncacheable content.

It’s important to note that an object can’t be re-cached (let’s say the origin no longer sends a private directive) until either the hit-for-pass TTL expires or the hit-for-pass object is purged.

See this Varnish blog post for the full details.

Serving Stale

If we get a 5xx error from our origins we don’t cache them.

But instead of serving that 5xx to the user (or even a custom 500 error page), we’ll attempt to locate a ‘stale’ version of the content and serve that to the user instead (i.e. ‘stale’ in this case means a resource that was requested and cached previously, but the object was marked as being something that could be served stale if its ‘stale ttl’ has yet to expire).

The reason we do this is because serving old (i.e. stale) content is better than serving an error.

stale content is always better, right! ^^

In order to serve stale we need to add some conditional checks into our VCL logic.

You’ll typically notice in both vcl_fetch and vcl_deliver there are checks for a 5xx status code in the response we got back from origin, and subsequently a further check for stale.exists if we found a match for a 5xx status code. It looks something like the following:

ℹ️ NOTE

you don’t have to run this code only from vcl_deliver. It can be beneficial to do this via vcl_error as well because if your origin is unreachable, then it means you’ll want to check for stale in vcl_error as well. Fastly gives an example of this in their documentation.

if (<object>.status >= 500 && <object>.status < 600) {
  if (stale.exists) {
    # ...
  }
}

Where <object> is either beresp (vcl_fetch) or resp (vcl_deliver).

When looking at the value for stale.exists, if it returns ‘true’, it is telling us that there is an object in the cache whose ttl has expired but as far as Varnish is concerned is still valid for serving as stale content. The way Varnish knows whether to keep a stale object around so it can be used for serving as stale content depends on the following settings:

set beresp.stale_while_revalidate = <N>s;
set beresp.stale_if_error = <N>s;

Where <N> is the amount of time in seconds you want to keep the object for after its ttl has expired.

stale_while_revalidate: request received, obj found in cache, but ttl has expired (results in cache MISS), so we serve stale while we go to origin and request the content (†).
stale_if_error: request received, obj found in cache, but ttl has expired, so we go to origin and the origin returns an error, so we serve stale.

† if successful, new content is cached and the TTL is updated to whatever the cache response headers dictate.

Ultimately this means that stale_if_error will only ever be initialized if stale_while_revalidate fails. Which begs the question…

Why use stale_if_error at all when we could just set stale_while_revalidate to a large value, and it would effectively result in the same outcome: stale content served to the user for a set period of time?

This is a question I don’t have an answer for, and I’ve yet to receive a satisfactory answer to from either Fastly or the dev community.

My own opinion on this is that if my origin is unable to serve a 200 OK response after 1hr of trying via stale_while_revalidate then something must be seriously wrong with my origin and so maybe that’s the appropriate indicator for what value I should give to it in comparison to the value I would set for stale_if_error (which would be the longest value possible).

It feels like setting both stale_while_revalidate and stale_if_error is kind of like ‘cargo culting’ (i.e. doing something because it’s always been done and so people presume it’s the correct way), but at the same time I would hate to find myself in a situation where there was a subtle difference between the two which had a very niche scenario that broke my user’s experience because I neglected to set both stale configurations.

Additionally, if these ‘stale’ settings aren’t configured in VCL, then you’ll need to provide them as part of Fastly’s Surrogate-Control header (see here for the details, but be aware that VCL configured values will take precedence over response headers from origin):

"Surrogate-Control": "max-age=123, stale-while-revalidate=172800, stale-if-error=172800"

The use of beresp.stale_if_error is effectively the same as Varnish’s beresp.grace. But be careful if your VCL already utilises Varnish’s original grace feature, because it will override any Fastly behaviour that is using the stale_if_error.

You can find more details on Fastly’s implementation here as well as a blog post announcing this feature here. If you want details on Varnish 4.0’s implementation of serving stale, see this post.

Why you might not be serving stale?

You might find that you’re not serving stale even though you would expect to be. This can be caused by a lack of shielding (an additional Fastly feature that’s designed to improve your hit ratio) as you can only serve a stale cached object if the request ended up being routed through the POP that has the content cached (which is more difficult without shielding enabled).

Another reason would be to use “soft purging” rather than hard purges.

Lastly, there’s one quirk of Fastly’s caching implementation you might need to know about: if you specify a max-age of less than 61 minutes, then your content will only be persisted into memory (and there are many situations where a cached object in memory can be removed). To make sure the object is persisted (i.e. cached) on disk and thus available for a longer period of time for serving stale, you must set a max-age above 61 minutes.

It’s worth reading fastly’s notes on why you might not be serving stale which additionally includes notes on things like the fact that their implementation is an LRU (Least Recently Used) cache, and so even if a cached object’s TTL has not expired it might be so infrequently requested/accessed that it’ll be evicted from the cache any way! The LRU affects both fresh (i.e. non-expired) objects and ‘stale’ objects (i.e. TTL has expired but they have stale config values that haven’t expired).

Stale for Client Devices

It’s worth reiterating a segment of the earlier caching priority list which is that Cache-Control can include serving stale directives such as stale-while-revalidate and stale-if-error, but they are typically utilized with Surrogate-Control more than they are with Cache-Control. If Fastly receives no Surrogate-Control but it does get Cache-Control with those directives it will presume those are defined for its benefit.

Client devices (e.g. web browsers) can respect those stale directives, but it’s not very well supported currently (see MDN compatibility table).

Different actions for different states

So if we find a stale object, we need to deliver it to the user. But the action you take (as far as Fastly’s implementation of Varnish is concerned) depends on which state Varnish currently is in (vcl_fetch or vcl_deliver).

When you’re in vcl_fetch you’ll return(deliver_stale) and in vcl_deliver you’ll return(restart).

Why the difference?

The reason for this difference is to do with Fastly’s Varnish implementation and how they handle ‘clustering’.

According to Fastly you’ll typically find, due to the way their ‘clustering’ works, that vcl_fetch and vcl_deliver generally run on different servers (although that’s not always the case, as we’ll soon see) and so different servers will have different caches.

“What we refer to as ‘clustering’ is when two servers within the same POP communicate with each other for cache optimization purposes” – Fastly

The happy path (stale found in `vcl_fetch`)

If a stale object is found in vcl_fetch, then deliver_stale will send the found stale content to vcl_deliver with a status of 200. This means when vcl_deliver checks the status code it’ll not see a 5xx and so it’ll just deliver that stale content to the user.

The longer path (stale found in `vcl_deliver`)

Imagine vcl_fetch is running on server A and vcl_deliver is running on server B (due to Fastly’s ‘clustering’ infrastructure) and you looked for a stale object in vcl_fetch. The stale object might not exist there and so you would end up passing whatever the origin’s 5xx response was onto vcl_deliver running on server B.

Now in vcl_deliver we check stale.exists and it might tell us that an object was found, remember: this is a different server with a different cache.

In this scenario we have a 5xx object that we’re about to deliver to the client, but on this particular server (B) we’ve since discovered there is actually a stale object we can serve to the user instead. So how do we now give the user this stale object and not the 5xx content that came from origin?

In order to do that, we need to restart the request cycle. When we return(restart) in vcl_deliver it forces Fastly’s version of Varnish to break its clustering behaviour and to now route the request completely through whatever server vcl_deliver is currently running on (in this example it’ll ensure the entire request flow will go through server B - which we know has a stale object available).

This means we end up processing the request again, but this time when we make a request to origin and get a 5xx back, we will (in vcl_fetch) end up finding the stale object (remember: our earlier restart has forced clustering behaviour to be broken so we’re routing through the same server B). Now we find stale.exists has matched in vcl_fetch we will return(deliver_stale) (which we failed to do previously due to the stale object not existing on server A) and this means the stale content with a status of 200 is passed to vcl_deliver and that will subsequently deliver the stale object to the client.

The unhappy path (stale not found anywhere)

In this scenario we don’t find a stale object in either vcl_fetch or vcl_deliver and so we end up serving the 5xx content that we got from origin to the client. Although you may want to attempt to restart the request and use a custom header (e.g. set req.http.X-Serve-500-Page = "true") in order to indicate to vcl_recv that you want to short-circuit the request cycle and serve a custom error page instead.

Test Serving Stale

In order to test that you’re serving stale content you’ll need to:

ensure that you are able to force a resource to succeed and fail when necessary.
ensure that you have max-age, stale-while-revalidate and stale-if-error set to short values.

For point 1. we need this because the serving of stale content during stale-if-error will only happen if the backend/origin server fails or returns an error status. But if you’re testing a real backend/origin server then you want it to be running 😅 so being able to force it to look like it failed is ideal for testing our ‘serve stale’ VCL code.

For point 2. we need this because the serving of stale content will only happen when the TTL of the cached resource has expired. If we didn’t do this, then we would have to wait for whatever TTL was set by the backend/origin server to expire (which could be a very long time!).

We’ll serve stale content for a period of time while we try and acquire fresher content (e.g. stale-while-revalidate), followed by serving stale for a period of time if the backend/origin server fails to respond after the revalidation period (e.g. stale-if-error). So again, if either of those stale directives have long TTLs set by the backend/origin server, then it’ll make testing our ability to serve stale impractical.

In order to achieve these two requirements, I’ve found the following approach to work quite well.

First, add the following code to your vcl_fetch subroutine…

if (req.http.X-TTLs == "shorten") { 
  set beresp.ttl = 10s;
  set beresp.stale_while_revalidate = 20s;
  set beresp.stale_if_error = 60s;

  unset beresp.http.ETag;
  unset beresp.http.Last-Modified;
}

if (req.http.X-ResponseState == "fail") {
  set beresp.status = 500;
  set beresp.cacheable = false;
}

if (beresp.status >= 500 && beresp.status < 600) {
  if (stale.exists) {
    return(deliver_stale);
  }
}

ℹ️ NOTE

you should already have the code that checks for stale.exists etc, but I’m including it for the sake of clarity.

Mistake 1.

Now the mistake I initially made was instead of setting beresp.ttl, beresp.stale_while_revalidate, beresp.stale_if_error (as shown above) I tried to manipulate beresp.http.Surrogate-Control like so:

set beresp.http.Surrogate-Control = "max-age=10, stale-while-revalidate=20, stale-if-error=60";)

The reason that doesn’t work is that it would only affect downstream caches (inc. a Fastly cache if this cache node was a ‘shield’ node). This is because the Surrogate-Control and Cache-Control headers are parsed before vcl_fetch is called.

So if we want to affect the duration for which Fastly caches an object (inc. stale TTLs) we need to manipulate the beresp.ttl, beresp.stale_while_revalidate, beresp.stale_if_error variables.

With the above VCL in place we can make a request with the X-TTLs header set with a value of shorten to cause the cached object to have a shorter set of caching TTLs than the origin intended.

Then we can wait for the max-age TTL to expire before making the same request again and see that now Fastly will attempt to revalidate the content and so during its stale-while-revalidate time period Fastly will serve stale content back to us.

Then we can wait for the stale-while-revalidate TTL to expire before making the same request again but changing X-TTLs to X-ResponseState and setting the value to fail, which means when Fastly attempts to acquire the content from the origin it’ll think it received an error from the backend/origin server and go ahead and serve stale content back to us.

Mistake 2.

Another thing I had initially done wrong was not handle the situation where the origin would serve back a 304 Not Modified. In this scenario Varnish will not execute vcl_fetch (why would it? there’s no change in the content, based upon its ‘conditional request’ it made and so no FULL request needs to be made to the origin to fetch the full response).

The reason this causes an issue is because along with a 304 not executing vcl_fetch (and thus our cache directives and failure modifications wont be actioned) it means the object in the cache will be updated in a way that can break our expectations.

Specifically, when Varnish receives a 304 from origin, it will re-cache the object with the original TTLs (normally this would be fine as this would be our modified cache directives) but it’ll only do that if there is no valid Cache-Control or Surrogate-Control header found in the 304 response from origin.

So in my case, our origin responded with a 304 and also provided the original (i.e. unmodified) Surrogate-Control header, and thus those TTLs were being applied to the re-cached object. Making my testing code’s assertions about the state of the object incorrect.

This had me confused for a long time. With the help of Fastly support we realized the best way to avoid this 304 response from messing up our expectations was to cause strip the ETag and Last-Modified headers from the first 200 OK response we got back from the origin.

The reason we do this is so that the Varnish cache has no way of making a conditional request when it comes to us testing stale-while-revalidate and so it is forced to make a full request to the origin, where upon Varnish will have to execute vcl_fetch and our code modifications will again be applied.

If we didn’t do that, then there would literally be no way of testing our website’s ability to serve stale unless we changed the backend to a different origin server that was pre-configured to respond how we needed it to.

Having a different backend was something we had already done in the past and helped us to validate our VCL code and cache headers were correct, but it didn’t tell us if our real pages would serve stale at the critical time that we needed them to – so that’s way we wanted to control everything from VCL rather than using a special backend.

ℹ️ NOTE

here is an example Python3 script that demonstrates how we verify these behaviours.

Disable Caching

It’s possible to disable caching for either the client or fastly, or both! But it gets confusing with all the various documentation pages fastly provides to know which one is the source of truth (useful information is spread across all of them).

In my experience I’ve found the following to be sufficient…

ℹ️ NOTE

any time you want Fastly to cache your content use Surrogate-Control, and this will take precedence over Cache-Control EXCEPT when Cache-Control has the value private included somewhere inside it. So generally if I’m doing that I’ll just make sure I’m not sending a Surrogate-Control as it’ll just be confusing for anyone reading that code.

Disable Client Caching: Cache-Control: no-store, must-revalidate (docs)
Disable CDN Caching: Cache-Control: private (docs)
Disable ALL Caching: Cache-Control: no-cache, no-store, private, must-revalidate, max-age=0, max-stale=0, post-check=0, pre-check=0 + Pragma: no-cache + Expires: 0 (docs)

Logging

With Fastly, to set-up logging you’ll need to use their UI, as this means they can configure the relevant integration with your log aggregation provider. This is fairly straight forward and obvious but what people don’t realise is that by default Fastly will generate a subroutine called vcl_log.

Now, as part of the log configuration in the Fastly UI you must specify a “log format” (think an Apache web server log format syntax). If you don’t set a value for the log format (e.g. it’s an empty field) then the VCL will have a vcl_log subroutine that looks something like the following:

sub vcl_log {
#--FASTLY LOG START
  # default response conditions
  log {"syslog <service_id> <service_name> :: "} ;
#--FASTLY LOG END
}

Effectively it’s an empty log message :-/

If you do provide a log format value, then it could look something like the following:

sub vcl_log {
#--FASTLY LOG START
  # default response conditions
  log {"syslog <service_id> <service_name> :: "} 
  req.http.Fastly-Client-IP " [" strftime({"%Y-%m-%d %H:%M:%S"}, time.start) "." 
  time.start.msec_frac {"] ""} cstr_escape(req.request) " " cstr_escape(req.url)
  " " cstr_escape(req.proto) {"" "} resp.status " " resp.body_bytes_written " "
  tls.client.protocol " " fastly_info.state " " req.http.user-agent;
#--FASTLY LOG END
}

OK, so another important thing to think about is that the vcl_log subroutine executes after vcl_deliver. So it’s the very last routine to be executed before Varnish completes the request (so this is even after it has delivered the response to the client).

The reason this subroutine exists is because logging inside of vcl_deliver (which is how Fastly used to work, i.e. they would auto-generate the log call inside of vcl_deliver) Fastly wouldn’t have certain response variables available, such as determining how long it took to send the first byte of the response to the client.

What confused me about the logging set-up was the fact that I didn’t realise I was looking at things too much from an ‘engineering’ perspective. By that I mean, not all of the UI features Fastly provides exist just for my benefit :-) and I should probably elaborate on what I mean by that…

So what I was confusedly thinking was: “why is Fastly generating only a single log call in vcl_log, is there a technical reason for that or a limitation with Varnish? I want to make multiple log calls throughout my request/response flow, I don’t want just a single log at the end as that’s not very useful to me!”.

Well it ended up simply just being that some Fastly users don’t want to write their own VCL (or don’t know anything about writing code) and so they are quite happy with knowing that the UI will trigger a single log call at the end of each request cycle.

Where (as a programmer) I was expecting to add log calls all over the place! Turns our this was possible of course, but I also never realised a vcl_log subroutine existed nor that it was being auto-generated for me by Fastly once I had configured my logging endpoint in their UI.

This meant I also never realised that an extra log call (after all the log calls I was manually adding to my custom VCL) was being executed!

So when I eventually stumbled across a Fastly documentation page talking about “duplicate logs”, I started digging deeper into why that might be …and that’s where I discovered vcl_log was a thing and I needed to read up on it and how it worked and why it existed.

Now, the documentation then goes on to mention using the UI to create a custom ‘condition’ to prevent Fastly’s auto-generated log from being executed. This intrigued me. I was thinking “what is this magical ‘condition feature’ they have?”. Well, turns out that I was again thinking too much like an engineer and not enough like a normal Joe Bloggs user who knows nothing about programming, as this ‘feature’ just generates a standard conditional ‘if statement’ code block around the auto-generated log call, like so:

sub vcl_log {
#--FASTLY LOG START
  # default response conditions
  # Response Condition: generate-a-log Prio: 10
  if( !req.url ) {
    log {"syslog <service_id> <service_name> :: "} ;
  }
#--FASTLY LOG END
}

As you can probably tell, this recommended condition will never match and so the final log call isn’t executed, thus avoiding an unneccessary duplicate log call.

Mystery solved.

Filtering and Sampling Logs

I want to also mention that logging can be expensive if you’re sending ALL your Fastly traffic logs to a log aggregation/analysis service (such as Datadog), and so two things we did to help combat this problem was…

only send ‘uncacheable request’ logs
only send a sample of our logs

The following VCL snippet demonstrates how to do this…

vcl_log {
  ...

  // 75% of uncacheable logs will be sampled
  if (fastly_info.state ~ "^(MISS|PASS)" && randombool(3,4)) {
    log ...;
  }
}

Logging Memory Exhaustion

We discovered an issue with our logging code which meant we would see (null) log lines being generated. The cause of this turned out to be the Fastly ‘workspace’ (as they refer to it) would run out of memory while generating our log output that they would stream to either GCS or AWS S3.

You can inspect the workspace using the undocumented workspace.bytes_free property, and this value will change as you make modifications to your request/response and other related objects.

The ‘workspace’ is the amount of memory allocated to each in-flight request, and its value is set to 128k. So although the JSON object we’re building is small, prior to that, our request flow might be making lots of HTTP header modifications and all those things are counting towards the overall memory being consumed.

So ultimately be wary of making too many HTTP modifcations as you might discover you’ll end up losing log data.

JSON Structured Logging

ℹ️ NOTE

see the update for a way to use ‘long strings’ to construct easier to read JSON logs.

Generating JSON structured output is not easy in VCL. The below VCL snippet demonstrates the ‘manual’ approach (encoding " as %22 and concatenating across multiple lines). This was the best solution for us as the alternatives were unsuitable:

github.com/fastly/vcl-json-generate is wildly verbose in comparison, although it’s supposed to make things easier?
using a “long string” to avoid encoding nested double quote, but then everything must be all on one line!?, e.g. {"{ "foo":""} req.http.X-Foo {"", "bar":"} req.http.X-Bar {" }"};

ℹ️ NOTE

we have two seperate services, one production and one staging and we run the same VCL code in both. This requires us to have logic that checks whether our code is running in the stage environment or not (see below for example).

declare local var.stage_service_id STRING;
declare local var.prod_service_id STRING;
declare local var.running_service_id STRING;

set var.stage_service_id = req.http.X-Fastly-ServiceID-Stage;
set var.prod_service_id = req.http.X-Fastly-ServiceID-Prod;
set var.running_service_id = var.stage_service_id;

if (req.service_id == var.prod_service_id) {
  set var.running_service_id = var.prod_service_id;
}

declare local var.json STRING;

set var.json = "{" +
  "%22http%22: {" +
    "%22body_size%22: %22" + resp.body_bytes_written + "%22," +
    "%22content_type%22: %22" + resp.http.Content-Type + "%22," +
    "%22host%22: %22" + req.http.host + "%22," +
    "%22method%22: %22" + req.method + "%22," +
    "%22path%22: %22" + json.escape(req.url) + "%22," +
    "%22protocol%22: %22" + req.proto + "%22," +
    "%22request_time%22: " + time.to_first_byte + "," +
    "%22served_stale%22: " + req.http.X-BF-Served-Stale + "," +
    "%22status_code%22: " + resp.status + "," +
    "%22tls_version%22: %22" + tls.client.protocol + "%22," +
    "%22uri%22: %22" + json.escape(if(req.http.Fastly-SSL, "https", "http")
    + "://" + req.http.host + req.url) + "%22," +
    "%22user_agent%22: %22" + json.escape(req.http.User-Agent) + "%22" +
  "}," +
  "%22network%22: {" +
    "%22client%22: {" +
      "%22ip%22: %22" + req.http.Fastly-Client-IP + "%22" +
    "}," +
    "%22server%22: {" +
      "%22state%22: %22" + fastly_info.state + "%22" +
    "}" +
  "}," +
  "%22timestamp%22: " + time.start.msec + "," +
  "%22timestamp_sec%22: " + time.start.sec + "," + # exists to support BigQuery
  "%22upstreams%22: {" +
    "%22service%22: %22" + req.http.X-BF-Backend + "%22" +
  "}"
"}";

There was a way to combine long strings ({"..."}, e.g. to use double quote inside of a string:{"""}) over multiple lines (demonstrated below) but I’d argue it’s still not as readable as you might imagine, and missing just one opening or closing bracket will cause you nothing but pain and heartache:

declare local var.json STRING;

set var.json = "{" +
  {""http": {"} +
    {""body_size": ""} + resp.body_bytes_written + {"","} +
    {""content_type": ""} + resp.http.Content-Type + {"","} +
    {""host": ""} + req.http.host + {"","} +
    {""method": ""} + req.method + {"","} +
    {""path": ""} + json.escape(req.url) + {"","} +
    {""protocol": ""} + req.proto + {"","} +
    {""request_time": "} + time.to_first_byte + "," +
    {""status_code": "} + resp.status + "," +
    {""tls_version": ""} + tls.client.protocol + {"","} +
    {""uri": ""} + json.escape(if(req.http.Fastly-SSL, "https", "http") 
    + "://" + req.http.host + req.url) + {"","} +
    {""user_agent": ""} + json.escape(req.http.User-Agent) + {"""} +
  "}," +
  {""network": {"} +
    {""client": {"} +
      {""ip": ""} + req.http.Fastly-Client-IP + {"""} +
    "}," +
    {""server": {"} +
      {""state": ""} + fastly_info.state + {"""} +
    "}" +
  "}," +
  {""timestamp": "} + time.start.msec + "," +
  {""timestamp_sec": "} + time.start.sec + # exists to support BigQuery
"}";

log var.json;

So manually constructing the JSON was what we opted for, and once you get used to the endless %22 it’s actually quite readable IMHO (when compared to the alternatives we just described).

Fixing the Memory Issue

Now here’s how we worked-around the memory issue causing a (null) log line: before we trigger our log call we first check that Fastly’s workspace hasn’t been exhausted (the variable contains the error code raised by the last function). If we find an ‘out of memory’ error, then we won’t attempt to call the log function (as that would result in the (null) for a log line).

Documentation: docs.fastly.com/vcl/variables/fastly-error/

It’s important to reiterate that this isn’t a solution, but a work-around. We don’t know how many logs we’re losing due to memory exhaustion (it could be lots!) so this is a problem we need to investigate further (as of October 2019, …wow did I really start writing this post two years ago!!? time flies heh).

if (fastly.error != "ESESOOM") {
  log "syslog " var.running_service_id " logs_to_s3 :: " var.json;
  log "syslog " var.running_service_id " logs_to_gcs :: " var.json;
}

ℹ️ NOTE

logs_to_s3 and logs_to_gcs are references to the two different log streams we setup within the Fastly UI.

Update 2020.09.14

Whilst reading through a fastly blog post on NEL “Network Error Logging” I noticed the secret sauce that makes using long strings for JSON formatting actually work! You have to remove the line breaks introduced by the long strings 🤦🏼‍♂️

Specifically:

set var.json = regsuball(var.json, {"\s(?=([^"]*\"[^"]*")*[^"]*$)"},"");

Here’s the full example:

declare local var.json STRING;

set var.json = {"{
  "http": {
    "body_size": ""} resp.body_bytes_written {"",
    "content_type": ""} resp.http.Content-Type {"",
    "host": ""} req.http.host {"",
    "method": ""} req.method {"",
    "path": ""} json.escape(req.url) {"",
    "protocol": ""} req.proto {"",
    "request_time": "} time.to_first_byte {",
    "status_code": "} resp.status {",
    "tls_version": ""} tls.client.protocol {"",
    "uri": ""} json.escape(if(req.http.Fastly-SSL, "https", "http") 
    + "://" + req.http.host + req.url) + {"",
    "user_agent": ""} json.escape(req.http.User-Agent) {"",
  },
  "network": {
    "client": {
      "ip": ""} req.http.Fastly-Client-IP {"",
    "server": {
      "state": ""} fastly_info.state {""
    },
  },
  "timestamp": "} time.start.msec {",
  "timestamp_sec": "} time.start.sec /* exists to support BigQuery */ {"}
}"};

set var.json = regsuball(var.json, {"\s(?=([^"]*\"[^"]*")*[^"]*$)"},"");

log var.json;

I’ll admit this is definitely easier to read than the HTML encoding equivalent. It’s still a bit confusing to get right, when creating big nested structures, but no worse than a bunch of %22 throughout the structure 😬

Restricting requests to another Fastly service

We had a requirement where by we had a request flow that looked something like the following…

Client > Fastly (service: foo) > Fastly (service: bar) > Origin

In this request flow both the Fastly services were managed by us (i.e. managed by BuzzFeed) and we wanted to ensure that the service bar could only accept requests when they came via service foo.

We didn’t want to rely on HTTP request headers as these can be spoofed very easily. Fastly suggested the following VCL based solution…

if (!req.http.myservice && (client.ip ~ fastly_ip_ranges))
 error 808;
}

ℹ️ NOTE

808 is a custom error status (see Error Handling for more context about using non-standard status codes). I typically prefer to use the 9xx range.

The idea being we can check if the client.ip matches a known Fastly POP IP range. If it doesn’t match that then we’ll reject the request. Additionally, service bar would check to see if a special request header was set by service foo (e.g. set req.http.myservice = "yes";).

An alternative approach also suggested by Fastly was to replace the IP check for a request header check for a Fastly specific header Fastly-FF (which is only set when the request came from a Fastly POP):

if (req.http.fastly-ff && !req.http.myservice) {
  error 808;
}

Custom Error Pages

Near the beginning of this post we looked at error handling in VCL. I want to now revisit this topic with regards to serving your own custom error pages from the edge synthetically, so as to give you a real example of how this might be put together.

Below is example code that you can also run via Fastly’s Fiddle tool:

sub vcl_recv {
  if (req.restarts == 1) {
    if (req.http.X-Error) {
      error std.strtol(req.http.X-Error, 10);
    }
  }
}

sub vcl_deliver {
  if (req.restarts == 0 && resp.status >= 400) {
    if(fastly_info.state ~ "HIT($|-)") {
      set req.http.X-FastlyInfoStatus = fastly_info.state;
      set req.http.X-CachedObjectHitCounter = obj.hits;
    }
    set req.http.X-Error = resp.status;
    return(restart);
  }

  if (req.restarts == 1) {
    if (req.http.X-FastlyInfoStatus) {
      set resp.http.X-Cache = req.http.X-FastlyInfoStatus;
    }
    
    if (req.http.X-CachedObjectHitCounter) {
      set resp.http.X-Cache-Hits = std.strtol(req.http.X-CachedObjectHitCounter, 10);
    }
  }
}

vcl_error {
  synthetic {"
    <!doctype html>
    <html>
    <head>
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <title>Page Not Found</title>
      <meta name="generator" content="fastly">
    </head>
      <body>
        error happened
      </body>
    </html>
  "};

  return(deliver);
}

The flow is actually quite simple once you have these pieces put together. The principle of what we’re trying to achieve is that we want to serve a custom error page from the edge server (rather than using the error page that may have be sent by an origin).

The way we’ll do this is that when we get the error response from the origin we’ll restart the request and upon restarting the request we’ll trigger vcl_error (via the error directive) and construct a synthetic response.

Once we have that synthetic response we’ll also need to make some modifications so that Fastly’s default VCL doesn’t make it look like we don’t get cache HITs from future requests for this same resource.

This otherwise happens because Fastly’s default VCL has a basic check in vcl_deliver for an internal ‘hit’ state which does occur, but then because we restart the request and trigger vcl_error, that internal hit state is replaced by an ‘error’ state and so it ends up confusing Fastly’s VCL logic.

Let’s dig into the code a bit…

So first of all we need to check in vcl_deliver for an error status (and because we plan on restarting the request, we don’t want to have an endless loop so we’ll check the restart count as well):

if (req.restarts == 0 && resp.status >= 400) {

If we find we have indeed come from vcl_hit, then we store off contextual information about that cached object into the req so it will persist the restart and thus be usable later on when required:

set req.http.X-FastlyInfoStatus = fastly_info.state;
set req.http.X-CachedObjectHitCounter = obj.hits;

Regardless of whether we came from vcl_hit or if we came from vcl_fetch, we store off the error response status code into the req object so again we may access it after the request is restarted:

set req.http.X-Error = resp.status;

Finally, we’ll restart the request. At this point let’s now look at vcl_recv where we first check if we are dealing with a restart, and if so we extract the response error status code and pass it over to vcl_error by way of the error directive (notice we have to use a Fastly function to convert it from a STRING to a INTEGER):

if (req.restarts == 1) {
  if (req.http.X-Error) {
    error std.strtol(req.http.X-Error, 10);
  }
}

Once we reach vcl_error we can now construct our custom error page. In the example code I give I just use a single synthetic object to represent the error but in real-life you’d check the status code provided to vcl_error and then construct a specific error page. Once we create the object we jump back over to vcl_deliver.

Now we’re back at vcl_deliver for the second time you’ll find the first check we had will no longer pass (avoiding an endless loop) but the second check will pass:

if (req.restarts == 1) {

Once we know we’re into the restarted request we can check the req object for the headers/information we were tracking earlier and use that information to update two response headers that otherwise are set by Fastly: X-Cache and X-Cache-Hits.

The reason we need to override these two headers is because otherwise Fastly records them as X-Cache: MISS and X-Cache-Hits: 0 even after a secondary request for the same resource. This would be concerning to people, because it seems to suggest we went back to the origin again when we should have gotten a cache HIT (we in fact do get a cache HIT but Fastly is incorrectly reporting that fact).

To understand what’s going on with Fastly incorrectly reporting the state of the request, let’s look at the request flow…

# Request 1

vcl_recv > vcl_hash > vcl_miss > vcl_fetch > vcl_deliver (restart) > vcl_recv > vcl_error > vcl_deliver.

# Request 2

vcl_recv > vcl_hash > vcl_hit > vcl_deliver (restart) > vcl_recv > vcl_error > vcl_deliver.

For request 1 we have a cold cache scenario whereby the requested resource (e.g. /i-dont-exist) isn’t cached. So we go to the origin and the origin returns a 404 Not Found. At that point vcl_deliver identifies the error and restarts the request. We construct a custom error page within vcl_error and finally serve the custom error page via vcl_deliver.

After request 1 the response headers would suggest (correctly) that we got a X-Cache: MISS and X-Cache-Hits: 0.

So what happens if we request /i-dont-exist again? Well, unless we fixed those headers, the cache miss and zero hits would be reported again even though we know that the second request got a cache HIT.

This happens because Fastly’s internal logic for setting X-Cache: MISS and X-Cache-Hits looks a bit like this:

set resp.http.X-Cache = if(resp.http.X-Cache, resp.http.X-Cache ", ","") 
if(fastly_info.state ~ "HIT($|-)", "HIT", "MISS");

if(!resp.http.X-Cache-Hits) {
  set resp.http.X-Cache-Hits = obj.hits;
} else {
  set resp.http.X-Cache-Hits = resp.http.X-Cache-Hits ", " obj.hits;
}

Even though the second request gets a cache HIT for the origin’s original error response, the serving of the custom error page from the edge causes the Fastly internal state (i.e. fastly_info.state) to report as ERROR instead of a HIT (that happens because we call error from within vcl_recv after restarting the request).

This is why when we get back into vcl_deliver we override those headers according to the values we tracked within the initial request flow.

Security

There are some aspects of the CDN where you need to take extra precautions to protect your upstreams (and in some cases your customers data!). Let’s dig into some of the concepts…

Rate Limiting

Rate limiting clients at the edge currently (as of March 2020) isn’t easy because we don’t have a fully fledged programming environment to work with (although Fastly is releasing a new ‘compute@edge’ serverless service which aims to tackle that problem).

ℹ️ NOTE

there’s an alternative approach to what I discuss below, and it is slightly more involved because it attempts to handle web scraper/fuzzer tools. I’ve blogged about it as a separate article here: Rate Limiting at the CDN Edge.

In the mean time this leaves us with various ways of implementing rate limiting with Fastly’s version of Varnish and VCL. One approach I have is a multi-pronged approach and it ultimately requires your origin (or at least some service behind Fastly) to handle the rate limiting and to be able to return 429 Too Many Requests for a client that is hitting certain exposed endpoints too hard.

Once you have rate limiting in place, the next question becomes: how can I leverage the CDN and the edge to help protect my origin and its infrastructure from having to handle requests that my origin will just end up rejecting with a 429 response any way?

For this, I present to you a solution that works like so:

We initially allow POST, PUT and DELETE requests to be looked up in the cache (they’re typically uncached because they’ll usually result in exposing sensitive data if otherwise cached and returned as a cache HIT to multiple clients).
Once we get a response back from the origin we check if the status was a 429 and if so we assign a Vary header and we set the response to be cacheable (because 429 status isn’t cached by default).
If the response was a 200 OK for a POST request, then we’ll skip storing the response in the cache.
To ensure POST, PUT and DELETE requests don’t become sequentially blocking requests when doing cache lookups, we’ll utilize req.hash_ignore_busy to disable request collapsing.

Here’s what the code looks like (it makes changes to two states: vcl_recv and vcl_fetch):

sub vcl_recv {
  if (req.restarts == 0) {
    if (!req.http.User-Agent) {
      set req.http.UserAgent = "Fallback";
    }
  }

  // force cache 'lookup' for methods that are otherwise normally uncached
  // and mimic the standard behaviour of disabling 'request collapsing'
  if (req.method ~ "(POST|PUT|DELETE)") {
    set req.http.X-PassMethod = "true";
    set req.hash_ignore_busy = true;
    return(lookup);
  }
}
    
sub vcl_fetch {
  // this conditional is used for testing the logic
  // and isn't required for actual implementation
  if (req.http.X-429) {
    set beresp.status = 429;
  }

  declare local var.vary STRING;
  set var.vary = "Fastly-Client-IP, User-Agent";

  if (beresp.status == 429) {
    if (!beresp.http.Vary) {
      set beresp.http.Vary = var.vary;
    } else {
      set beresp.http.Vary = beresp.http.Vary + ", " + var.vary;
    }

    set beresp.cacheable = true;
    if (!beresp.http.Surrogate-Control:max-age && !beresp.http.Cache-Control:max-age) {
      set beresp.ttl = 1m; // 1 minute
    }
  }

  if (req.http.X-PassMethod) {
    if (beresp.status != 429) {
      set beresp.cacheable = false;
    }
  }

  return(deliver);
}

You should pay attention to the fact that we Vary on the client’s IP and their User-Agent, both of which can be changed. So it’s important to realize this solution doesn’t solve DDoS attacks, and in some cases won’t help with standard DoS attacks if the attacker is rotating their IP or User-Agent.

One caveat to this approach (as noted by Fastly) is as follows:

POST requests are also ineligible for clustering, ie transferring the request to a consistently-hashed storage node. Since we don’t expect POSTs to be cached, this is an optimisation and we unfortunately don’t provide a way for you to override it. The effect of this is that even if you do enable POSTs to be cached, your cache hit ratio will be poor. This may be fine if your intention is just to use it for rate limiting.

One thing you might be wondering is why I chose to allow return(lookup) on POST, PUT and DELETE methods, then set the response to cacheable in vcl_fetch, rather than keep the typical boilerplate logic which would check for those methods and then return(pass).

This is because once a request has been ‘passed’ in vcl_recv then it can’t be set to be cacheable later (in vcl_fetch) as we would have disabled caching completely. You’ll be able to see tell this if you check the special fastly_info.state variable (or test the code via fastly’s fiddle tool) as this will report back a pass state.

Path Traversal

A path traversal vulnerability is the ability to craft a request that will cause the server to make a request to somewhere different to what the request would initially suggest.

This is best explained by a real example. This isn’t a real example in the sense of a real vulnerability, we’re just going to utilize two separate tools (curl and httpbin.org) to help us understand the concept of the vulnerability.

So we’ll use the popular httpbin.org service as an upstream server. The service provides (among many others) the endpoint /status/200 which will return a 200 OK response, but it also provides an /anything endpoint which will display JSON data about the incoming request (including the path you specified).

If I made a request to https://httpbin.org/anything/foo then it would show /anything/foo as the request path.

Below is the request we’re going to make (pay attention to the path):

curl -v "https://httpbin.org/status/200/../../anything/status/404/"

This ^^ request has ../../ inside of it, which curl (and this all depends on the server and/or the software you test with/against) will collapse down. This means the request path becomes /anything/status/404.

We can see the response from httpbin.org was:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.64.1",
    "X-Amzn-Trace-Id": "Root=1-5f48c827-200a853c97290c8e6fce74ca"
  },
  "json": null,
  "method": "GET",
  "origin": "185.192.70.151",
  "url": "https://httpbin.org/anything/status/404/"
}

ℹ️ NOTE

you can actually tell curl to not collapse the path when it sees ../../, by using the --path-as-is flag, and so if we made the earlier request with this flag added, then httpbin.org would have rejected the request as it wouldn’t have a route defined on the server end that recognized the full path.

We can now see how this might be a very dangerous attack because you could construct a request to a server and trick it into communicating with any private/internal APIs it has access to. In essence this type of attack could enable the caller to access whatever data the server would normally only have direct access to.

So how can we prevent this from happening? Well the solution is to parse the request looking for invalid characters such as .. or the encoded variation of a . which is %2E. This can happen at the application layer but I prefer to handle this at the edge so our upstream services and infrastructure don’t have to deal with the request at all.

Here is an example of some VCL that handles this (it’s a separate file security.vcl):

// services that include this shared VCL should ensure they do not utilize the
// same error code, otherwise they may end up sending the wrong response.
//

sub security_recv {
  // we want to prevent path traversal vulnerabilities such as:
  //
  // curl -v "https://httpbin.org/status/200/../../anything/status/404/"
  //
  // this ^^ would cause the server to go to /anything/status/404/ not /status/200
  //
  // this could be an issue because the upstream server might be able to
  // communicate with private/internal APIs, and so this type of attack could
  // enable the caller to access whatever data the server would normally only
  // have access to.
  //
  // example pattern match:
  // https://regex101.com/r/RYhmwW/2
  //
  // NOTES:
  // we utilize a 'long string' {"..."} instead of a string literal "..."
  // to avoid interpretting the %2E as a period character when VCL statically
  // compiles a regex (which would change the pattern quite significantly!)
  //
  // DOCUMENTATION:
  // https://developer.fastly.com/reference/vcl/types/string/
  //
  if (req.url.path ~ {"(?i)(\.|%2E){2}(/|%2F)"}) {
    error 699 "Bad Request";
  }
}

sub security_error {
  if (obj.status == 699) {
    set obj.status = 400;
    synthetic {"Bad Request"};
    return(deliver);
  }
}

You can then include this file into your main.vcl like so:

include "security"

sub vcl_recv {
  #FASTLY recv

  call security_recv;

  return(lookup);
}

sub vcl_error {
  #FASTLY error

  call security_error;
}

ℹ️ NOTE

CAREFUL! when VCL statically compiles a regex it will attempt to ‘interpret’ it unless the string literal (used to provide the regex pattern) is a ‘long string’ such as {"..."}. Please refer to the string documentation.

Conclusion

So there we have it, a run down of how some important aspects of Varnish and VCL work (and specifically for Fastly’s implementation).

One thing I want to mention is that I am personally a HUGE fan of Fastly and the tools they provide. They are an amazing company and their software has helped BuzzFeed (and many other large organisations) to scale massively with ease.

You’ve been a wonderful audience. Hope you enjoyed this post. Be sure to tip your waiter.

And remember…

Reference material

I would highly recommend watching this talk by Rogier Mulhuijzen (Senior Varnish Engineer - who currently works for Fastly) on “Advanced VCL”: vimeo.com/226067901. It goes into great detail about some complex aspects of VCL and Varnish and really does a great job of elucidating them.

Also recommended is this Fastly talk which details how ‘clustering’ works (see also this fastly community support post that details how to utilize the request headers Fastly-Force-Shield and Fastly-No-Shield).

Lastly there is this talk about vcl_fetch that also covers clustering and shielding, that can help tie up any other loose ends related to Fastly’s system design.

Lastly, there was a recent article from an engineer working at the Financial Times, detailing the complete request flow from DNS to Delivery. It’s very interesting and covers a lot of information about Fastly. Highly recommended reading.

Profiling Python

Tue, 31 Oct 2017 00:00:00 +0000

Memory Management

Before we dive into the techniques and tools available for profiling Python applications, we should first understand a little bit about its memory model as this can help us to understand what it is we’re seeing in relation to memory consumption.

Python manages memory using reference counting semantics. What this means is, every time an object is referenced, either by a variable assignment or similar, the counter for that object is incremented. Once an object is not referenced anymore its memory is deallocated (its counter is decremented every time a reference is removed, until it reaches zero). But as long as there is a reference (somewhere in the program), then the object will not be deallocated (as the internal counter will be greater than zero).

Now this can cause problems when dealing with cyclical references, so that’s something to be aware of when investigating memory leaks and other memory related concerns.

ℹ️ NOTE

for lots of details of how Python allocates memory, I highly recommend this presentation.

Types of Profiling

There are a couple of approaches available to us for monitoring performance…

Timers: useful for benchmarking, as well as comparing before and after fixes.
Profilers: useful for high-level verification.

Tools Matrix

	Pros	Cons
timer (decorator)	- Simple, quick and easy.	- Requires code change. - Adds latency & skews results.
timeit module	- Calculate repeat averages. - Doesn’t require code change.	- More complicated API.
profiler module	- Granular CPU by file. - Can be run from terminal.	- More complicated results. - Read docs to understand.
line_profiler	- Granular line-by-line CPU.	- Slow. - Not built-in package. - Requires code change.
memory_profiler	- Clear and easy results.	- Slow † - Not built-in package. - † additional packages help.
tracemalloc	- Built-in memory package.	- Requires code change. - More complicated API.
pyflame	- Visualise problem area easily. - Details CPU and Memory	- Requires Linux. - Most complex to setup.

Analysis Steps

Regardless of the tool you use for analysis, a general rule of thumb is to:

Identify a bottleneck at a high-level

For example, you might notice a long running function call.

Reduce the operations

Look at time spent, or number of calls, and figure out an alternative approach.
Look at the number of memory allocations, figure out an alternative approach.

Drill down

Use a tool that gives you data at a lower-level.

Think about more performant algorithms or data structures.
There may also be simpler solutions.
Take a pragmatic look at your code.

Base Example

Let’s begin with a simple program written using Python 3…

def get_number():
    for i in range(10000000):
        yield i


def expensive_function():
    for n in get_number():
        r = n ^ n ^ n
    return f"some result! {r}"


print(expensive_function())

ℹ️ NOTE

I’m currently using Python version 3.6.3

Running this program can take ~1.8 seconds and returns the value:

some result! 9999999

Timer

We can use a simple decorator to time the length of our expensive_function call…

from time import time
from functools import wraps


def timefn(fn):
    """Simple timer decorator."""
    @wraps(fn)
    def measure_time(*args, **kwargs):
        t1 = time()
        result = fn(*args, **kwargs)
        t2 = time()
        print(f"@timefn: {fn.__name__} took {str(t2 - t1)} seconds")
        return result
    return measure_time

The problem with this approach is that the decorator results in additional latency. Meaning the program takes slightly longer to complete. Not a lot, but if you’re after precision then this can skew the results (which is a common theme when benchmarking or profiling).

Built-in module: timeit

The built-in timeit module is another simple way of benchmarking the time it takes for a function to execute. Simply import the module and call its interface.

import timeit
…
result = timeit.timeit(expensive_function, number=1)  # one repetition
print(result)

ℹ️ NOTE

you can tweak the number param to determine how many repetitions it’ll run.

Along with the timeit method, there is a repeat method that returns a set of averages across the number of repeated code executions.

result = timeit.repeat(expensive_function, number=1, repeat=5)

In this case result would contain something like:

[1.7263835030025803, 1.7080924350011628, 1.6802870190003887, 1.6736655100103235, 1.7003267239924753]

ℹ️ NOTE

according to the Python documentation when utilising the repeat method you should only be interested in the min() because…

“In a typical case, the lowest value gives a lower bound for how fast your machine can run the given code snippet; higher values in the result vector are typically not caused by variability in Python’s speed, but by other processes interfering with your timing accuracy”.

Finally, there is also a command line version you can utilise:
python -m timeit

Built-in module: profiler

There are two flavours of profiler, a pure Python version (import profile) and a C extension version (import cProfile) which is preferred, as the former is remarkably slower.

For example, the C profile took ~3 seconds to complete, whereas the Python version took over a minute.

ℹ️ NOTE

if using cProfile you would execute: cProfile.run("expensive_function()") otherwise you would execute profile.run("expensive_function()").

You should see something like the following displayed after executing your program:

         10000005 function calls in 3.132 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 10000001    1.042    0.000    1.042    0.000 3_profile.py:4(get_number)
        1    2.090    2.090    3.132    3.132 3_profile.py:9(expensive_function)
        1    0.000    0.000    3.132    3.132 <string>:1(<module>)
        1    0.000    0.000    3.132    3.132 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

So from these results we can see:

There were a total of 10000005 function calls.
The get_number function was called the most (10000001).
Every other function in the total recorded, were called just the once.
The expensive_function took a total of 2.090 seconds (exc. sub function calls).
The cumulative time (cumtime) is the tottime plus sub function calls.

ℹ️ NOTE

when there are two numbers in the first column (for example 3/1), it means that the function recursed. The second value is the number of primitive calls and the former is the total number of calls. Note that when the function does not recurse, these two values are the same, and only the single figure is printed.

Instead of printing the results you can pass the run method a second argument which is a filename you want to store the results in. Once there you can use the pstats.Stats module to carry out some post-processing on those results.

Finally, there is also a command line version you can utilise:
python -m cProfile [-o output_file] [-s sort_order] <your_script.py>

Line Profiler

The Line Profiler option gives much more granular detail than the built-in profile module, but it is an external package and so needs to be installed:

pip install line_profiler

Once installed you can write a decorator to wrap the functionality and make it easier for applying to specific functions you want to profile (as demonstrated below).

from line_profiler import LineProfiler


def do_profile(follow=None):
    if not follow:
        follow = []

    def inner(func):
        def profiled_func(*args, **kwargs):
            try:
                profiler = LineProfiler()
                profiler.add_function(func)
                for f in follow:
                    profiler.add_function(f)
                profiler.enable_by_count()
                return func(*args, **kwargs)
            finally:
                profiler.print_stats()
        return profiled_func
    return inner


def get_number():
    for i in range(10000000):
        yield i


@do_profile(follow=[get_number])
def expensive_function():
    for n in get_number():
        r = n ^ n ^ n
    return f"some result! {r}"


print(expensive_function())

We can see the result of executing this program below…

Timer unit: 1e-06 s

Total time: 7.59566 s
File: 4_line_profile.py
Function: get_number at line 23

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    23                                           def get_number():
    24  10000001      3924533      0.4     51.7      for i in range(10000000):
    25  10000000      3671129      0.4     48.3          yield i

Total time: 27.477 s
File: 4_line_profile.py
Function: expensive_function at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @do_profile(follow=[get_number])
    29                                           def expensive_function():
    30  10000001     22122124      2.2     80.5      for n in get_number():
    31  10000000      5354911      0.5     19.5          r = n ^ n ^ n
    32         1            3      3.0      0.0      return f"some result! {r}"

some result! 9999999

ℹ️ NOTE

the Line Profiler is pretty slow (~35s) in comparison to the cProfiler (~4s)

The Line Profiler will typically only analyse the function being decorated. In order for it to include sub function calls, you’ll need to specify them (hence the decorator allows you to provide a list of functions and in there we’ve specified the get_number function).

The order in which you list the sub functions to ‘follow’ doesn’t matter, as the results will always display (top to bottom) the deepest nested sub function to the top level function (so we can see that get_number is nested inside of expensive_function and so it is top of the output).

From these results we can get a line-by-line breakdown of how long (in percentages) each function took to complete. So expensive_function spent ~20% of its time calculating a value to assign to the variable r, and the remaining 80% was spent calculating a value to assign to the variable n (which was the call out to the get_number function).

As for get_number, it was approximately ⁵⁰⁄₅₀ for time between looping the range(10000000) and yield’ing a value back to the caller context (i.e. expensive_function).

Finally, there is also a command line version you can utilise:

kernprof -l [-v view_results] <your_script.py>

If you omit the -l flag, then you can view the results at a later time using:

python -m line_profiler <your_script.py>.lprof

Basic Memory Profiler

There is a module called memory_profiler which is very simple to use, although with the example code we’ve been using it was so painfully slow it was pretty much unusable (I gave up after 5 minutes of waiting). So, because of that issue, I’ll demonstrate a simpler example.

But first you need to install the module:

pip install memory_profiler
pip install psutil  # recommended to help speed up the reporting time

Now you can import the decorator, and apply that to our new ‘slow’ function:

from memory_profiler import profile

@profile
def expensive_memory_allocations():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

print(len(expensive_memory_allocations()))

When you run this program, you’ll see a memory breakdown similar to the following:

Line #    Mem usage    Increment   Line Contents
================================================
    35     27.8 MiB      0.0 MiB   @profile
    36                             def beep():
    37     35.4 MiB      7.6 MiB       a = [1] * (10 ** 6)
    38    188.0 MiB    152.6 MiB       b = [2] * (2 * 10 ** 7)
    39     35.4 MiB   -152.6 MiB       del b
    40     35.4 MiB      0.0 MiB       return a

The second column “Mem usage” indicates the memory consumption for the Python interpreter after that line was executed. The third column “Increment” indicates the difference in increased memory compared to the previous line that was executed. So you can see, for example, when we delete the b variable we are able to reclaim the memory it was holding on to.

Tracemalloc

There is another basic memory profiler that provides similar features called tracemalloc but this particular tool is part of the standard library in Python so it might be preferable to the external library memory_profiler (shown earlier).

import tracemalloc


def get_number():
    for i in range(10000000):
        yield i


def expensive_function():
    for n in get_number():
        r = n ^ n ^ n
    return f"some result! {r}"


tracemalloc.start()

print(expensive_function())

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")

print("[ Top 10 ]")
for stat in top_stats[:10]:
    print(stat)

The output from this example is as follows:

$ time python tracemalloc_example.py

some result! 9999999

[ Top 10 ]
tracemalloc_example.py:17: size=106 B, count=2, average=53 B

ℹ️ NOTE

You might also consider Pympler or ObjGraph for tracking memory usage & object refs.

PyFlame (Flame Graphs)

Flame graphs are a visualization of profiled software (stack traces), allowing the most frequent code-paths to be identified quickly and accurately. Flame graphs allows hot code-paths to be identified quickly.

PyFlame is an effective tool (written in C++) for generating flame graph data, which can help you to understand the CPU and memory characteristics of your services. In some cases it can report more accurate results than those provided by the built-in Python modules.

For more details on the design decisions behind PyFlame and the shortcomings of the other built-in options, then I would recommend reading this article.

PyFlame only works with Linux operating systems and so in order to profile our code (if you’re using macOS like I am), then we’ll have to utilise Docker to help us. Below is a Dockerfile you can use as a basic starting point to try out PyFlame.

ℹ️ NOTE

we also require FlameGraph in order to generate the flame graphs.

FROM python:3.6.3

WORKDIR /pyflame

# Install dependencies required to ‘build’ PyFlame
RUN apt-get update -y
RUN apt-get install -y git autoconf automake autotools-dev g++ pkg-config python-dev python3-dev libtool make

# Build PyFlame
RUN git clone https://github.com/uber/pyflame.git && \
    cd pyflame && ./autogen.sh && ./configure && make

WORKDIR /flamegraph

RUN git clone https://github.com/brendangregg/FlameGraph

COPY 7_pyflame.py /app/app.py

WORKDIR /app

CMD /pyflame/pyflame/src/pyflame -o prof.txt -t python app.py &&\
    /flamegraph/FlameGraph/flamegraph.pl ./prof.txt

In order to build and run this Dockerfile, you’ll need to execute the following code…

docker build -t pyflame .

docker run --privileged pyflame > output.svg && tail -n+2 output.svg > output_stripped.svg

ℹ️ NOTE

our application sends data to stdout (e.g. some result! 9999999) and so this ends up at the top of our output.svg file. This means we need to remove it. We could either modify the application code or you could do what I’ve done and strip it after the file is created by using the tail command and redirecting the stripped output to a new file: output_stripped.svg.

If we now open output_stripped.svg we should see the following interactive flame graph.

Conclusion

That’s our tour of various tools for profiling your Python code. I’ll follow this article up with a Go based one in the very near future. But if you’re interested in further reading then the following blog posts from rushter.com are worth a look:

Profiling Go

Tue, 31 Oct 2017 00:00:00 +0000

ℹ️ NOTE

I highly recommend also reading this official diagnostics documentation.

Memory Management

Before we dive into the techniques and tools available for profiling Go applications, we should first understand a little bit about its memory model as this can help us to understand what it is we’re seeing in relation to memory consumption.

Go’s implementation is a parallel mark-and-sweep garbage collector. In the traditional mark-and-sweep model, the garbage collector would stop the program from running (i.e. “stop the world”) while it detects unreachable objects and again while it clears them (i.e. deallocates the memory). This is to prevent complications where the running program could end up moving references around during the identification/clean-up phase. This would also cause latency and other issues for users of the program while the GC ran. With Go the GC is executed concurrently, so users don’t notice pauses or delays even though the GC is running.

Types of Profiling

There are a couple of approaches available to us for monitoring performance…

Timers: useful for benchmarking, as well as comparing before and after fixes.
Profilers: useful for high-level verification.

Tools Matrix

	Pros	Cons
ReadMemStats	- Simple, quick and easy. - Only details memory usage.	- Requires code change.
pprof	- Details CPU and Memory. - Remote analysis possible. - Image generation.	- Requires code change. - More complicated API.
trace	- Helps analyse data over time. - Powerful debugging UI. - Visualise problem area easily.	- Requires code change. - UI is complex. - Takes time to understand.

Analysis Steps

Regardless of the tool you use for analysis, a general rule of thumb is to:

Identify a bottleneck at a high-level

For example, you might notice a long running function call.

Reduce the operations

Look at time spent, or number of calls, and figure out an alternative approach.
Look at the number of memory allocations, figure out an alternative approach.

Drill down

Use a tool that gives you data at a lower-level.

Think about more performant algorithms or data structures.
There may also be simpler solutions.
Take a pragmatic look at your code.

Base Example

Let’s begin with a simple program written using Go 1.9.2…

package main

import (
    "log"
)

// bigBytes allocates 100 megabytes
func bigBytes() *[]byte {
    s := make([]byte, 100000000)
    return &s
}

func main() {
    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }
}

Running this program can take ~0.2 seconds to execute.

So this isn’t a slow program, we’re just using it as a base to measure memory consumption.

ReadMemStats

The easiest way to look what our application is doing with regards to memory allocation is by utilising the MemStats from the runtime package.

In the following snippet we modify the main function to print out specific memory statistics.

func main() {
    var mem runtime.MemStats

    fmt.Println("memory baseline...")

    runtime.ReadMemStats(&mem)
    log.Println(mem.Alloc)
    log.Println(mem.TotalAlloc)
    log.Println(mem.HeapAlloc)
    log.Println(mem.HeapSys)

    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }

    fmt.Println("memory comparison...")

    runtime.ReadMemStats(&mem)
    log.Println(mem.Alloc)
    log.Println(mem.TotalAlloc)
    log.Println(mem.HeapAlloc)
    log.Println(mem.HeapSys)
}

If I run this program I’ll see the following output:

memory baseline…

2017/10/29 08:51:56 56480
2017/10/29 08:51:56 56480
2017/10/29 08:51:56 56480
2017/10/29 08:51:56 786432

memory comparison...

2017/10/29 08:51:56 200074312
2017/10/29 08:51:56 1000144520
2017/10/29 08:51:56 200074312
2017/10/29 08:51:56 200704000

So we can see the difference between what the go application was using at the point in time that the main function started and when it finished (after allocating a lot of memory via the bigBytes function). The two items we’re most interested in are TotalAlloc and HeapAlloc.

The total allocations shows us the total amount of memory accumulated (this value doesn’t decrease as memory is freed). Whereas the heap allocations indicate the amount of memory at the point in time when the snapshot was taken, and it can include both reachable and unreachable objects (e.g. objects the garbage collector hasn’t freed yet). So it’s important to realise that the amount of memory ‘in use’ could have dropped after the snapshot was taken.

Take a look at the MemStats docs for more properties (inc. Mallocs or Frees).

Pprof

Pprof is a tool for visualization and analysis of profiling data.
It’s useful for identifying where your application is spending its time (CPU and memory).

You can install it using:
go get github.com/google/pprof

To begin with, let’s understand what a “profile” is:

A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation. Packages can create and maintain their own profiles; the most common use is for tracking resources that must be explicitly closed, such as files or network connections. – pkg/runtime/pprof

Now there are a couple of ways to use this tool:

Instrument code within binary to generate a .profile for analysis during development.
Remotely analyse binary via a web server (no .profile is explicitly generated).

ℹ️ NOTE

the profile file doesn’t have to use a .profile extension (it can be whatever you like)

Generate .profile for analysis during development

In this section we’ll look at profiling both CPU and Memory allocation.
We’ll start with CPU profiling.

CPU Analysis

In the following example we’ve modified the application to import "runtime/pprof" and added the relevant API calls in order to record CPU data:

package main

import (
    "log"
    "os"
    "runtime/pprof"
)

// bigBytes allocates 10 sets of 100 megabytes
func bigBytes() *[]byte {
    s := make([]byte, 100000000)
    return &s
}

func main() {
    pprof.StartCPUProfile(os.Stdout)
    defer pprof.StopCPUProfile()

    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }
}

ℹ️ NOTE

we use os.Stdout to make the example easier (i.e. no need to create a file) We’ll just use the shell’s ability to redirect output to create the profile file instead.

We can then build and run the application and save the profile data to a file:
go build -o app && time ./app > cpu.profile

Finally we can inspect the data interactively using go tool like so:
go tool pprof cpu.profile

From here you’ll see an interactive prompt has started up.
So let’s execute the top command and see what output we get:

(pprof) top
Showing nodes accounting for 180ms, 100% of 180ms total
      flat  flat%   sum%        cum   cum%
     180ms   100%   100%      180ms   100%  runtime.memclrNoHeapPointers /.../src/runtime/memclr_amd64.s
         0     0%   100%      180ms   100%  main.bigBytes /.../code/go/profiling/main.go (inline)
         0     0%   100%      180ms   100%  main.main /.../code/go/profiling/main.go
         0     0%   100%      180ms   100%  runtime.(*mheap).alloc /.../src/runtime/mheap.go
         0     0%   100%      180ms   100%  runtime.largeAlloc /.../src/runtime/malloc.go
         0     0%   100%      180ms   100%  runtime.main /.../src/runtime/proc.go
         0     0%   100%      180ms   100%  runtime.makeslice /.../src/runtime/slice.go
         0     0%   100%      180ms   100%  runtime.mallocgc /.../src/runtime/malloc.go
         0     0%   100%      180ms   100%  runtime.mallocgc.func1 /.../src/runtime/malloc.go
         0     0%   100%      180ms   100%  runtime.systemstack /.../src/runtime/asm_amd64.s

So this suggests the biggest CPU consumer is runtime.memclrNoHeapPointers.

Let’s now view a “line by line” breakdown to see if we can pinpoint the CPU usage further.

We’ll do this by using the list <function regex> command.
We can see from the top command that our main function is available via main.main.

So let’s list any functions within the main namespace:

(pprof) list main\.
Total: 180ms
ROUTINE ======================== main.bigBytes in /.../go/profiling/main.go
         0      180ms (flat, cum)   100% of Total
         .          .      6:   "runtime/pprof"
         .          .      7:)
         .          .      8:
         .          .      9:// bigBytes allocates 10 sets of 100 megabytes
         .          .     10:func bigBytes() *[]byte {
         .      180ms     11:   s := make([]byte, 100000000)
         .          .     12:   return &s
         .          .     13:}
         .          .     14:
         .          .     15:func main() {
         .          .     16:   pprof.StartCPUProfile(os.Stdout)
ROUTINE ======================== main.main in /.../code/go/profiling/main.go
         0      180ms (flat, cum)   100% of Total
         .          .     15:func main() {
         .          .     16:   pprof.StartCPUProfile(os.Stdout)
         .          .     17:   defer pprof.StopCPUProfile()
         .          .     18:
         .          .     19:   for i := 0; i < 10; i++ {
         .      180ms     20:           s := bigBytes()
         .          .     21:           if s == nil {
         .          .     22:                   log.Println("oh noes")
         .          .     23:           }
         .          .     24:   }
         .          .     25:}

OK, so yes we can see that 180ms is spent in the bigBytes function and pretty much all of that function’s time is spent allocating memory via make([]byte, 100000000).

Memory Analysis

Before we move on, let’s look at how to profile our memory consumption.

To do this we’ll change our application slightly so that StartCPUProfile becomes WriteHeapProfile (we’ll also move this call to the bottom of our main function otherwise if we keep it at the top of the function no memory has been allocated at that point). We’ll also remove the StopCPUProfile call altogether (as recording the heap is done as a snapshot rather than an ongoing process like with the CPU profiling):

package main

import (
    "log"
    "os"
    "runtime/pprof"
)

// bigBytes allocates 10 sets of 100 megabytes
func bigBytes() *[]byte {
    s := make([]byte, 100000000)
    return &s
}

func main() {
    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }

    pprof.WriteHeapProfile(os.Stdout)
}

Again, we’ll build and execute the application and redirect the stdout to a file (for simplicity), but you could have created the file dynamically within your application if you so choose:
go build -o app && time ./app > memory.profile

At this point we can now run pprof to interactively inspect the memory profile data:
go tool pprof memory.profile

Let’s run the top command and see what the output is:

(pprof) top
Showing nodes accounting for 95.38MB, 100% of 95.38MB total
      flat  flat%   sum%        cum   cum%
   95.38MB   100%   100%    95.38MB   100%  main.bigBytes /...ain.go (inline)
         0     0%   100%    95.38MB   100%  main.main /.../profiling/main.go
         0     0%   100%    95.38MB   100%  runtime.main /.../runtime/proc.go

For a simple example application like we’re using, this is fine as it indicates pretty clearly which function is responsible for the majority of the memory allocation (main.bigBytes).

But if I wanted a more specific breakdown of the data I would execute list main.:

(pprof) list main.
Total: 95.38MB
ROUTINE ======================== main.bigBytes in /.../go/profiling/main.go
   95.38MB    95.38MB (flat, cum)   100% of Total
         .          .      6:   "runtime/pprof"
         .          .      7:)
         .          .      8:
         .          .      9:// bigBytes allocates 10 sets of 100 megabytes
         .          .     10:func bigBytes() *[]byte {
   95.38MB    95.38MB     11:   s := make([]byte, 100000000)
         .          .     12:   return &s
         .          .     13:}
         .          .     14:
         .          .     15:func main() {
         .          .     16:   for i := 0; i < 10; i++ {
ROUTINE ======================== main.main in /.../code/go/profiling/main.go
         0    95.38MB (flat, cum)   100% of Total
         .          .     12:   return &s
         .          .     13:}
         .          .     14:
         .          .     15:func main() {
         .          .     16:   for i := 0; i < 10; i++ {
         .    95.38MB     17:           s := bigBytes()
         .          .     18:           if s == nil {
         .          .     19:                   log.Println("oh noes")
         .          .     20:           }
         .          .     21:   }
         .          .     22:

This indicates where all our memory is allocated on a “line-by-line” basis.

Remotely analyse via web server

In the following example we’ve modified the application to start up a web server and we’ve imported the "net/http/pprof" package which automatically profiles what’s happening.

ℹ️ NOTE

if your application already uses a web server, then you don’t need to start another. The pprof package will hook into your web server’s multiplexer.

package main

import (
    "fmt"
    "log"
    "net/http"
    _ "net/http/pprof"
    "sync"
)

// bigBytes allocates 10 sets of 100 megabytes
func bigBytes() *[]byte {
    s := make([]byte, 100000000)
    return &s
}

func main() {
    var wg sync.WaitGroup

    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()

    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }

    wg.Add(1)
    wg.Wait() // this is for the benefit of the pprof server analysis
}

When you build and run this binary, first visit the web server the code is running.
You’ll find that the pprof data is accessible via the path /debug/pprof/:
http://localhost:6060/debug/pprof/

You should see something like:

profiles:
0    block
4    goroutine
5    heap
0    mutex
7    threadcreate

full goroutine stack dump

/debug/pprof/

Where block, goroutine, heap, mutex and threadcreate are all links off to the recorded data; and by ‘recorded data’ I mean they each link through to a different .profile. This isn’t particularly useful though. A tool is needed to process these .profile data files.

We’ll come back to that in a moment, first let’s understand what these five profiles represent:

block: stack traces that led to blocking on synchronization primitives
goroutine: stack traces of all current goroutines
heap: a sampling of all heap allocations
mutex: stack traces of holders of contended mutexes
threadcreate: stack traces that led to the creation of new OS threads

The web server can also generate a 30 second CPU profile, which you can access via http://localhost:6060/debug/pprof/profile (it won’t be viewable in the browser, instead it’ll be downloaded to your file system).

The CPU profile endpoint is not listed when viewing /debug/pprof/ simply because the CPU profile has a special API, the StartCPUProfile and StopCPUProfile functions that stream output to a writer during profiling, hence this hidden endpoint will ultimately download the results to your file system (we looked at how to use this API in the previous section).

The web server can also generate a “trace” file, which you can access via http://localhost:6060/debug/pprof/trace?seconds=5 (again, it’s not listed for similar reasons as the CPU profile - in that it generates file output that is downloaded to your file system). This trace file out requires the use of go tool trace (which we’ll cover in the next section).

ℹ️ NOTE

more info on pprof options can be found here: golang.org/pkg/net/http/pprof/

If you’re using a custom URL router, you’ll need to register the individual pprof endpoints:

package main

import (
    "net/http"
    "net/http/pprof"
)

func message(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Hello World"))
}

func main() {
    r := http.NewServeMux()
    r.HandleFunc("/", message)

    r.HandleFunc("/debug/pprof/", pprof.Index)
    r.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    r.HandleFunc("/debug/pprof/profile", pprof.Profile)
    r.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    r.HandleFunc("/debug/pprof/trace", pprof.Trace)

    http.ListenAndServe(":8080", r)
}

So ideally you would use go tool pprof on the command line.
As this allows you to more easily interpret and interrogate the data interactively.

To do this, you have to run your binary as before and then, in a separate shell, execute:
go tool pprof http://localhost:6060/debug/pprof/<.profile>

For example, let’s look at the memory heap profile data:
go tool pprof http://localhost:6060/debug/pprof/heap

From here you’ll see an interactive prompt has started up:

Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in /.../pprof.alloc_objects.alloc_space.inuse_objects.inuse_space.005.pb.gz
Type: inuse_space
Time: Oct 27, 2017 at 10:01am (BST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

ℹ️ NOTE

you’ll see the “type” is set to inuse_space (meaning how much memory is still in use)

As shown, you can type either help or o to see what’s available to use.

Here are a couple of useful commands:

top: outputs top entries in text form
topK: where K is a number (e.g. top2 would show top two entries)
list <function regex>: outputs top entries in text form

For example, if I execute top then we’d see the following output:

Showing nodes accounting for 95.38MB, 100% of 95.38MB total
      flat  flat%   sum%        cum   cum%
   95.38MB   100%   100%    95.38MB   100%  main.bigBytes /...ain.go (inline)
         0     0%   100%    95.38MB   100%  main.main /.../profiling/main.go
         0     0%   100%    95.38MB   100%  runtime.main /.../runtime/proc.go

For a simple example application like we’re using, this is fine as it indicates pretty clearly which function is responsible for the majority of the memory allocation (main.bigBytes).

But if I wanted a more specific breakdown of the data I would execute list main.main:

Total: 95.38MB
ROUTINE ======================== main.main in /.../profiling/main.go
   95.38MB    95.38MB (flat, cum)   100% of Total
         .          .      8:   "sync"
         .          .      9:)
         .          .     10:
         .          .     11:// bigBytes allocates 10 sets of 100 megabytes
         .          .     12:func bigBytes() *[]byte {
   95.38MB    95.38MB     13:   s := make([]byte, 100000000)
         .          .     14:   return &s
         .          .     15:}
         .          .     16:
         .          .     17:func main() {
         .          .     18:   fmt.Println("starting...")

This indicates where all our memory is allocated on a “line-by-line” basis.

I noted earlier that the default “type” for the heap analysis was “memory still in use”. But there is an alternative type which indicates the amount of memory that was allocated in total throughout the lifetime of the program. You can switch to that mode using the -alloc_space flag like so:

go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

Let’s see the difference in the output by executing the list command:

(pprof) list main.bigBytes

Total: 954.63MB
ROUTINE ======================== main.bigBytes in /.../go/profiling/main.go
  953.75MB   953.75MB (flat, cum) 99.91% of Total
         .          .      7:   "sync"
         .          .      8:)
         .          .      9:
         .          .     10:// bigBytes allocates 10 sets of 100 megabytes
         .          .     11:func bigBytes() *[]byte {
  953.75MB   953.75MB     12:   s := make([]byte, 100000000)
         .          .     13:   return &s
         .          .     14:}
         .          .     15:
         .          .     16:func main() {
         .          .     17:   var wg sync.WaitGroup

ℹ️ NOTE

if you wanted to be explicit you could have used the “in use” type like so:
go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap

The reason to choose either -inuse_space or -alloc_space will depend on where your specific concerns are focused. For example, if you’re concerned about garbage collection performance then you’ll want to look at the “allocated” memory (i.e. -alloc_space).

ℹ️ NOTE

you can also inspect the number of objects (not just their space) with -inuse_objects and -alloc_objects.

Image Generation

You can also generate an image of your analysis data using either the flag -png, -gif or -svg and then redirecting stdout to a filename like so:

go tool pprof -png http://localhost:6060/debug/pprof/heap > data.png

This generates an image that looks like the following (notice how the bigger the box, the more resources it’s consuming - this helps you ‘at a glance’ to identify a potential problem zone):

ℹ️ NOTE

you can also output as a PDF with -pdf.

Web UI

Finally, there is a interactive web ui coming for pprof in the near future (as of November 2017).

See this post for more details.

But in short you can get the updated pprof tool from GitHub and then execute it with the new flag -http (e.g. -http=:8080).

Trace

Trace is a tool for visualization and analysis of trace data.
It’s suited at finding out what your program is doing over time, not in aggregate.

ℹ️ NOTE

if you want to track down slow functions, or generally find where your program is spending most of its CPU time, then you should consider using go tool pprof instead.

Let’s first modify our application to utilise tracing…

func main() {
    trace.Start(os.Stdout)
    defer trace.Stop()

    for i := 0; i < 10; i++ {
        s := bigBytes()
        if s == nil {
            log.Println("oh noes")
        }
    }

    var wg sync.WaitGroup
    wg.Add(1)

    var result []byte
    go func() {
        result = make([]byte, 500000000)
        log.Println("done here")
        wg.Done()
    }()

    wg.Wait()
    log.Printf("%T", result)
}

So, as far as utilising tracing, all we’ve done is import "runtime/trace" and then added calls to the trace.Start and trace.Stop functions (we defer the trace.Stop in order to ensure we trace everything our application is doing).

Additionally we create a goroutine and allocate a large 500mb slice of bytes. We wait for the goroutine to complete and then we log the type of the result. We’re doing this just so we have some additional spike data to visualise.

Now let’s re-compile our application, generate the trace data and open it with the trace tool…

$ go build -o app
$ time ./app > app.trace
$ go tool trace app.trace

ℹ️ NOTE

you can also generate a pprof compatible file from a trace by using the -pprof flag (if you decided you wanted to dynamically inspect the data that way). See the go documentation for more details.

Here’s the output from running go tool trace app.trace:

2017/10/29 09:30:40 Parsing trace...
2017/10/29 09:30:40 Serializing trace...
2017/10/29 09:30:40 Splitting trace...
2017/10/29 09:30:40 Opening browser

You’ll now see your default web browser should have automatically opened to:
http://127.0.0.1:60331

ℹ️ NOTE

it’s best to use Chrome, as go tool trace is designed to work best with it.

The page that is loaded will show the following list of links:

View trace
Goroutine analysis
Network blocking profile
Synchronization blocking profile
Syscall blocking profile
Scheduler latency profile

Each of these items can give a good insight as to what your application is doing, but we’re most interested in the first one “view trace”. Click it and it’ll give you a complete overview of what your application is doing in an interactive graph.

ℹ️ NOTE

press <Shift-?> to show shortcut keys, like w and s for zooming in/out.

Goroutines

If you zoom in enough on the graph you’ll see the “goroutines” segment is made up of two colours: a light green (runnable goroutines) and a dark green (running goroutines). If you click on that part of the graph you’ll see the details of that sample in the bottom preview of the screen. It’s interesting to see how, at any given moment, there can be multiple goroutines but not all of them are necessarily running all at once.

So in our example you see the program moves between having one goroutine ready to run, but not actually running (e.g. it’s “runnable”) and then we move towards two goroutines running (e.g. they’re both “running” and so there are no goroutines left marked as “runnable”).

It’s also interesting to see the correlation between the number of goroutines running and the number of actual underlying OS threads being utilised (i.e. the number of threads the goroutines are being scheduled on to).

Threads

Again, if you zoom in enough on the graph you’ll see the “threads” segment is made up of two colours: a light purple (syscalls) and a dark purple (running threads).

What’s interesting about the “heap” segment of the UI is that we can see for a short while we never allocate more (total) than 100mb to the heap because the go garbage collection is running concurrently (we can see it running on various processes/threads) and is clearing up after us.

This makes sense because in our code we allocate 100mb of memory and then assign it to a variable s which is scoped to exist only within the for loop block. Once that loop iteration ends the s value isn’t referenced anywhere else so the GC knows it can clean up that memory.

Heap

We can see as the program moves on that we eventually start seeing some contention and so the total allocated heap becomes 200mb then back and forth between 100mb and 200mb (this is because the GC isn’t running all the time). Then near the end we see the 500mb spike that I added to our code as the total allocated amount of memory in the heap shoots to 600mb.

But at that point, if we click on the spike in heap allocation, we can see in the bottom preview window the “NextGC” run indicates that the total allocated should be zero (which makes sense as that’s the end of the program).

Procs

In the “procs” section of the UI we can see during our large 500mb allocation that Proc 3 has a new goroutine running main.main.func1 (which is our go function that’s doing the allocating).

If you were to “View Options” and then tick “Flow events” you’ll see an arrow from the main.main function going towards the main.main.func1 function running on a separate process/thread (probably a bit difficult to see in the below image, but it’s definitely there).

So it’s good to be able to visually see the correlation between first of all the main.main.func1 goroutine running and the memory allocation occurring at that time, but also being able to see the cause and effect (i.e. the flow) of the program (i.e. knowing what exactly triggered the new goroutine to be spun up).

Conclusion

That’s our tour of various tools for profiling your Go code. Take a look at my earlier article on profiling Python for more of the same.

Dev Environments Within Docker Containers

Wed, 29 Mar 2017 00:00:00 +0000

Introduction

You’re a software engineer with a new laptop.
You’re going to be writing code in multiple languages.
You’re going to have projects of varying dependencies.

But you want to avoid the issues you’ve had in the past:

Messy installations of lots of different software packages.
Clogging up your system with programming language version managers.

You decide to use Docker.
It’s a dependency sure, but you’ve got to install something!
With Docker it allows your environment to stay clean.

So let’s see two simple examples:

Python
Go

ℹ️ NOTE

I don’t claim this to be ‘the way’ to do this.
This is just a setup that works well enough for me.
I’m also a Vim user, so your mileage may vary.

Python

You have a Python project you need to work on.

I won’t explain how Python works,
I’ll just presume you’re a Pythonista

Here’s the folder structure we’re dealing with:

foo-project/
| Dockerfile
| Makefile
| app.py
| requirements.txt

We have a Dockerfile (naturally), along with a Makefile to allow us to more easily build our docker image. We also have an application script + a set of dependencies. Nice and simple.

So here’s what the Dockerfile looks like:

FROM python:3.6.1

WORKDIR /tmp

RUN apt-get update -y
RUN apt-get install -y git ncurses-dev
RUN git clone https://github.com/vim/vim.git && cd vim && ./configure --enable-python3interp=yes && make && make install

ADD ./requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

COPY .vim /root/.vim
COPY .vimrc /root/.vimrc

WORKDIR /app

As you can see we’re building our Docker image from an official Python base image (at the time of writing it’s the latest Python version).

We jump into a tmp directory so that we can install some dependencies required to get our setup just how we want it. So this means installing git so we can clone down the latest build of Vim and we also install ncurses-dev which is necessary in order to compile Vim.

After that we copy our requirements.txt file into the image and install the packages specified inside that file. We also add in our .vim directory and .vimrc file to the image.

ℹ️ NOTE

the Makefile has a command for copying .vim/.vimrc into the current project directory, as docker build has a specific context environment that is effectively the location of the Dockerfile

Next we have the Makefile:

copy_vim_files:
        @if [ ! -d "./.vim" ]; then cp -r "$$HOME/.vim" ./.vim; fi
        @if [ ! -f "./.vimrc" ]; then cp "$$HOME/.vimrc" ./.vimrc; fi

remove_vim_files:
        @rm -rf ./.vim
        @rm ./.vimrc

build: copy_vim_files
        @docker build -t python-container-with-vim .

run: build
        @docker run -it -v "$$(pwd)":/app python-container-with-vim /bin/bash

clean: remove_vim_files
        -@docker rmi -f python-container-with-vim &> /dev/null || true

rebuild: clean run

There’s lots going on in there, but the important thing to know is that to start up your docker container (with Python and Vim pre-installed) is to use:

make run

This will copy over the host .vim/.vimrc directory/files, then build a new image and then call docker run ... where it’ll mount the project directory as a volume into the running container.

Once you’re inside the container just execute vim app.py and off you go writing code.

Just for completion, here is our application script:

print('hi')
food = "is a thing"  # if linting is installed properly this will error


def hello(message):
    """
    My summary line starts capitalized and ends with a period.

    my bigger description is going here
    so pay attention to what it says
    """
    print(message)

The above script has some ‘linting’ issues, so if our packages (see below) are installed correctly then we should see Vim highlight the issue to us.

flake8==3.2.1
flake8-deprecated==1.1
flake8-docstrings==1.0.3
flake8-mock==0.3
flake8-quotes==0.9.0
mypy==0.501
pep8-naming==0.4.1
pylint==1.6.4
pytest==3.0.5

We can see from the requirements.txt file that we’ve installed a few different linters along with the MyPy static analysis tool.

That’s it really. You can reuse the Dockerfile and Makefile for all your projects as they don’t do anything specific to this project. Just setup the docker image/container so you can execute make run and start developing.

Go

You have a Go project you need to work on.

I won’t explain how Go works,
I’ll just presume you’re a Gopher

Here’s the folder structure we’re dealing with:

bar-project/
| Dockerfile
| Dockerfile-compile
| Godeps
| Makefile
| app.go

So here’s our main Dockerfile:

FROM golang:1.8

RUN apt-get update -y
RUN apt-get install -y wget git ncurses-dev time

WORKDIR /tmp
RUN git clone https://github.com/vim/vim.git && cd vim && make && make install

WORKDIR /go/src
COPY .vim /root/.vim
COPY .vimrc /root/.vimrc
COPY ./Godeps /go/src

RUN wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && mv gpm /usr/local/bin
RUN gpm install
RUN cp -r ./github.com /github.com  # backup packages to root to prevent volume mount removing it

# Install Go binaries that are utilised by the vim-go plugin:
# https://github.com/fatih/vim-go/blob/master/plugin/go.vim#L9
#
# We don't manually install them, we let vim-go handle that
# We use vim's `execute` command to pipe commands
# This helps avoid "Press ENTER or type command to continue"
RUN time vim -c "execute 'silent GoUpdateBinaries' | execute 'quit'"

Again we’re not doing anything too crazy (not until the end, which I’ll explain). We’re building a new image from an official base image, then we’re installing dependencies that allow us to manually compile the Vim editor.

Next we copy over our vim files and the Godeps dependencies file and we install our dependency manager gpm and install the packages we want to use within our application.

Next we back up the installed depedencies (./github.com) to another directory. The reason we do that is because when we mount our host project directory into the running container we will end up accidentally replacing the installed packages.

Finally we run vim and pass it a script to be executed once Vim has loaded. What this does is allow the statically built image to include the updated set of depedencies that the vim-go plugin requires. I could have installed them manually, but then using the built in command provided by vim-go means I don’t have to ensure my list of go tools still matches up to what vim-go is using.

The downside to this is that when you build the image, you’ll see (for ~20-30 seconds) Vim started and you wont be able to interact with it at all during that time. This is because it’s installing the dependencies it uses. But after that, Vim will close and you’ll be placed at the containers shell prompt. From there you can run Vim and start coding.

Here’s the Go Makefile (it works similarly to the Python one):

bin := "/usr/local/bin/fastly"
vim_dir := "./.vim"
vimrc := "./.vimrc"
container_env := "go-container-with-vim"
container_compiler := "go-compiler"

copy_vim_files:
  @if [ ! -d $(vim_dir) ]; then cp -r "$$HOME/.vim" $(vim_dir); fi
  @if [ ! -f $(vimrc) ]; then cp "$$HOME/.vimrc" $(vimrc); fi

remove_vim_files:
  @rm -rf $(vim_dir) &> /dev/null || true
  @rm $(vimrc) &> /dev/null || true

remove_compiled_files:
  @rm ./fastly.{darwin,linux,windows.exe} &> /dev/null || true

clean: remove_vim_files remove_compiled_files
  @docker rmi -f $(container_env) &> /dev/null || true
  @docker rmi -f $(container_compiler) &> /dev/null || true

uninstall: clean
  @rm $(bin) &> /dev/null || true

build: copy_vim_files
  @docker build -t $(container_env) .

dev: build remove_vim_files
  @docker run -it -v "$$(pwd)":/go/src $(container_env) /bin/bash

rebuild: clean run

compile:
  @docker build -t $(container_compiler) -f ./Dockerfile-compile .
  @docker run -it -v "$$(pwd)":/go/src $(container_compiler) || true

copy_binary:
  cp ./fastly.darwin $(bin)

install: compile copy_binary remove_compiled_files

One thing I did this time was change the make run for make dev as I feel that’s more indicative of what we’re doing (the ‘run’ suggests we’re running our application when we’re really just wanting to drop into a development environment).

There’s a few more steps in the Go Makefile and that’s just for the purposes of having a separate Dockerfile for compiling our application. The following is the other Dockerfile we have in our project:

FROM golang:1.8

WORKDIR /go/src
COPY ./Godeps /go/src
COPY ./compile.sh /go/src

RUN apt-get update && apt-get install wget
RUN wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && mv gpm /usr/local/bin
RUN gpm install
RUN cp -r ./github.com /github.com  # backup packages to root to prevent volume mount removing it

CMD ["./compile.sh"]

This Dockerfile does much the same thing: get dependency file, install those specified dependencies, then back them up to another directory.

This time though, when the container is run we use a separate script as the CMD value as our script was getting quite long (as you’ll see).

Here is the contents of compile.sh:

ℹ️ NOTE

make sure you chmod +x ./compile.sh from your host

#!/bin/sh

# copy packages back into our source code directory
cp -fr /github.com ./github.com

# compile application for the major operating systems
gox -osarch='linux/amd64' -osarch='darwin/amd64' -osarch='windows/amd64' -output='fastly.{{.OS}}'

# run the relevant compatible compiled binary for this container's OS
./fastly.linux

The tasks we run are:

copy the backed up dependencies back into the mounted project directory
build the app using the default compiler for the OS †
execute the compiled binary to show it can run correctly inside the container

† this means our compiled binary will be a linux based binary, so you can’t run it on your host machine if it’s not linux based (e.g. I’m using macOS). You’ll see that to allow me to compiled my application for multiple OS’s I’ve installed Gox

Now here is our Go application, it simply uses the logging dependency we’ve installed and that’s it. Nothing too fancy necessary for this example.

package main

import (
  "fmt"

  log "github.com/Sirupsen/logrus"
)

func init() {
  log.SetLevel(log.DebugLevel) // otherwise would not be shown
}

func main() {
  fmt.Println("Hello World!")

  logger := log.WithFields(log.Fields{
    "name": "hello-world-app",
  })
  logger.Debug("this is my debug log message")
  logger.Info("this is my info log message")
  logger.Warn("this is my warn log message")
  logger.Error("this is my error log message")
  logger.Fatal("this is my Fatal log message")
  logger.Panic("this is my Panic log message") // we don't actually reach here
}

Here is our dependency file’s content, where we can see the Logrus dependency we’ve specified (as well as Gox for the purposes of our container responsible for compiling our application for multiple OS’):

github.com/mitchellh/gox c9740af9c6574448fd48eb30a71f964014c7a837
github.com/sirupsen/logrus 10f801ebc38b33738c9d17d50860f484a0988ff5

Mounting Volumes

Just remember that when making changes inside the container, because you’ve mounted your host project directory as a volume, if you make a change or add a new file or compile something inside of the container; then it’ll be available on your host machine.

This is all fine, but you might want to look at setting up a .gitignore file to ensure you don’t accidentally commit any unwanted items into your git repository.

Key Architecture

Sat, 10 Dec 2016 00:00:00 +0000

Introduction

Just a quick post to cover the key architecture I’m using currently. I’m very interested to know how others are doing things in the hope that I can improve the security of my setup.

ℹ️ NOTE

I’m not a security paranoia nut, so I’m not looking for the most concrete solution. But definitely want to be sure I’m not missing anything obvious either

Visual

Here is a high-level view of what I have currently:

Breakdown

Let’s start at the top…

Laptop

The laptop is password protected and the hard drive is automatically encrypted

Password Data Store

The laptop contains a password data store consisting of GPG keys for every record in the data store.

In order to access the data store (or any of its records), you need a private GPG key.

The data store is backed up to a private online git repository.

Private GPG Key

The private key is itself protected by a secure passphrase (i.e. one that is almost impossible to crack - by today’s standards and recommendations - and I’ve not written it down but memorised it entirely).

The private/public key pair is stored on the laptop, but as the key is protected by a secure passphrase I feel (without going over the top on security) this is as good as I’m probably going to get with it.

The private key is also backed up onto a remote USB drive in case I lose my laptop and I need the key in order to access the data in the password data store.

Private Git Repository

The private git repo has two routes of access. The account itself with the relevant service provider and an SSH key that exists on my laptop to allow it to gain access to the contents of the repository (but not the contents of the files in the data store itself as they are protected by my private GPG key).

The passphrase for both the account and the SSH key are secure (by today’s standards and recommendations) and are stored inside the password data store.

If my laptop was lost, then I’d need to rely on the service provider’s ability to reset my account password via email, where by I could then remove the existing SSH key and replace it with a new one.

I mention replacing the SSH key because I don’t back it up. I feel this SSH key doesn’t need to be a long lasting key (unlike my private GPG key which I intend to keep as safe as possible + the offline backup as a fallback).

Vulnerabilities?

There are a few layers to this architecture and so I’m hoping this makes it harder for a compromise to be effective.

If someone compromises my laptop (i.e. gets access to it), then they can’t access the password data store as they can’t access my private key.

Although a compromise could result in a key logger being installed and record what I type for my passphrase

The only way to get my private key is to locate my remotely stored and safe USB that homes the raw private key contents.

If someone compromises the private git repository, they again can’t do anything with the contents without my private GPG key.

ℹ️ NOTE

when exporting a private key from GPG, it is by default encrypted with its passphrase (it’s not the raw key)

Conclusion

So what do you think? Is this good, bad or just plain terrible?

Please let me know your thoughts on twitter (@integralist)

Hitchhikers Guide to Go

Fri, 02 Dec 2016 00:00:00 +0000

Introduction

A few years ago when I was learning the Go programming language I created a gist and updated it on a regular basis as a sort of cheat sheet. I stumbled across this gist recently and decided I’d try and port it over to some form of semi-coherent blog post.

ℹ️ NOTE

the code will not be updated in any way so YMMV

What this isn’t, is a walk-through of how to write Go code. I’d suggest trying the official Go Tour which is really good and covers a lot of ground. Instead I’m going to provide lots of example code in the vein of a resource like Go by Example.

What this is, is a barren wasteland of old code. Passers by are forewarned to tread carefully.

ℹ️ NOTE

there’s also not much in the way of code explanation, to the extent that some of the testing examples are long and very context specific - you’ve been warned

Most of this Go code is old, so you may find some packages or information possibly out of date (as in not the latest awesome thing) or maybe not that great quality either.

Take this as what it is, a sharing exercise. Take what you need and leave the rest. I wont be offended.

Private Repo Access

go get uses HTTPS so to be able to pull dependencies from a private repository, you’ll need to force it to use SSH so it can access your keys and authorise the connection:

git config --global url."git@github.com:".insteadOf "https://github.com/"

You can also restrict this to a single specific organisation if you prefer:

git config --global url."git@github.com:foo/".insteadOf "https://github.com/foo/"`

So when you want a private dependency like: git@github.com:foo/private.git:

go get github.com/foo/private

Build and Compilation

As of Go 1.5 you can use:

GOOS=darwin GOARCH=386 go build foo.go

Here’s a quick reference of the values you can specify:

$GOOS     $GOARCH
darwin    386      -- 32 bit MacOSX
darwin    amd64    -- 64 bit MacOSX
freebsd   386
freebsd   amd64
linux     386      -- 32 bit Linux
linux     amd64    -- 64 bit Linux
linux     arm      -- RISC Linux
netbsd    386
netbsd    amd64
openbsd   386
openbsd   amd64
plan9     386
windows   386      -- 32 bit Windows
windows   amd64    -- 64 bit Windows

You can get a full list with:

go tool dist list

Gox

Gox is an alternative build tool.

One time only commands for purpose of download/setup:

go get github.com/mitchellh/gox
gox -build-toolchain (only necessary for Go 1.4.x and lower)

Compilation example:

gox -osarch="linux/amd64" -osarch="darwin/amd64" -osarch="windows/amd64" -output="foobar.{{.OS}}"

This will generate three files:

foobar.darwin
foobar.linux
foobar.windows.exe

Other information

Use the -a flag when running go build.

In short, if you dont’ use go build -a -v . then Go won’t know if any packages are missing (you can find the gory details here)

Build Time Dynamic Variables

Imagine you have a global variable called version in your main package and you want to update that value at build time:

go build -ldflags "-X main.version=foobar"

A more realistic example would be to use some form of revision number of commit hash:

go build -ldflags "-X main.version=$(git rev-parse HEAD)"

Another approach would be to have separate files for different ‘environments’ (all under the main package). You would then use a code comment to indicate what the environment was, and at build time you would tell the compiler which version of the file to compile.

// +build prod

package main

func init() {
    version = 123
}

You would compile the above version variable using:

go build -tags prod

Package Naming

In Go, the name of the package is used to refer to the exported item: fmt.Println, http.RegisterFunc etc. Because the package name is so visible, the package name should describe what the exported items are. Meaning, we shouldn’t have packages named util (as a common example of bad package naming) because util.JSONMarshal isn’t as efficient and effective as json.Marshal.

Another example of this that I found in my own code was utils.CreateUser. Later on during the project I had added utils.CreateLegacyUser. When I discovered what I had done I went back and made two separate packages legacy and aws so that I could have a consistent CreateUser function within both (e.g. aws.CreateUser and legacy.CreateUser).

Dependency Information

To see a list of dependencies for a given Go package you can utilise the go list command:

go list -json strconv

Which returns:

{
  "Dir": "/usr/local/Cellar/go/1.5.2/libexec/src/strconv",
  "ImportPath": "strconv",
  "Name": "strconv",
  "Doc": "Package strconv implements conversions to and from string representations of basic data types.",
  "Target": "/usr/local/Cellar/go/1.5.2/libexec/pkg/darwin_amd64/strconv.a",
  "Goroot": true,
  "Standard": true,
  "Root": "/usr/local/Cellar/go/1.5.2/libexec",
  "GoFiles": [
    "atob.go",
    "atof.go",
    "atoi.go",
    "decimal.go",
    "doc.go",
    "extfloat.go",
    "ftoa.go",
    "isprint.go",
    "itoa.go",
    "quote.go"
  ],
  "IgnoredGoFiles": [
    "makeisprint.go"
  ],
  "Imports": [
    "errors",
    "math",
    "unicode/utf8"
  ],
  "Deps": [
    "errors",
    "math",
    "runtime",
    "unicode/utf8",
    "unsafe"
  ],
  "TestGoFiles": [
    "internal_test.go"
  ],
  "XTestGoFiles": [
    "atob_test.go",
    "atof_test.go",
    "atoi_test.go",
    "decimal_test.go",
    "example_test.go",
    "fp_test.go",
    "ftoa_test.go",
    "itoa_test.go",
    "quote_test.go",
    "strconv_test.go"
  ],
  "XTestImports": [
    "bufio",
    "bytes",
    "errors",
    "fmt",
    "log",
    "math",
    "math/rand",
    "os",
    "reflect",
    "runtime",
    "strconv",
    "strings",
    "testing",
    "time",
    "unicode"
  ]
}

If you don’t specify the -json flag then the default behaviour is to filter out the ImportPath field from the above JSON output. For example:

go list strconv

Will return just the import path strconv.

Documentation: go help list | less

You can also utilise Go’s templating functionality on the returned JSON object by adding the -f flag:

go list -f '{{join .Deps " "}}' strconv

Which filters out the Deps field, joins up all items it contains using whitespace and subsequently returns:

errors math runtime unicode/utf8 unsafe

You can do more complex things such as:

go list -f '{{.ImportPath}} -> {{join .Imports " "}}' compress/...

Which will return something like:

compress/bzip2 -> bufio io sort
compress/flate -> bufio fmt io math sort strconv
compress/gzip -> bufio compress/flate errors fmt hash hash/crc32 io time
compress/lzw -> bufio errors fmt io
compress/zlib -> bufio compress/flate errors fmt hash hash/adler32 io

Dependency Management

Update (August 2017): there is an official tool now called dep.

There are many dependency management tools, these are the few that I’ve tried (in this order): godeps, gb, glide. Let’s review each of them to see how they work:

Godeps

When running go get <dependency> locally, Go will stick the dependency in the folder defined by your $GOPATH variable. So when you build your code into a binary using go build <script> it’ll bake the dependencies into the binary (i.e. the binary is statically linked).

But if someone pulls down your repo and tries to do a build from your code, then they’ll need to have a network connection to pull down the dependencies as their $GOPATH might not have those dependencies yet (unless the user manually executes go get for each dependency required). Also the dependencies they subsequently pull down could be a more recent (and untested version) of each dependency.

So to make this situation better we can use Godep to stick all your dependencies within a Godeps folder inside your project directory. You can then use godep save -r ./... to automatically update all your references to point to that local folder.

ℹ️ NOTE

you might need to remove the Godeps folder and run go get if you get strange conflicts. The ./... means to target all .go files

This way users who clone your repo don’t need an internet connection to pull the dependencies, as they already have them. But also they’ll have the correct versions of the dependencies. This acts like a Gemfile.lock as you would typically find in the Ruby world.

Gb

go get -u github.com/constabulary/gb/...
gb vendor fetch <pkg>
gb build all

You’ll need the following structure:

├── src
│   ├── foo
│   │   └── main.go
└── vendor
    ├── manifest
    └── src

The vendor directory is auto-generated by the gb vendor fetch <pkg> command.

Glide

This is now my preferred dependency management tool, as it works just like existing tools in other languages (e.g. Ruby’s Bundler or Node’s NPM) and so consistency is a plus.

It also provides the ability (like gb) to not commit dependencies but have specific versions vendored when running a simple command.

go get github.com/Masterminds/glide
export GO15VENDOREXPERIMENT=1       # or use 1.6
glide init                          # generates glide.yaml
glide install                       # installs from lock file (creates it if not found)
glide update                        # updates dependencies and updates lock file
glide list                          # shows vendored deps
go test $(glide novendor)           # test only your package (not vendored packages)

ℹ️ NOTE

to add a new dependency glide get <pkg_name>

Documentation

Godoc is the original implementation for viewing documentation. Previous to Godoc there was go doc, but that was removed and then added back with totally different functionality.

The syntax structure for go doc is as follows:

go doc <pkg>
go doc <sym>[.<method>]
go doc [<pkg>].<sym>[.<method>]

Here are some examples of using go doc:

go doc json # same as go doc encoding/json
go doc json.Number
go doc json.Number.Float64

Here is the same thing but using godoc (where the syntax structure is godoc <pkg> <symbol>):

godoc encoding/json # unlike "go doc json", "godoc json" doesn't work as it's not a fully qualified path
godoc encoding/json Number
godoc -src builtin make | less

Unlike with go doc, godoc doesn’t allow filtering by <method>
It only goes as far as <pkg> <symbol>

You can use <pkg> <symbol> <method>
and the method will be included in the results
but you’ll need to search for the method manually
godoc -src net/http Request ParseForm | less
here is a similar result using go doc
go doc http.Request.ParseForm | less

The purpose of go doc was to provide a simplistic cli documentation viewer, whereas Godoc has many more features available.

The go doc command also works not only with Go’s own library’s but your own custom packages as well.

There are some differences in what is returned though between godoc and go doc (mainly the latter is more succinct/compact so you can find the functions/types you’re after and then you can expand into those once you’ve found them; godoc is harder to sift through on the command line)…

`godoc encoding/json Encoder`

type Encoder struct {
    // contains filtered or unexported fields
}
    An Encoder writes JSON objects to an output stream.

func NewEncoder(w io.Writer) *Encoder
    NewEncoder returns a new encoder that writes to w.

func (enc *Encoder) Encode(v interface{}) error
    Encode writes the JSON encoding of v to the stream, followed by a
    newline character.

    See the documentation for Marshal for details about the conversion of Go
    values to JSON.

`go doc encoding/json Encoder`

type Encoder struct {
        // Has unexported fields.
}

    An Encoder writes JSON objects to an output stream.

func NewEncoder(w io.Writer) *Encoder
func (enc *Encoder) Encode(v interface{}) error

Notice the functions don’t have their documentation notes printed with go doc

One other thing godoc has over go doc is the ability to view the source code using the -src flag:

godoc -src builtin make | less

The godoc tool also has a full browser documentation suite available and allows you to generate HTML documentation for your project…

Full Browser Documentation

Start a local documentation server and allow indexing (which takes a few minutes; you have to just keep trying the search until it’s done)

godoc -http ':6060' -index

You can then open a new terminal pane and search via cli if you prefer (rather than open up a browser to http://localhost:6060/)

godoc -q tls | less

You can also have the playground available if you need it in the browser, but it does require an internet connection to compile:

godoc -http ':6060' -play

Testing

ℹ️ NOTE

More Test Examples

Faking HTTP and WebServers can be a bit tricky:

package requester

import (
  "bytes"
  "encoding/json"
  "fmt"
  "io/ioutil"
  "net/http"
  "net/http/httptest"
  "os"
  "strconv"
  "testing"
  "time"

  "github.com/bbc/mozart-requester/src/aggregator"
  "github.com/julienschmidt/httprouter"
)

func TestSuccessResponse(t *testing.T) {
  upstream := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      w.Header().Set("Content-Type", "application/json")
      fmt.Fprintln(w, `{"head":[ "foo" ],"bodyInline":"bar","bodyLast":[ "baz" ]}`)
    }
  ))
  defer upstream.Close()

  router := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      Process(w, r, httprouter.Params{})
    }
  ))
  defer router.Close()

  var config = []byte(fmt.Sprintf(`{
    "components":[
      {"id":"foo","endpoint":"%s","must_succeed":true},
      {"id":"bar","endpoint":"%s","must_succeed":true}
    ]
  }`, upstream.URL, upstream.URL))

  req, err := http.NewRequest("POST", router.URL, bytes.NewBuffer(config))

  client := &http.Client{}
  resp, err := client.Do(req)
  if err != nil {
    panic(err)
  }

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  var result aggregator.Result
  json.Unmarshal(body, &result)

  expectedStatus := "success"
  if result.Summary != expectedStatus {
    t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'",
    result.Summary, expectedStatus)
  }

  expectedLength := 2
  if len(result.Components) != expectedLength {
    t.Errorf("The response:\n '%d'\ndidn't match the expectation:\n '%d'", 
    len(result.Components), expectedLength)
  }
}

func TestFailureResponse(t *testing.T) {
  healthyUpstream := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      w.Header().Set("Content-Type", "application/json")
      fmt.Fprintln(w, `{"head":[ "foo" ],"bodyInline":"bar","bodyLast":[ "baz" ]}`)
    }
  ))
  defer healthyUpstream.Close()

  failingUpstream := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      w.Header().Set("Content-Type", "text/plain; charset=utf-8")
      w.WriteHeader(http.StatusNotFound)
      fmt.Fprintln(w, "404 page not found")
    }
  ))
  defer failingUpstream.Close()

  router := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      Process(w, r, httprouter.Params{})
    }
  ))
  defer router.Close()

  var config = []byte(fmt.Sprintf(`{
    "components":[
      {"id":"foo","endpoint":"%s","must_succeed":true},
      {"id":"bar","endpoint":"%s","must_succeed":true}
    ]
  }`, healthyUpstream.URL, failingUpstream.URL))

  req, err := http.NewRequest("POST", router.URL, bytes.NewBuffer(config))

  client := &http.Client{}
  resp, err := client.Do(req)
  if err != nil {
    panic(err)
  }

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  var result aggregator.Result
  json.Unmarshal(body, &result)

  expectedSummary := "failure"
  if result.Summary != expectedSummary {
    t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'", 
    result.Summary, expectedSummary)
  }

  expectedLength := 2
  if len(result.Components) != expectedLength {
    t.Errorf("The response length:\n '%d'\ndidn't match the expectation:\n '%d'", 
    len(result.Components), expectedLength)
  }

  expectedStatus := []int{}
  for _, value := range result.Components {
    if value.Status == 404 {
      expectedStatus = append(expectedStatus, value.Status)
    }
  }
  if len(expectedStatus) < 1 || len(expectedStatus) > 1 {
    t.Errorf("The response length:\n '%d'\ndidn't match the expectation:\n '%d'", 
    len(expectedStatus), 1)
  }
}

func TestSlowResponse(t *testing.T) {
  healthyUpstream := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      w.Header().Set("Content-Type", "application/json")
      fmt.Fprintln(w, `{"head":[ "foo" ],"bodyInline":"bar","bodyLast":[ "baz" ]}`)
    }
  ))
  defer healthyUpstream.Close()

  slowUpstream := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      timeout, err := strconv.Atoi(os.Getenv("COMPONENT_TIMEOUT"))
      if err != nil {
        t.Errorf("COMPONENT_TIMEOUT: %s", err.Error())
      }
      time.Sleep(time.Duration(timeout) * time.Millisecond)
      w.Header().Set("Content-Type", "application/json")
      fmt.Fprintln(w, `{"head":[ "foo" ],"bodyInline":"bar","bodyLast":[ "baz" ]}`)
    }
  ))
  defer slowUpstream.Close()

  router := httptest.NewServer(http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
      Process(w, r, httprouter.Params{})
    }
  ))
  defer router.Close()

  var config = []byte(fmt.Sprintf(`{
    "components":[
      {"id":"foo","endpoint":"%s","must_succeed":true},
      {"id":"bar","endpoint":"%s","must_succeed":true}
    ]
  }`, healthyUpstream.URL, slowUpstream.URL))

  req, err := http.NewRequest("POST", router.URL, bytes.NewBuffer(config))

  client := &http.Client{}
  resp, err := client.Do(req)
  if err != nil {
    panic(err)
  }

  defer resp.Body.Close()
  body, _ := ioutil.ReadAll(resp.Body)

  var result aggregator.Result
  json.Unmarshal(body, &result)

  expectedStatus := 408
  for _, value := range result.Components {
    if value.ID == "bar" && value.Status != expectedStatus {
      t.Errorf("The response:\n '%d'\ndidn't match the expectation:\n '%d'", 
      value.Status, expectedStatus)
    }
  }

  expectedSummary := "failure"
  if result.Summary != expectedSummary {
    t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'", 
    result.Summary, expectedSummary)
  }
}

I typically run my tests using Make, but it ultimately looks like this:

pushd src && APP_ENV=test COMPONENT_TIMEOUT=100 go test -v $(glide novendor) && popd

Here’s another example of a test needing to fake things:

package retriever

import (
  "bytes"
  "io/ioutil"
  "net/http"
  "strings"
  "testing"

  "github.com/PuerkitoBio/goquery"
)

const href = "http://bar.com/"
const url = "http://foo.com/"

var body string

func fakeNewDocument(url string) (*goquery.Document, error) {
  body = strings.Replace(body, "{}", href, 1)

  resp := &http.Response{
    Status:        "200 OK",
    StatusCode:    200,
    Proto:         "HTTP/1.0",
    ProtoMajor:    1,
    ProtoMinor:    0,
    Body:          ioutil.NopCloser(bytes.NewBufferString(body)),
    ContentLength: int64(len(body)),
    Request:       &http.Request{},
  }

  return goquery.NewDocumentFromResponse(resp)
}

func TestRetrieveReturnValue(t *testing.T) {
  // {} interpolated with constant's value
  body = `
    <html>
      <body>
        <div class="productInfo">
          <a href="{}">Bar</a>
        </div>
      </body>
    <html>
  `
  coll, _ := Retrieve(url, fakeNewDocument)

  if response := coll[0]; response != href {
    t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'", 
    response, href)
  }
}

func TestRetrieveMissingAttributeReturnsEmptySlice(t *testing.T) {
  // href attribute is missing from anchor element
  body = `
    <html>
      <body>
        <div class="productInfo">
          <a>Bar</a>
        </div>
      </body>
    <html>
  `
  coll, _ := Retrieve(url, fakeNewDocument)

  if response := coll; len(response) > 0 {
    t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'", 
    response, "[http://bar.com/]")
  }
}

And…

package scraper

import "testing"

func TestScrapeResults(t *testing.T) {
  getItem = func(url string) {
    defer wg.Done()

    ch <- Item{
      "FooTitle",
      "FooSize",
      "10.00",
      "FooDescription",
    }
  }

  urls := []string{
    "http://foo.com/",
    "http://bar.com/",
    "http://baz.com/",
  }

  result := Scrape(urls)
  first := result.Items[0]

  var suite = []struct {
    response string
    expected string
  }{
    {first.Title, "FooTitle"},
    {first.Size, "FooSize"},
    {first.UnitPrice, "10.00"},
    {first.Description, "FooDescription"},
    {result.Total, "30.00"},
  }

  for _, v := range suite {
    if v.response != v.expected {
      err(v.response, v.expected, t)
    }
  }
}

func err(response, expected string, t *testing.T) {
  t.Errorf("The response:\n '%s'\ndidn't match the expectation:\n '%s'", 
  response, expected)
}

Logging

Using the standard Logger:

info := log.New(os.Stdout, "STUFF: ", log.Ldate|log.Ltime|log.Lshortfile)
info.Println("Starting up!!!")

f, e := os.Create("test.log")
if e != nil {
  log.Fatal("Failed to create log file")
}

logfile := log.New(f, "STUFF: ", log.Ldate|log.Ltime|log.Lshortfile)
logfile.Println("Starting up!!!")

Using Logrus:

package main

import (
  "os"

  log "github.com/Sirupsen/logrus"
)

func main() {
  // Standard stdout ASCII logging
  log.WithFields(log.Fields{
    "animal": "walrus",
  }).Info("A walrus appears")

  // JSON style structured logging
  log.SetFormatter(&log.JSONFormatter{})
  f, e := os.Create("logs")
  if e != nil {
    log.Fatal("Failed to create log file")
  }
  log.SetOutput(f)
  log.WithFields(log.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")
  /*
      {
        "animal": "walrus",
        "level": "info",
        "msg": "A group of walrus emerges from the ocean",
        "size": 10,
        "time": "2015-12-22T13:58:46Z"
      }
  */
}

Godo

Godo is a build tool in a similar vein to rake or gulp.

The following example is taken from my own project go-requester:

package main

import (
  "fmt"
  "os"

  do "gopkg.in/godo.v2"
)

func tasks(p *do.Project) {
  if pwd, err := os.Getwd(); err == nil {
    do.Env = fmt.Sprintf("GOPATH=%s/vendor::$GOPATH", pwd)
  }

  p.Task("server", nil, func(c *do.Context) {
    c.Start("main.go ./config/page.yaml", do.M{"$in": "./"})
  }).Src("**/*.go")
}

func main() {
  do.Godo(tasks)
}

Import Race Conditions

When you import a package within a Go script, only the public functions and variables are exposed for the caller to utilise. So if you need a package to execute some bootstrapping code at the point of it being loaded, then you’ll need to stick it inside of an init function.

ℹ️ NOTE

you can have multiple init functions inside a package e.g. one per file within the package namespace

But be careful using init as it can result in a race condition.

I’ve hit an issue where I had something like:

main.go
foo.go (imported by main.go)
bar.go (imported by foo.go)

Each one of these packages had its own init function and ultimately the bar.go’s init function was being run first, followed by the foo.go’s init function and finally followed by the main.go’s init function.

The reason this was an issue was because main.go was loading some environment variables needed by bar.go but those variables weren’t available by the time the bar.go was running as that happened before main.go’s init function had executed.

The solution was to rename all the init functions to Init and explicitly call them to bootstrap the package when needed (i.e. they didn’t automatically bootstrap themselves and so we avoided that race condition).

New vs Make

func new(Type) *Type: allocate memory for custom-user type
func make(Type, size IntegerType) Type: allocate memory for builtin types (Slice, Map, Chan)

package main

import "fmt"

func main() {
  foo := make(map[string]string)
  fmt.Println(foo) // map[]
  foo["k1"] = "bar"
  fmt.Println(foo) // map[k1:bar]
  fmt.Println(foo["k1"]) // bar
  
  type bar [5]int
  b := new(bar)
  fmt.Println(b) // &[0 0 0 0 0]
  b[0] = 1
  fmt.Println(b) // &[1 0 0 0 0]
}

Custom Types

package main

import (
  "bytes"
  "fmt"
)

type path []byte // our custom Type

// method attached to our custom Type
func (p *path) TruncateAtFinalSlash() {
  i := bytes.LastIndex(*p, []byte("/"))

  if i >= 0 {
    *p = (*p)[0:i]
  }
}

func main() {
  pathName := path("/usr/bin/tso") // Conversion from string to path.

  pathName.TruncateAtFinalSlash()

  fmt.Printf("%s\n", pathName)
}

Alternative example:

package main

import "fmt"

type foo [5]int

func main() {
  f := new(foo)
  fmt.Println(f) // &[0 0 0 0 0]
  f[0] = 1
  fmt.Println(f) // &[1 0 0 0 0]
  f.Bar()
  fmt.Println(f) // &[1 2 0 0 0]

  // We can coerce custom types like we can with built-in types
  b := foo([5]int{9, 9, 9})
  fmt.Println(b) // [9 9 9 0 0]
  
  // Check the types
  fmt.Printf("%T\n", b)               // main.foo
  fmt.Printf("%T\n", [5]int{9, 9, 9}) // [5]int
}

func (f *foo) Bar() {
  f[1] = 2
}

Custom Errors

package main

import (
    "errors"
    "fmt"
)

type CustomErr interface {
    error
}

var (
    CustomError = errors.New("Some Custom Error")
)

// we return an 'error' type specifically
// CustomError is indeed an error type, so returning that works
// but also, the interface CustomErr means the CustomError var 
// _is_ a CustomErr type too
func succeed(b bool) (string, error) {
    if b {
        return "success", nil
    }
    return "", CustomError
}

func main() {
    fmt.Println(CustomError) // Some Custom Error

    resp, err := succeed(false)
    if err != nil {
        t, ok := err.(CustomErr)
        if ok {
            fmt.Printf("t is a custom error '%+v' (%T)\n", t, t)
        }
        fmt.Println(err) // Some Custom Error
    }
    fmt.Println(resp) // success (only if succeed(true) is called)
}

Another form of custom error can be seen using a struct:

type errCustom struct {
    message string
    code    int
}

func (e errCustom) Error() string {
    return fmt.Sprintf("error message: %s (code: %d)", e.message, e.code)
}

We would then implement the struct with the specific error context information and return it:

return errCustom{
    message: "whoops",
    code:    500
}

Function Types

package main

import "fmt"

type Foo func(int, string)

func (f Foo) Bar(s string) {
  fmt.Printf("s: %s\n", s)
}

func FooIt(x int, y string) {
  fmt.Printf("x: %d - y: %s\n", x, y)
}

// We HAVE to define the incoming type of "fn"
// Which in this case is a Foo type
func TestIt(fn Foo) {
  fn(99, "problems")
}

// We could do this without defining a func type
// But as you can see, this is a bit ugly
// Plus if we need this function passed around a lot
// then it means a lot of duplicated effort 
// typing the signature over and over
func TestItManually(fn func(int, string)) {
  fn(100, "problems")
}

func main() {
  // Here we're just demonstrating passing around the FooIt function
  // It demonstrates first-class function support in Go
  // But also that we can ensure the function passed around has the expected signature
  TestIt(FooIt)
  TestItManually(FooIt)
  
  x := Foo(FooIt) // Convert our function into a Foo type
  x(0, "hai")     // Now we can execute it as we would FooIt itself
  
  FooIt(1, "bye")
  
  // Notice the types are different
  // FooIt is just a function with a signature (no known type associated with it)
  // Whereas "x" is of known type "Foo"
  fmt.Printf("%T\n", FooIt) // func(int, string)
  fmt.Printf("%T\n", x)     // main.Foo
  
  // But we'll see that the function "x" 
  // which was converted into a Foo type
  // now has access to a Bar method
  // Although FooIt has a matching signature, it's not a Foo type
  // and so it doesn't have a Bar method available
  x.Bar("we have a Bar method")
  
  // We can't even execute:
  // FooIt.Bar("we don't have a Bar method")
  // Because the compiler will stop us
}

Enumerator IOTA

Within a constant declaration, the predeclared identifier iota represents successive untyped integer constants. It is reset to 0 whenever the reserved word const appears in the source.

package main

import "fmt"

const (
  foo = iota // 0
  bar
  _ // skip this value
  baz
)

const (
  beep = iota // 0 (reset)
  boop
)

func main() {
  fmt.Println(foo, bar, baz) // 0 1 3
  fmt.Println(beep, boop)    // 0 1
}

Struct: Var vs Type

A variable of Struct type doesn’t need to be instantiated like a type struct:

package main

import "fmt"

var data struct {
  A string
  B string
}

type data2 struct {
  A string
  B string
}

func main() {
  data.A = "Hai"
  data.B = "Bai"
  
  fmt.Printf(
    "%#v, %+v, %+v", 
    data.A, 
    data.B, 
    data2{A: "abc", B: "def"}
  )
  // "Hai", Bai, {A:abc B:def}
}

Embedded Structs

The first example demonstrates a ‘named’ field utilising an embedded Struct:

package main

import "fmt"

type Point struct {
  X, Y int
}

type Circle struct {
  Center Point // named embeded field
  Radius int
}

type Wheel struct {
  Circle Circle // named embeded field
  Spokes int
}

func main() {
  var w Wheel
  w.Circle.Center.X = 8
  w.Circle.Center.Y = 8
  w.Circle.Radius = 5
  w.Spokes = 20

  fmt.Printf("%+v", w)
}

Which prints:

{Circle:{Center:{X:8 Y:8} Radius:5} Spokes:20}

The second example demonstrates an ‘anonymous’ field instead:

package main

import "fmt"

type Point struct {
  X, Y int
}

type Circle struct {
  Point
  Radius int
}

type Wheel struct {
  Circle
  Spokes int
}

func main() {
  var w Wheel
  w.X = 8       // w.Circle.Point.X
  w.Y = 8       // w.Circle.Point.Y
  w.Radius = 5  // w.Circle.Radius
  w.Spokes = 20

  fmt.Printf("%+v", w)
}

Which prints:

{Circle:{Point:{X:8 Y:8} Radius:5} Spokes:20}

ℹ️ NOTE

anonymous fields don’t work shorthand literal Struct

The following example demonstrates how methods of a composited object can be accessed from the consuming object:

package main

import "fmt"

type Point struct {
  X, Y int
}

func (p Point) foo() {
  fmt.Printf("foo: %+v\n", p)
}

type Circle struct {
  Point
  Radius int
}

type Wheel struct {
  Circle
  Spokes int
}

func main() {
  var w Wheel
  w.X = 8      // w.Circle.Point.X
  w.Y = 8      // w.Circle.Point.Y
  w.foo()      // w.Circle.Point.foo()
  w.Radius = 5 // w.Circle.Radius
  w.Spokes = 20

  fmt.Printf("%+v", w)
}

Which prints:

foo: {X:8 Y:8}
{Circle:{Point:{X:8 Y:8} Radius:5} Spokes:20}

The following example shows how you can mix and match ‘named’ and ‘anonymous’ embeds:

package main

import "fmt"

// Point is ...
type Point struct {
    X, Y int
}

func (p Point) foo() {
    fmt.Printf("foo: %+v\n", p)
}

// Circle is ...
type Circle struct {
    p      Point
    Radius int
}

// Wheel is ...
type Wheel struct {
    Circle
    Spokes int
}

func main() {
    var w Wheel
    w.Spokes = 1
    w.Radius = 2
    w.p.X = 3 // w.Circle.p.X
    w.p.Y = 4 // w.Circle.p.X
    w.p.foo()
    fmt.Printf("%+v", w)
}

Here is a more practical example that demonstrates how embedded functionality can make code more expressive:

package main

import (
  "fmt"
  "sync"
)

// Anonymous struct
var cache = struct {
  sync.Mutex
  mapping map[string]string
}{
  mapping: make(map[string]string), // initial zero value for map
}

func setValue() {
  cache.Lock()
  cache.mapping["foo"] = "bar"
  cache.Unlock()
}

func main() {
  setValue()

  cache.Lock()
  v := cache.mapping["foo"]
  cache.Unlock()

  fmt.Printf("v: %s", v)
}

Reference vs Value

Summary: limit passing by reference unless the size of the copied value is a problem (i.e. memory allocations).

Map data structures are passed by reference, rather than a copied value.

Consider the following example:

package main

import "fmt"

func main() {
  m := make(map[string]int)
  fmt.Println("main before, m = ", m)
  foo(m)
  fmt.Println("main after, m = ", m)
}

func foo(m map[string]int) {
  fmt.Println("foo before, m = ", m)
  m["hai"] = 123
  fmt.Println("foo after, m = ", m)
}

The output:

main before, m =  map[]
foo before, m =  map[]
foo after, m =  map[hai:123]
main after, m =  map[hai:123]

Notice how the map is mutated even once the foo function has finished. This is because a reference to the underlying struct memory location was passed and so the changes made were effective everywhere.

In fact, anything with make is a reference, as well as any explicit interface.

Now consider the next example, which is a slice of integers:

package main

import "fmt"

func main() {
    m := []int{1, 2, 3}
    fmt.Println("main before, m = ", m)
    foo(m)
    fmt.Println("main after, m = ", m)
}

func foo(m []int) {
    fmt.Println("foo before, m = ", m)
    m = append(m, 4)
    fmt.Println("foo after, m = ", m)
}

The output:

main before, m =  [1 2 3]
foo before, m =  [1 2 3]
foo after, m =  [1 2 3 4]
main after, m =  [1 2 3]

Notice how the slice isn’t showing as mutated after the foo function has finished. This is because a copy of the slice was passed to the function for mutating.

In the following example we pass a struct by value (i.e. a copy of the person variable is passed):

package main

import "fmt"

type Person struct {
    firstName string
    lastName  string
}

func changeName(p Person) {
    p.firstName = "Bob"
}

func main() {
    person := Person {
        firstName: "Alice",
        lastName: "Dow",
    }

    changeName(person)

    fmt.Println(person)
}

In order to have that type of change we would need to pass a pointer reference:

package main

import "fmt"

type Person struct {
    firstName string
    lastName  string
}

func changeName(p *Person) {
    p.firstName = "Bob"
}

func main() {
    person := Person {
        firstName: "Alice",
        lastName: "Dow",
    }

    changeName(&person)

    fmt.Println(person)
}

See all methods on `<Type>`

errType := reflect.TypeOf(err)
for i := 0; i < errType.NumMethod(); i++ {
  method := errType.Method(i)
  fmt.Println(method.Name)
}

Convert Struct into JSON

package main

import (
  "encoding/json"
  "fmt"
  "os"
  "time"
)

func main() {
  type Message struct {
    Sequence  int    `json:"sequence"`
    Title     string `json:"title"`
    Timestamp time.Time   `json:"timestamp"`
  }
  msg := Message{1, "Foobar", time.Now()}
  b, err := json.Marshal(msg)
  if err != nil {
    fmt.Println("error:", err)
  }
  os.Stdout.Write(b)
}

Pretty Printing JSON String

package main

import (
  "encoding/json"
  "fmt"
  "os"
)

func main() {
  type ColorGroup struct {
    ID     int
    Name   string
    Colors []string
  }
  group := ColorGroup{
    ID:     1,
    Name:   "Reds",
    Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
  }
  b, err := json.MarshalIndent(group, "", "    ")
  if err != nil {
    fmt.Println("error:", err)
  }
  os.Stdout.Write(b)
}

Convert Struct into YAML

package main

import (
  "fmt"

  "gopkg.in/yaml.v2"
)

type ComponentYaml struct {
  Id  string `yaml:"id"`
  Url string `yaml:"url"`
}

type ComponentsYamlList struct {
  Components []ComponentYaml `yaml:"components"`
}

func main() {
  var y ComponentsYamlList

  yaml.Unmarshal([]byte(
    "components:
    \n  - id: google
    \n    url: http://google.com
    \n  - id: integralist
    \n    url: http://integralist.co.uk"), &y)

  fmt.Println(y)
}

Unknown YAML Structure

package main

import (
  "encoding/json"
  "fmt"
  "os"

  "gopkg.in/yaml.v2"
)

var yml = []byte(`
- key: foo
  value: bar
  secret: false
- key: beep
  value: boop
  secret: true
`)

type Data struct {
  Items []map[string]interface{}
}

func main() {
  y := []map[string]interface{}{}

  if err := yaml.Unmarshal(yml, &y); err == nil {
    fmt.Printf("%#v\n", y)
  } else {
    fmt.Println(err.Error())
  }

  myYaml := Data{Items: y}

  json.NewEncoder(os.Stdout).Encode(myYaml.Items)
}

Sorting Structs

package main

import (
  "fmt"
  "sort"
)

type vals []Value

type Value struct {
  Key string
  Value string
  Secure bool
}

// Satisfy the Sort interface
func (v vals) Len() int      { return len(v) }
func (v vals) Swap(i, j int) { v[i], v[j] = v[j], v[i] }
func (v vals) Less(i, j int) bool { 
  return v[i].Key < v[j].Key 
}

func main() {
  orig := vals{
    {"CK", "BV", false},
    {"DK", "AV", true},
    {"AK", "CV", false},
    {"BK", "DV", true},
  }
  
  fmt.Printf("%+v\n\n", orig)
  sort.Sort(orig)
  fmt.Printf("%+v\n\n", orig)
}

Here is a similar version that sorts by name and age:

package main

import (
  "fmt"
  "sort"
)

type person struct {
  Name string
  Age  int
}

type byName []person

func (p byName) Len() int {
  return len(p)
}
func (p byName) Less(i, j int) bool {
  return p[i].Name < p[j].Name
}
func (p byName) Swap(i, j int) {
  p[i], p[j] = p[j], p[i]
}

type byAge []person

func (p byAge) Len() int {
  return len(p)
}
func (p byAge) Less(i, j int) bool {
  return p[i].Age < p[j].Age
}
func (p byAge) Swap(i, j int) {
  p[i], p[j] = p[j], p[i]
}

func main() {
  kids := []person{
    {"Jill", 9},
    {"Jack", 10},
  }

  sort.Sort(byName(kids))
  fmt.Println(kids)

  sort.Sort(byAge(kids))
  fmt.Println(kids)
}

Which results in:

[{Jack 10} {Jill 9}]
[{Jill 9} {Jack 10}]

Read Users Input

reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter text: ")
text, _ := reader.ReadString('\n')
fmt.Println(text)

HTTP Middleware

This code was modified from @matryer:

package main

import (
  "fmt"
  "log"
  "net/http"
  "os"
)

type data struct {
  Greeting string
  Punct    string
  Who      string
}

func (s data) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  fmt.Fprint(w, s.Greeting, s.Punct, s.Who)
}

type adapter func(http.Handler) http.Handler

func adapt(h http.Handler, adapters ...adapter) http.Handler {
  // Ideally you'd do this in reverse
  // to ensure the order of the middleware
  // matches their specified order
  for _, adapter := range adapters {
    h = adapter(h)
  }
  return h
}

func notify(logger *log.Logger) adapter {
  return func(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      logger.Println("before")
      defer logger.Println("after")
      h.ServeHTTP(w, r)
    })
  }
}

func doSomething() adapter {
  return func(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      fmt.Println("before")
      defer fmt.Println("after")
      h.ServeHTTP(w, r)
    })
  }
}

func main() {
  http.Handle("/hello", &data{"Hello", " ", "Gophers!"})

  logger := log.New(os.Stdout, "server: ", log.Lshortfile)

  http.Handle("/hello-with-middleware", adapt(
    &data{"Hello", " ", "Gophers!"},
    notify(logger), // runs second
    doSomething(), // runs first
  ))

  http.ListenAndServe("localhost:4000", nil)
}

This code will run a web server with two valid endpoints:

/hello
/hello-with-middleware

The client sees the same output but the latter endpoint produces the following stdout output:

before
server: middleware.go:35: before
server: middleware.go:38: after
after

Sessions

package main

import (
  "fmt"
  "net/http"
  "time"
)

const cookiePrefix = "integralist-example-cookie-"

func main() {
  http.HandleFunc("/", login)
  http.HandleFunc("/admin", admin)
  http.HandleFunc("/logout", logout)
  http.ListenAndServe("localhost:4000", nil)
}

func login(w http.ResponseWriter, r *http.Request) {
  if r.Method == "GET" {
    fmt.Fprintf(w, `
<html>
  <body>
    <form method="POST">
      Username: <input type="text" name="username">
      <br />
      Password: <input type="password" name="password">
      <br />
      <input type="submit" value="Login">
  </body>
</html>
`)
  }

  if r.Method == "POST" {
    username := r.FormValue("username")
    password := r.FormValue("password")

    if username == "admin" && password == "password" {
      http.SetCookie(w, &http.Cookie{
        Name:  cookiePrefix + "user",
        Value: username,
      })
      http.Redirect(w, r, "/admin", 302)
    } else {
      fmt.Fprintf(w, `
<html>
  <body>
    Login details were incorrect. Sorry, <a href="/">try again</a>
  </body>
</html>
`)
    }
  }
}

func logout(w http.ResponseWriter, r *http.Request) {
  http.SetCookie(w, &http.Cookie{
    Name:    cookiePrefix + "user",
    Value:   "",
    Expires: time.Now(),
  })

  http.Redirect(w, r, "/", 302)
}

func admin(w http.ResponseWriter, r *http.Request) {
  cookie, err := r.Cookie(cookiePrefix + "user")
  if err != nil {
    http.Redirect(w, r, "/", 401) // Unauthorized
    return
  }

  fmt.Fprintf(w, `
<html>
  <body>
    Logged into admin area as: %s<br><br>
    <a href="/logout">Logout</a>
  </body>
</html>
`, cookie.Value)
}

HTTPS TLS Request

package requester

import (
  "crypto/tls"
  "crypto/x509"
  "flag"
  "io/ioutil"
  "log"
  "net/http"
)

var (
  certFile = flag.String(
    "cert", "/etc/pki/tls/certs/client.crt", "A PEM eoncoded certificate file."
  )
  keyFile  = flag.String(
    "key", "/etc/pki/tls/private/client.key", "A PEM encoded private key file."
  )
  caFile   = flag.String(
    "CA", "/etc/ca/cloud-ca.pem", "A PEM eoncoded CA's certificate file."
  )
)

func SecureClient() *http.Client {
  // Load client cert
  cert, err := tls.LoadX509KeyPair(*certFile, *keyFile)
  if err != nil {
    log.Fatal(err)
  }

  // Load CA cert
  caCert, err := ioutil.ReadFile(*caFile)
  if err != nil {
    log.Fatal(err)
  }
  caCertPool := x509.NewCertPool()
  caCertPool.AppendCertsFromPEM(caCert)

  // Setup HTTPS client
  tlsConfig := &tls.Config{
    Certificates:       []tls.Certificate{cert},
    RootCAs:            caCertPool,
    InsecureSkipVerify: true,
  }
  tlsConfig.BuildNameToCertificate()
  transport := &http.Transport{TLSClientConfig: tlsConfig}
  client := &http.Client{Transport: transport}

  return client
}

And to use it…

client := requester.SecureClient()

// GET
resp, err := client.Get(someEndpoint)

// POST
req, err := http.NewRequest("POST", someEndpoint, bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)

HTTP GET Web Page

package main

import (
  "fmt"
  "io/ioutil"
  "net/http"
  "os"
)

func main() {
  response, err := http.Get("http://www.integralist.co.uk/")
  if err != nil {
    fmt.Println(err.Error())
    os.Exit(1)
  }

  defer response.Body.Close()

  contents, err := ioutil.ReadAll(response.Body)
  if err != nil {
    fmt.Println(err.Error())
    os.Exit(1)
  }

  fmt.Println(string(contents))
}

ℹ️ NOTE

don’t use this ☝️ as it has timeout set to infinity by default

Safer option is to create your own http.Client struct…

package main

import (
    "fmt"
    "io/ioutil"
    "net"
    "net/http"
    "os"
    "sync"
    "time"
)

func defaultClient(wg *sync.WaitGroup) {
    // http.Get calls http.DefaultClient
    // https://golang.org/src/net/http/client.go?s=12086:12134#L359
    // https://golang.org/pkg/net/http/#Client
    response, err := http.Get("https://www.integralist.co.uk/")
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    defer response.Body.Close()

    contents, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    fmt.Println("\ndefaultClient...")
    fmt.Println(response.Status)
    fmt.Println(string(contents[0:10]))

    wg.Done()
}

func customClient(wg *sync.WaitGroup) {
    var netClient = &http.Client{
        Timeout: time.Second * 10,
    }
    response, err := netClient.Get("https://www.integralist.co.uk/")
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    defer response.Body.Close()

    contents, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    fmt.Println("\ncustomClient...")
    fmt.Println(response.Status)
    fmt.Println(string(contents[0:10]))

    wg.Done()
}

func customTransport(wg *sync.WaitGroup) {
    var netTransport = &http.Transport{
        Dial: (&net.Dialer{
            Timeout: 5 * time.Second,
        }).Dial,
        TLSHandshakeTimeout: 5 * time.Second,
    }

    var netClient = &http.Client{
        Timeout:   time.Second * 10,
        Transport: netTransport,
    }

    response, err := netClient.Get("https://www.integralist.co.uk/")
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    defer response.Body.Close()

    contents, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }

    fmt.Println("\ncustomTransport...")
    fmt.Println(response.Status)
    fmt.Println(string(contents[0:10]))

    wg.Done()
}

func main() {
    var wg sync.WaitGroup
    wg.Add(3)
    go defaultClient(&wg)
    go customClient(&wg)
    go customTransport(&wg)
    wg.Wait()
}

Custom HTTP Request Methods

Go doesn’t provide abstractions for all the various HTTP request types, so for things like PUT you have to implement it yourself. The following is an example that creates a secure (TLS/HTTPS) PUT abstraction…

func SecurePut(url, contentType string, configFile io.Reader) (*http.Response, error) {
  client := &http.Client{Transport: configureTLS()}
  req, err := http.NewRequest("PUT", url, configFile)
  if err != nil {
    return nil, err
  }
  req.Header.Add("Content-Type", contentType)
  resp, err := client.Do(req)

  return resp, err
}

func configureTLS() *http.Transport {
  certFilePath := "path/to/cert"
  keyFilePath := "path/to/privateKey"
  caPath := "path/to/ca"

  // Load client cert
  cert, err := tls.LoadX509KeyPair(certFilePath, keyFilePath)
  if err != nil {
    msg := fmt.Sprintf("Error loading developer cert, path: \"%s\"", certFilePath)
    output.Error(msg)
  }

  // Load CA cert
  caCert, err := ioutil.ReadFile(caPath)
  if err != nil {
    msg := fmt.Sprintf("Error loading CA cert, path: \"%s\"", caPath)
    output.Error(msg)
  }
  caCertPool := x509.NewCertPool()
  caCertPool.AppendCertsFromPEM(caCert)

  // Setup HTTPS client
  tlsConfig := &tls.Config{
    Certificates:       []tls.Certificate{cert},
    RootCAs:            caCertPool,
    InsecureSkipVerify: true,
  }
  tlsConfig.BuildNameToCertificate()

  return &http.Transport{TLSClientConfig: tlsConfig}
}

Pointers

package main

import "fmt"

// Point stores co-ordinates
type Point struct {
  x int
  y int
}

// If receiver (Point) isn't set to a pointer (*Point) 
// then the struct's field value won't be updated outside the method
func (p *Point) scaleBy(factor int) {
  fmt.Printf("scaleBy (before modification): %+v\n", p)

  // Don't need to derefence (*) struct fields
  // Compiler will perform an implicit &p for you
  // You only need to dereference in standard functions 
  // when a argument pointer is required (see below Array Pointer example)
  p.x *= factor
  p.y *= factor

  fmt.Printf("scaleBy (after modification): %+v\n", p)
}

func main() {
  // Doesn't matter if we do or don't get the address space (&) for foo/bar's Point
  foo := &Point{1, 2}
  bar := &Point{6, 8}

  fmt.Printf("Main foo.x: %+v\n", foo.x)
  fmt.Printf("Main bar.x: %+v\n", bar.x)

  foo.scaleBy(5)
  bar.scaleBy(5)

  fmt.Printf("Main foo.x: %+v\n", foo.x)
  fmt.Printf("Main foo.y: %+v\n", foo.y)

  fmt.Printf("Main bar.x: %+v\n", bar.x)
  fmt.Printf("Main bar.y: %+v\n", bar.y)
}

ℹ️ NOTE

compiler can only apply implicit dereference for variables and struct fields. This wouldn’t work Point{1, 2}.scaleBy(5)

Results in the following output:

Main foo.x: 1
Main bar.x: 6
scaleBy (before modification): &{x:1 y:2}
scaleBy (after modification): &{x:5 y:10}
scaleBy (before modification): &{x:6 y:8}
scaleBy (after modification): &{x:30 y:40}
Main foo.x: 5
Main foo.y: 10
Main bar.x: 30
Main bar.y: 40

Array Pointer

Deference an Array pointer so you can mutate the original Array values:

package main

import "fmt"

func main() {  
    x := [3]int{1,2,3}

    func(arr *[3]int) {
        (*arr)[0] = 7
        fmt.Println(arr) //prints &[7 2 3]
    }(&x)

    fmt.Println(x) //prints [7 2 3]
}

Alternatively you can utilise a Slice instead of an Array, as the slice ‘header’ already has a ‘pointer’ to an underlying Array:

package main

import "fmt"

func main() {  
    x := []int{1,2,3}

    func(arr []int) {
        arr[0] = 7
        fmt.Println(arr) //prints [7 2 3]
    }(x)

    fmt.Println(x) //prints [7 2 3]
}

Mutating Values

package main

import "fmt"

type Compose string

func (c *Compose) Details() string {
  *c = "beep boop"
  return fmt.Sprintf("Here are your details: %v", *c)
}

func main() {
  var c Compose
  c = "hai"
  fmt.Printf("c: %+v\n", c) // c
  fmt.Println(c.Details())
  fmt.Printf("c: %+v\n", c) // beep boop
}

Type Assertion

if e, ok := err.(net.Error); ok && e.Timeout() {
  //
}

type argError struct {
    arg  int
    prob string
}

func (e *argError) Error() string {
    return fmt.Sprintf("%d - %s", e.arg, e.prob)
}

if ae, ok := e.(*argError); ok {
  //
}

Line Counting

Demonstrates how to use bufio package to scan a file and read it line by line, and then how to increment a map integer value using the shortcut map[key]++. Finally, demonstrates nested maps and ranging over them:

package main

import (
  "bufio"
  "fmt"
  "os"
)

func main() {
  counts := make(map[string]map[string]int)
  files := os.Args[1:]
  if len(files) == 0 {
    countLines(os.Stdin, "n/a", counts)
  } else {
    for _, arg := range files {
      f, err := os.Open(arg)
      if err != nil {
        fmt.Fprintf(os.Stderr, "dup2: %v\n", err)
        continue
      }
      countLines(f, arg, counts)
      f.Close()
    }
  }
  for key, nestedMap := range counts {
    fmt.Printf("Text: %s\n", key)
    for filename, count := range nestedMap {
      fmt.Printf("\tFile: %s\n\tCount: %d\n", filename, count)
    }
    fmt.Println("")
  }
}

func countLines(f *os.File, filename string, counts map[string]map[string]int) {
  input := bufio.NewScanner(f)
  for input.Scan() {
    if val, ok := counts[input.Text()]; ok {
      val[filename]++
    } else {
      inner := make(map[string]int)
      inner[filename]++
      counts[input.Text()] = inner
    }
  }
}

Reading File in Chunks

package main

import (
  "fmt"
  "log"
  "os"
)

func main() {
  // Create file (truncates file if it already exists)
  file, err := os.Create("created.txt")
  if err != nil {
    log.Fatal(err)
  }

  // Populate byte slice with some content
  b := make([]byte, 0)
  for i := 0; i < 5; i++ {
    b = append(b, '!')
    b = append(b, '\n')
    // notice single quotes for Rune rather than double quote for String
  }
  for i := 0; i < 5; i++ {
    b = append(b, '?')
    b = append(b, '\n')
    // notice single quotes for Rune rather than double quote for String
  }
  for i := 0; i < 5; i++ {
    b = append(b, '%')
    b = append(b, '\n')
    // notice single quotes for Rune rather than double quote for String
  }

  // Write file contents
  bytesWritten, err := file.Write(b)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Printf("Bytes written: %+v\n", bytesWritten)

  // Although getting the bytes written was useful for us
  // in this example, you might need to get total bytes
  // which can be done by copying file contents into dev/null
    // io.Copy(ioutil.Discard, resp.Body)

  // Get current offset
  // 1st arg is how much to seek forward/backwards by
  // 2nd arg is relative to different settings
  //         0 == relative to start of file
  //         1 == current offset
  //         2 == relative to end of file
  currentOffset, err := file.Seek(0, 1)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Printf("Current offset: %d\n", currentOffset)
  file.Seek(-currentOffset, 1) // Return to start of file for next Read

  // Read buffered view of file
  // create slice with underlying Array capacity set to total file bytes size
  data := make([]byte, 10, bytesWritten) 
  eof := false
  for !eof {
    count, err := file.Read(data)
    if err != nil {
      eof = true
    }
    fmt.Printf("read %d bytes: %q\n", count, data[:count])
  }
}

Time

now := time.Now()
fmt.Println(now)
expiration := now.Add(time.Hour * 24 * 30)
fmt.Println("Thirty days from now will be : ", expiration)

Here we measure time:

package main

import (
  "fmt"
  "time"
)

// Sleep requires a Duration
// time has set of constants we can use (lowest is 1 Duration)
// Second constant is an abstraction over the other constants
func main() {
  start := time.Now()
  time.Sleep(time.Duration(5) * time.Second) // sleep 5 seconds
  secs := time.Since(start).Seconds()

  fmt.Printf("Time spent: %f seconds", secs)
}

Here is a basic example that pauses execution of a channel until the timer has expired (you would use this over a timer.Sleep because you can cancel a timer before it has expired):

package main

import (
  "fmt"
  "time"
)

func main() {
  timer := time.NewTimer(time.Second * 2)

  <-timer.C // pauses for two seconds

  fmt.Println("Timer expired")
}

Example of cancelling the timer:

package main

import (
  "fmt"
  "time"
)

func main() {
  timer := time.NewTimer(time.Second * 2)

  // Expensive process run in a separate thread
  go func() {
    <-timer.C
    fmt.Println("Timer expired")
  }()

  stop := timer.Stop() // cancel the timer
  fmt.Println(stop)    // true
}

We can do a similar thing with Timers but in a different way:

package main

import (
  "fmt"
  "time"
)

func main() {
  ticker := time.NewTicker(time.Millisecond * 500)

  // Repetitive process
  go func() {
    // Range over the channel rather than pull from it
    for t := range ticker.C {
      fmt.Println("Tick:", t)
    }
  }()

  // Stop ticker after three ticks/intervals
  time.Sleep(time.Millisecond * 1500)
  ticker.Stop()
}

We can combine all these items together with a select statement like so:

package main

import "time"
import "fmt"

func main() {
  timeChan := time.NewTimer(time.Second).C
  tickChan := time.NewTicker(time.Millisecond * 400).C

  // Used to signify we're done with this program
  doneChan := make(chan bool)

  // Sleep for two seconds, then notify the channel we're done
  go func() {
    time.Sleep(time.Second * 2)
    doneChan <- true
  }()

  for {
    select {
    case <-timeChan:
      fmt.Println("Timer expired")
    case <-tickChan:
      fmt.Println("Ticker ticked")
    case <-doneChan:
      fmt.Println("Done")
      return
    }
  }
}

The output of this program would be something like:

Ticker ticked
Ticker ticked
Timer expired
Ticker ticked
Ticker ticked
Done

Starting and Stopping things with Channels

I would imagine that for most cases you’ll want to use a time.NewTimer as seen in previous examples if you want to stop a goroutine that’s processing a long running program. The following example is more for stopping a goroutine that’s running code at a set interval (although using time.NewTicker would probably be more appropriate):

package main

import (
  "fmt"
  "time"
)

func main() {
  quit := make(chan bool)

  // Run a piece of code at a set interval
  go func() {
    for {
      select {
      case <-quit:
        return
      default:
        fmt.Println("Not ready to stop this goroutine")
        time.Sleep(time.Millisecond * 100)
      }
    }
  }()

  // Do other stuff for two seconds
  time.Sleep(time.Second * 2)

  // Quit goroutine
  quit <- true

  fmt.Println("Goroutine was stopped")
}

Starting a goroutine:

package main

import (
  "fmt"
  "time"
)

func main() {
  // Use a struct type channel as it clarifies your intent
  // Which is this channel is used for 'signalling'
  start := make(chan struct{})

  for i := 0; i < 10000; i++ {
    go func() {
      <-start // wait for the start channel to be closed
      fmt.Println("do stuff")
    }()
  }

  // at this point, all goroutines are ready to go
  // we just need to tell them to start by
  // closing the start channel
  close(start)

  fmt.Println("Let's pause briefly to give goroutines time to execute")

  time.Sleep(time.Millisecond * 10)
}

Stopping a goroutine:

package main

import (
  "fmt"
  "time"
)

func main() {
  // Use a struct type channel as it clarifies your intent
  // Which is this channel is used for 'signalling'
  done := make(chan struct{})

  // Long running process put onto a thread
  go func() {
    fmt.Println("Inside thread doing expensive processing")
    time.Sleep(time.Second * 5)
    close(done)
  }()

  fmt.Println("Do other things")

  // Wait for long running process to finish
  <-done

  fmt.Println("Do more things")
}

Channel Pipelines

The principle of a pipeline, is to take data from one function and pass it into another function, that receiving function will process the received data and then that result is returned and subsequently passed onto another function… rinse and repeat for however long your pipeline needs to be.

In the below example (copied from here) demonstrates how a set of functions accept a channel and return a channel and so channels is the ‘data’ that is passed around the pipeline functions:

package main

import "fmt"
import "sync"

func ingest() <-chan []string {
  out := make(chan []string)
  go func() {
    out <- []string{"aaaa", "bbb"}
    out <- []string{"cccccc", "dddddd"}
    out <- []string{"e", "fffff", "g"}
    close(out)
  }()
  return out
}

func process(concurrency int, in <-chan []string) <-chan int {
  var wg sync.WaitGroup
  wg.Add(concurrency)

  out := make(chan int)

  work := func() {
    for data := range in {
      for _, word := range data {
        out <- len(word)
      }
    }
    wg.Done()

  }

  go func() {
    for i := 0; i < concurrency; i++ {
      go work()
    }

  }()

  go func() {
    wg.Wait()
    close(out)
  }()
  return out
}

func store(in <-chan int) <-chan struct{} {
  done := make(chan struct{})
  go func() {
    defer close(done)
    for data := range in {
      fmt.Println(data)
    }
  }()
  return done
}

func main() {
  // stage 1 ingest data from source
  in := ingest()

  // stage 2 - process data
  reduced := process(4, in)

  // stage 3 - store
  <-store(reduced)
}

Templating

Here is a basic program that uses a Struct for its data source:

package main

import (
  "log"
  "os"
  "text/template"
)

type dataSource struct {
  Baz int
}

func (ds dataSource) Foo() string {
  return "I am foo"
}

func (ds dataSource) Bar() string {
  return "I am bar"
}

const templ = `
  Foo: {{.Foo}}
  Piping: {{.Bar | printf "Bar: %s"}}
  Function: {{.Baz | qux}}
`

func qux(baz int) int {
  return baz * 2
}

// template.Must handles parsing errors better
var setupTemplate = template.Must(
  template.New("whatever").
    Funcs(template.FuncMap{"qux": qux}).
    Parse(templ),
)

func main() {
  ds := dataSource{5}

  if err := setupTemplate.Execute(os.Stdout, ds); err != nil {
    log.Fatal(err)
  }
}

ℹ️ NOTE

printf is a built-in function for templating and is functionally equivalent to fmt.Sprintf

Program output:

Foo: I am foo
Piping: Bar: I am bar
Function: 10

Here is a HTML templating version:

package main

import (
  "html/template"
  "log"
  "os"
)

var data struct {
  A string        // untrusted plain text
  B template.HTML // trusted HTML
}

const templ = `<p>A: {{.A}}</p><p>B: {{.B}}</p>`

func main() {
  t := template.Must(template.New("escape").Parse(templ))

  data.A = "<b>Hello</b>"
  data.B = "<b>Hello</b>"

  if err := t.Execute(os.Stdout, data); err != nil {
    log.Fatal(err)
  }
}

The output would be:

<p>A: &lt;b&gt;Hello&lt;/b&gt;</p>
<p>B: <b>Hello</b></p>

Error handling

The following code outputs:

This is our custom error with some more context prefixed: oh noes!

Here’s the code:

package main

import (
  "errors"
  "fmt"
)

type errWithContext struct {
  err error
  msg string
}

func (e errWithContext) Error() string {
  return e.msg + ": " + e.err.Error()
}

func triggerError() (bool, error) {
  return false, errors.New("oh noes!")
}

func main() {
  var e *errWithContext

  _, err := triggerError()
  if err != nil {
    e = &errWithContext{
      err,
      "This is our custom error with some more context prefixed",
    }
  }

  fmt.Print(e.Error())
}

Socket Programming

There are two main types of sockets:

STREAM sockets (e.g. TCP)
DATAGRAM sockets (e.g. UDP)

ℹ️ NOTE

a “unix domain socket” is actually a physical file
it’s useful for local (same host) data communication

The principle steps behind a socket is:

Create the socket
Bind the socket to an address (e.g. 127.0.0.1:80)
Listen for socket connections
Accept the socket connection

There are two main packages in our below example: server.go and client.go.

Run both of them in separate terminals (e.g. go run ...)

Then for the client.go type your message followed by a new line, for example:

Hello World
Message from server: HELLO WORLD

Whilst in the server.go terminal you should see:

Starting TCP server...
Message Received: Hello World

The code for this program is as follows:

server.go

package main

import (
  "bufio"
  "fmt"
  "net"
  "strings"
)

func main() {
  fmt.Println("Starting TCP server...")

  // Listen on all network interfaces (e.g. 0.0.0.0)
  // Documentation: godoc net Listener | less
  listener, _ := net.Listen("tcp", ":8081")

  // Accept connection on the port we specified (see above)
  connection, _ := listener.Accept()

  // Handle incoming connections forever
  for {
    // Listen for message to process ending in newline (\n)
    // Note: single quotes needed for type byte (double quotes is a string)
    message, _ := bufio.NewReader(connection).ReadString('\n')

    // Output message received
    fmt.Println("Message Received:", string(message))

    // Do something with the message (e.g. uppercase it)
    newmessage := strings.ToUpper(message)

    // Send new string back to client
    connection.Write([]byte(newmessage + "\n"))
  }
}

client.go

package main

import (
  "bufio"
  "fmt"
  "net"
  "os"
)

func main() {
  // Open socket connection to a locally runnning TCP server
  connection, _ := net.Dial("tcp", "127.0.0.1:8081")

  // Handle incoming responses
  for {
    // Read the input
    reader := bufio.NewReader(os.Stdin)

    // Message to be sent
    // Note: single quotes needed for type byte (double quotes is a string)
    // Documentation: godoc bufio ReadString | less
    // ReadString reads until the first occurrence of the delimiter \n in the input
    text, _ := reader.ReadString('\n')

    // Send message to open Socket
    fmt.Fprintf(connection, text+"\n")

    // Listen for response
    // Note: single quotes needed for type byte (double quotes is a string)
    message, _ := bufio.NewReader(connection).ReadString('\n')

    fmt.Println("Message from server: " + message)
  }
}

Comparing Maps

This code demonstrates how to be careful about false positives!

package main

import "fmt"

func equal(x, y map[string]int) bool {
  if len(x) != len(y) {
    // fail fast
    return false
  }

  for k, xv := range x {
    // Verify "missing" key and "present but zero" key value
    if yv, ok := y[k]; !ok || yv != xv {
      return false
    }
    
    /*
    // The following condition would incorrectly return "true" for the below example comparison!
    // This is because the empty value for an int type is a zero, while the actual value of x's key is zero
    if xv != y[k] {
      return false
    }
    */
  }

  return true
}

func main() {
  fmt.Println(
    equal(map[string]int{"A": 0}, map[string]int{"B": 42}),
  )
}

Zip File Contents

package main

import (
  "compress/zlib"
  "io"
  "log"
  "os"
)

func main() {
  var err error

  // This defends against an error preventing `defer` from being called
  // As log.Fatal otherwise calls `os.Exit`
  defer func() {
    if err != nil {
      log.Fatalln("\nDeferred log: \n", err)
    }
  }()

  src, err := os.Create("source.txt")
  if err != nil {
    return
  }
  src.WriteString("source content")
  src.Close()

  dest, err := os.Create("new.txt")
  if err != nil {
    return
  }

  openSrc, err := os.Open("source.txt")
  if err != nil {
    return
  }

  zdest := zlib.NewWriter(dest)
  if _, err := io.Copy(zdest, openSrc); err != nil {
    return
  }

  // Close these explicitly
  zdest.Close()
  dest.Close()

  n, err := os.Open("new.txt")
  if err != nil {
    return
  }

  r, err := zlib.NewReader(n)
  if err != nil {
    return
  }
  defer r.Close()
  io.Copy(os.Stdout, r)

  err = os.Remove("source.txt")
  if err != nil {
    return
  }

  err = os.Remove("new.txt")
  if err != nil {
    return
  }
}

Shell Commands

Here is a simple example that writes the output of a command to a file:

package main

import (
  "os"
  "os/exec"
)

func main() {
  cmd := exec.Command("ls")

  outfile, err := os.Create("./out.txt")
  if err != nil {
    panic(err)
  }
  defer outfile.Close()

  cmd.Stdout = outfile
  cmd.Stderr = outfile

  err = cmd.Start()
  if err != nil {
    panic(err)
  }

  cmd.Wait()
}

Here is an older example that printed the results of a command:

var (
  cmdOut []byte
  err    error
)
cmdName := "spurious"
cmdArgs := []string{"ports", "--json"}
if cmdOut, err = exec.Command(cmdName, cmdArgs...).Output(); err != nil {
  fmt.Fprintln(os.Stderr, "There was an error running 'spurious ports --json' command: ", err)
  os.Exit(1)
}
fmt.Println(string(cmdOut))

New Instance Idiom

package main

import "fmt"

type Sqs struct {
  foo string
}

func (s *Sqs) create() {
  fmt.Println("I'll create stuff")
}

func NewSqs() *Sqs {
  return &Sqs{"bop"}
}

func main() {
  s := NewSqs()
  fmt.Println(s.foo)
  s.create()
}

JSON Connection Draining

When using json.NewDecoder:

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        var u User
        if r.Body == nil {
            http.Error(w, "Please send a request body", 400)
            return
        }
        err := json.NewDecoder(r.Body).Decode(&u)
        if err != nil {
            http.Error(w, err.Error(), 400)
            return
        }
        fmt.Println(u.Id)
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

…it doesn’t read the response Body completely. So when closing the response you might get an error as a stray \n could be present later on. You’ll need to drain the response instead:

defer func() {
  io.CopyN(ioutil.Discard, r.Body, 512)
  r.Body.Close()
}()

ℹ️ NOTE

https://github.com/google/go-github/pull/317

Writing your own Marshal/Unmarshal functions

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "strconv"
)

func main() {
    a := make(Stuff)
    a[1] = "asdf"
    a[-1] = "qwer"
    fmt.Println("Initial:     ", a)

    stuff, _ := json.Marshal(a)
    fmt.Println("Serialized:  ", string(stuff))

    b := make(Stuff)
    _ = json.Unmarshal(stuff, &b)
    fmt.Println("Deserialized:", b)
}

type Stuff map[int]string

// MarshalJSON works the same as the underlying json.Marshal would
func (this Stuff) MarshalJSON() ([]byte, error) {
    buffer := bytes.NewBufferString("{")
    length := len(this)
    count := 0
    for key, value := range this {
        jsonValue, err := json.Marshal(value)
        if err != nil {
            return nil, err
        }
        buffer.WriteString(fmt.Sprintf("\"%d\":%s", key, string(jsonValue)))
        count++
        if count < length {
            buffer.WriteString(",")
        }
    }
    buffer.WriteString("}")
    return buffer.Bytes(), nil

  // for example you could totally change the behaviour...
  /*
    buffer := bytes.NewBufferString("{")
    buffer.WriteString("}")
    return buffer.Bytes(), nil
    */
}

// UnmarshalJSON works the same as the underlying json.Unmarshal would
func (this Stuff) UnmarshalJSON(b []byte) error {
    var stuff map[string]string
    err := json.Unmarshal(b, &stuff)
    if err != nil {
        return err
    }
    for key, value := range stuff {
        numericKey, err := strconv.Atoi(key)
        if err != nil {
            return err
        }
        this[numericKey] = value
    }
    return nil
}

This means you can also leverage custom type checking:

See: leveraging interfaces in golang

Concepts From the C Programming Language

Mon, 28 Nov 2016 00:00:00 +0000

Introduction

I decided recently to read a book on the C programming language. The idea was to learn some more low-level concepts that other higher-level languages were abstracting away from me.

This write up is the result of some of those learnings. This is not a “how do you write C code”. This is a grouping of topics and concepts that I found interesting and thought might be useful to other developers with similar interests.

Some of the stuff covered will be obvious, but I appreciate some concepts won’t be obvious for all readers, so this write up will end up crossing back and forth between different ‘levels’ of understanding and experience.

In other words: your mileage may vary…

Compilation

When writing a program in a language like C, you’ll find that by itself it is not executable (i.e. you can’t run a C file directly). You need to convert the C source code into machine code (i.e. something the computer’s CPU can understand).

Machine code is as low-level as you can get when interacting with a computer. So the C language is considered a ‘higher-level’ abstraction to save us from having to write machine code ourselves.

A language like Python is an even ‘higher-level’ abstraction, to save us from having to write C

ℹ️ NOTE

the Python language is actually written in C
Although there are other Python interpreters implemented in different languages

In order to convert C code into machine code, we need a compiler.

Strictly speaking you also need a linker which takes multiple compiled objects and places them into a single executable file. Generally speaking, when we say “compile a C file”, we’re really combining two separate steps (compiling and linking) into the single generic term “compile”

Compilers

To compile C source code into an executable you need a compiler, of which there are many options. The two most popular being LLVM’s clang and GNU’s gcc. You might also find on your computer a cc command, but typically this is aliased to an existing compiler.

The Mac OS doesn’t provide a compiler by default. But if you install X-Code you’ll get the LLVM’s suite of compilers. Below we see that we get quite a few alias’ and all of them point to the same embeded LLVM compiler:

$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
                                                                   
$ llvm-gcc --version
Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
                                                                   
$ clang --version
Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
                                                                   
$ cc --version
Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

The first two alias’ gcc and llvm-gcc are a little bit confusing and also a bit misleading, as they’re not GNU’s version. They’re still the LLVM’s compiler but with some modifications. In the first instance (gcc) the compiler is configured to use some additional libraries that are provided by c++.

You can tell this by the flag --with-gxx-include-dir

It’s worth noting that with a standard/simple C source code file, all these alias’ work to compile the source code into an executable. But some compilers allow you to utilise additional extensions not provided by the standard C language (so you need to be careful your code doesn’t try to utilise something that’s not available at compilation time).

LLVM’s licensing is BSD, meaning Apple can embed it within their own software that is not GPL-licensed. Typically LLVM’s compiler is faster than GNU’s, but in some cases it might not support all the same targets as GNU’s.

For more comparison details see http://clang.llvm.org/comparison.html

C11 safe functions?

You’ll likely be told that some functions provided within C aren’t safe (usually around string manipulation). For example, some string functions allow for overflow of data because they don’t check that the underlying array data structure is able to contain the strings being manipulated.

C11 compatible compilers will provide an additional set of string functions that are considered safe (although this is a contentious area of discussion as some C programmers believe that these functions are just as unsafe and if anything the claims are misleading! See this comment on Stack Overflow which goes into more detail).

ℹ️ NOTE

I’ve also discovered that none of the compilers I have on my OS support these functions, and I’ve since discovered one of the few environments to support these functions is the Windows platform

Below is an example C file that demonstrates how to check if your compiler supports these additional safe functions:

#include <stdio.h>

int main(void)
{
  #if defined __STDC_LIB_EXT1__
    printf("Optional functions are defined.\n");
  #else
    printf("Optional functions are not defined.\n");
  #endif
  
  return 0;
}

If your compiler supports these optional (safe) string functions, then to enable them you’ll need to add a define directive that modifies the subsequent header file. But you also need to add this directive before you include the preprocessor directive that imports the string.h header:

#define __STDC_WANT_LIB_EXT1__ 1
#include <string.h>

If you don’t set __STDC_WANT_LIB_EXT1__ to 1, then the header string.h will utilise the old (unsafe) string functions.

Hello World

Below is a simple ‘hello world’ C example:

#include <stdio.h>   // pre-processor directive to include code file at compile time
#define NAME "World" // pre-processor directive to substitute any reference to NAME _before_ compilation

// returns an int type and takes in no arguments (void)
int main(void) {
  printf("hello %s", NAME); 
  return 0;          // zero indicates no problems (i.e. EXIT CODE)
}

It’s important to note that the directives #include and #define are ‘processed’ at the start of the compilation process. This is at the request of the compiler.

So it’ll be one of the compiler’s first steps to pull in the preprocessor and have it ensure the file is setup ready for the rest of the compilation.

You would compile this file like so:

cc hello-world.c -o hw

Now you have a macOS compatible executable:

./hw # prints the message "Hello World"

To cross-compile for another OS (e.g. Linux) then use Docker or a VM
Other modern languages like Go or Rust allow you to cross-compiler without a VM

Constants vs Directives

We saw in the above ‘Hello World’ example the use of the directive #define which allowed us to use a single identifier (NAME in this case) throughout our program. The benefit is that we can change the value once and have it updated everywhere.

But do not get this confused with a variable. It is not. This is just a sequence of characters that are blindly replaced at the pre-processing stage. The value assigned to NAME will be replaced inside your program regardless of whether it’s valid code or not. Meaning it could cause an ambiguous compiler error.

On the other hand you can define a proper constant like so:

#include <stdio.h>

int main(void) {
  const char NAME[] = "World";
  printf("Hello %s", NAME);
  return 0;
}

What this gives you is a variable that has an actual type assigned to it (char). Meaning the compiler will help you identify an incorrect value if necessary, much more easily than using the #define directive.

Quotations

In C single quotes denote a char type and double quotes denote a string.

So if you had the following code:

char foo = 'a';
printf("foo: %s\n", foo);

It would error with:

format specifies type 'char *' but the argument has type 'char'

To get it to work you need to provide the memory address location of foo using the address-of operator (&):

char foo = 'a';
printf("foo: %s\n", &foo);

We’ll come back to the & operator (and understand what * means) later, when we discuss pointers.

Char Type

When creating a variable, and assigning a string to it, the value assigned is really a pointer to a location in memory. The char type is used when storing characters such as 'a', but it also allows storing of strings such as "abc".

When assigning a string, the pointer is to an array where each element of the array is a character of the provided string. For example the string "abc" would be stored in an array that looked something like:

["a", "b", "c"]

ℹ️ NOTE

see section ‘Null Terminator’ to clarify above code

This happens even if the string you provide is just one character. Although, depending on your program’s design, it could be argued that you should not have assigned a single character string, but instead used single quotes to represent a single char.

When assigning a character (e.g. a) to a variable of type char it takes on dual duty. Meaning the char type variable can represent the specific character a but really it stores the ASCII integer code that defines that character.

Take a look at an ASCII table to identify the code associated with a particular character

This means we could also directly assign the integer 97 instead of the character a to the char type variable. But also, and more interestingly, because of these characteristics we can perform arithmetic on the variable:

#include <stdio.h>

int main(void) {
  // print character and its associated ascii code integer
  char foo = 'a';
  printf("foo: %c (%d)\n", foo, foo); // a (97)

  // now modify the variable by adding to it
  foo = foo + 2;

  // we can see the character/integer reflects the change
  printf("foo: %c (%d)\n", foo, foo); // c (99)
  return 0;
}

Null Terminator

Consider the following code:

char my_string[4] = "abc";

The reason we specify a length of 4 and not 3 (as you would expect with a string that is three characters in length), is because the underlying array that my_string is being pointed towards looks like this:

["a", "b", "c", "\0"] // yes it does actually have four elements

The last element is known as the null terminator. When this data is stored in memory, we can start at the location in memory (the address) where the first element is stored and then step through memory until we reach the null terminator; where we’ll then find the end of the string.

ℹ️ NOTE

you can set your variable to be the actual length of the content (e.g. char my_string[1] = "a";) but in some instances this can cause strange overlaps of data and strictly speaking isn’t valid code either

Pointers

When declaring a variable, the computer sets aside some memory for the variable.

Next, the variable name is linked to the location in memory that was set aside for it.

Lastly, the value you want to assign to the variable is placed into the relevant location of memory.

Let’s consider the following code:

#include <stdio.h>

int main(void) {
  int foo = 1;
  printf("foo: %d\n", foo);

  int *bar;
  int bar_val = 1;
  printf("bar_val: %d\n", bar_val);
  bar = &bar_val;
  printf("bar: %p\n", bar);
  int bar_get_val = *bar;
  printf("bar_get_val: %d\n", bar_get_val);

  return 0;
}

So we see that we create a foo variable and assign 1 to it. We then print that integer in the typical way.

Next, we make a slightly more convoluted version, but this time we’re utilising a pointer in order to help us understand what they are and how to use them.

Here are each of the steps broken down:

int *bar;: we declare a pointer variable † called bar of type int but we don’t initialize it with a value
int bar_val = 1;: we both declare and initialize the variable bar_val as type int
bar = &bar_val;: we initialize the pointer variable bar with the memory address of bar_val
int bar_get_val = *bar;: we dereference the address (i.e. follow the pointer) assigned to bar which leads us to the value stored in that memory slot

† meaning we will be assigning an address to this pointer
and the content at that memory address location will also be of type int

The output of this program is:

foo: 1
bar_val: 1
bar: 0x7fff59a1769c
bar_get_val: 1

OK, so there are some things that we need to clarify and that’s the * and & operators.

*: value-at-address operator (used when declaring a pointer and when dereferencing a pointer)
&: address-of operator (used to reference the memory address of a variable)

The first thing we should be aware of is that we’re not able to print a declared variable that has no value initialized for it. So imagine the following code:

int *bar;
printf("bar: %d\n", bar);

This would cause the following compiler error:

format specifies type 'int' but the argument has type 'int *'

Which makes sense, as we’ve declared the variable as the type *bar. So we can start by fixing that issue and correctly specifying the second argument to printf as *bar:

int *bar;
printf("bar: %d\n", *bar);

ℹ️ NOTE

printf will now try to use * to dereference the value from the variable bar

Unfortunately this code still causes an error. This time:

variable 'bar' is uninitialized when used here

Which again makes sense. Nothing more to say about that portion of the code, I just wanted to make it clear what happens when you try to print an uninitialized variable (and also what happens when that variable is a pointer type).

So continuing through the program, the next line of interest is:

bar = &bar_val;

This gives us the actual location in memory for the variable bar_val (i.e. 0x7fff59a1769c). So the value assigned to bar isn’t 1, but the address of 1 in memory.

Finally, we declare and initialize the variable bar_get_val with the actual value of 1, and we do that by using * to deference the variable bar which contains a memory address:

int bar_get_val = *bar;

What that means, is bar holds a memory address, which isn’t a concrete value, it’s an indirection to somewhere else. Hence we would say bar points to the actual value’s location and explains why we use the ‘value-at-address’ operator * to deference the value.

The following code shows how to print the location in memory of a variable even if it wasn’t declared as a pointer, simply by using the address-at operator & which itself indicates a pointer to another location:

char foo = 'a';
printf("address of foo: %p\n", &foo);

ℹ️ NOTE

the & isn’t tied to * in any way.
Its purpose is just to return the memory address for a given variable

Remember, a memory address isn’t the value itself but a reference to where the value can be found. One analogy I’ve seen is of your home address on an envelope: the envelope isn’t your home, nor is the address written on the envelope. The envelope just indicates where your home can be found.

One last thing to consider/remember is that C doesn’t have a String type. It stores strings in an array data structure. An array will automatically return the address location of its first element to the variable it is assigned to.

This is why you may have seen a char pointer being assigned a variable without the need to use the & operator to get the memory address of that variable (because the variable, in this case an array, already provides a memory address).

The following example shows this:

char message[6] = "hello"; // array data structure used and memory location for message[0] returned
char *messagePtr = message; // no need to use &message now
printf("my pointer: %p\n", messagePtr);
printf("my message: %s\n", message);

ℹ️ NOTE

compare C pointers and Go pointers here https://dave.cheney.net/2014/03/17/pointers-in-go

In C there are two ways to define a pointer:

char* foo
char *foo

Both of which are equivalent.

Although the first seems like the clearer option (as someone new to C would read it: “define a variable called foo of type ‘character pointer’”) compared to the second option (which could lead them to think the variable name was *foo not foo). For me the second option is preferred because otherwise the following code becomes a bit ambiguious:

char* foo, bar;

You might (incorrectly) think this would create two variables, both of type ‘character pointer’, but really only foo is the pointer and bar is a normal char type.

Whereas using the second format (char *foo), this code becomes much clearer:

char *foo, bar;

Lastly, if you want to create a const that happens to be a pointer, then the syntax is as follows:

int count = 43;
int *const pcount = &count;

We prefix the const keyword with the value-at operator * and not the variable name.

Arrays

Consider the following code (which is broken by the way):

#include <stdio.h>

int main(void) {
  char my_string = "abc";
  printf("my_string: %s", my_string);

  return 0;
}

This code has the following compiler error:

incompatible pointer to integer conversion initializing 'char' with an expression of type 'char [4]'

What this error tells us is that the variable my_string has a type of char [4], meaning it is actually an array (hence the [4] syntax) and so we should have declared the variable like so:

char my_string[4] = "abc";

We saw earlier why this is required, when talking about null terminators. But just to recap, it’s because a string should be stored within an array data structure. So we need to declare it as such.

We also learned earlier (using the above example) why the length of the array is 4 and not 3 (which you may initially have expected which a string of three characters), again to recap: this is because of the extra element added to the array for you “\0” (the null terminator).

So, the reason I’m talking about arrays is because in the original code above (the one before declaring the variable correctly) there were actually two errors linked together. The second part of the error was:

format specifies type 'char *' but the argument has type 'char'

What this error tells us is that printf was expecting a pointer but all it got was something of a char type.

When declaring the variable as an array, we fix both errors.

But both these errors has led some people to incorrectly assume that an array is a pointer, when it is not.

Let’s recap why this works.

When assigning a string, the compiler expects the contents to be stored within an array. Each element within the array is an address to the value given to it in memory.

So in our example above (i.e. the string "abc"), "a" is stored in memory and the address of that memory is placed in my_string[0]. Next "b" is stored in memory and the address of that memory is placed in my_string[1] and so on.

A pointer in contrast is a single location in memory, whereas an array hold lots of memory addresses.

Because of this, an array variable automatically points to the first element within the array.

This is why if you try to printf a string, the compiler will complain if you don’t provide a pointer. Because it expects a string to have been stored within an array (which our earlier example didn’t). But when storing a string inside an array, the variable that is passed to printf would already be a pointer, due to it automatically referencing the first array element as its value.

Interestingly, an array’s type is made up of the element type + the overall array dimension. So int foo[3] is a different type to int bar[4]. Even though the value type int is the same, the array dimension (size/length) is different.

If you want to know how many bytes an array will occupy, then you calculate it based upon the number of elements multiplied by the size of each element.

Lastly, you can define and initialize a string without specifying an array dimension (i.e. no size):

char foobar[] = "No dimension provided!";

What this does is leave the decision of how much memory to allocate to the compiler. But you can only do this when you initialize the variable with a value. Although you couldn’t do char foobar[]; as there is no value for the compiler to utilise to know how much memory to allocate.

Enumerators

Enumerators allow you to define new variable types. They automatically assign numerical values to each of the identifiers within the enumerator (although you do also have control over the specific values as well).

This concept is best explained by way of example:

#include <stdio.h>

int main(void) {
  enum weekend {Saturday, Sunday};     // 0, 1
  enum weekend today = Sunday;         // 1
  enum weekend saturday = Saturday;    // 0
  enum weekend yesterday = today - 1;  // 0 now yesterday is Saturday
  printf("today %d\n", today);
  printf("saturday %d\n", saturday);
  printf("yesterday %d\n", yesterday);
  return 0;
}

You can see that we define a weekend type and each element in that set is assigned an automatic numerical value (0 and 1).

Next we create a new variable today of type weekend and assign it the value Sunday, which means it is effectively assigned the value 1.

You’ll also notice this is why we’re able to do simple arithmetic with the today variable (e.g. today - 1 to get 0).

If you wish to provide your own values you can:

enum bool {
  true = 1,
  false = 2
};

enum bool on = true;
enum bool off = false;

printf("on: %d\n", on);   // 1
printf("off: %d\n", off); // 2

Memory Allocation with different Types

Read this article if you need a refresher on understanding RAM, bits, binary and stuff like that

Array

Consider the following code:

int foo[3] = {1,2,3};
printf("foo variable points to = %p\n", foo);

int i = 0;
do {
  printf("foo[%u] = %p\n", i, (void *)(&foo[i]));
  i++;
} while(i < 3);

printf("sizeof(foo) = %lu\n", sizeof(foo));

So in this piece of code we create an array and assign it to foo (where the first element’s memory address will ultimately be referenced).

Next we print out what the foo variable points to, which for me outputs:

foo variable points to = 0x7fff525fd6ac

Then we loop over the foo array and print out each element’s address, which for me outputs:

foo[0] = 0x7fff525fd6ac
foo[1] = 0x7fff525fd6b0
foo[2] = 0x7fff525fd6b4

Notice that the foo variable and the first element of the foo array are the same value: 0x7fff525fd6ac, which is the address of the memory location for the value 1 assigned to foo[0].

So this demonstrates clearly that when a variable is assigned an array, then it will actually point to the memory address of that array’s first element.

If we look at (void *)(&foo[i]), shown in the above example code, we find this interesting setup where by we’re casting the address to void. Well you don’t need to do that unless you’re passing a variable that is itself a pointer variable to another variable.

For example, int foo = 1; int *pFoo = foo. Here you would cast to void to prevent a possible warning from the compiler when using %p within a printf statement.

This is because printf and %p will expect the value to be a pointer type, but in cases where you have a pointer variable assigned an int variable, then the type of &pFoo would actually be a ‘pointer to pointer to int’. It just highlights the importance of paying attention to the variables you’re defining and what they’re pointing to.

Finally we print the size of the foo variable, which outputs:

sizeof(foo) = 12

You might wonder where the value 12 comes from? Well, this goes back to how data is stored in memory (i.e. a block of memory is 1 byte; so 8 bits). We defined the type of the array to be int and (depending on the compiler) that will be either two bytes, four bytes or eight bytes.

For most 32 bit systems the size of int would be four bytes. You can always use sizeof(int) to check:

printf("sizeof(int): %ld\n", sizeof(int)); // 4

So looking back at our example, we have three array elements, and if each element takes up four bytes then it makes sense the size of the array would be 12 (i.e. 4 + 4 + 4 = 12).

To calculate the number of items within the array itself, rather than the number of bytes, you can use:

size_t element_count = sizeof foo/sizeof foo[0];
printf("element_count: %zu\n", element_count); // 3

ℹ️ NOTE

%z is for size_t and the u prevents an invalid conversion specifier error.

Signed vs Unsigned

In C you can define an integer to be either signed or unsigned. The former means the number can be both negative and positive as well as zero. Whereas the latter is always positive.

ℹ️ NOTE

typically, if a number is negative, you’ll prefix it with -. If the number is positive, then it is just the number. For example, -1 and 1. This is a little more convoluted in binary though (resulting in concepts such as ‘ones complement’ and ‘twos complement’ - Google it if you want to know more though).

You don’t need to explicitly provide the signed keyword (e.g. signed int <var_name>), it’s just implied.

The latter (unsigned) is an integer that can only be positive. So if you need to store an integer and you know the value will always be zero or positive, then you can define it as being unsigned and the compiler can make appropriate optimisations based on that understanding.

So although the underlying memory allocation is the same for signed or unsigned, the actual values represented are slightly different, in that unsigned allows for storing values that are twice the size of signed, because half of signed’s values have to account for negatives.

Reallocating Memory

With strings you typically define them as follows (i.e. the underlying data structure is an array):

char names[20] = "hello";

But this can result in wasted reserved memory. Also, when reading input from stdin (e.g. user input) the amount of characters entered could exceed the specified reserved memory allocation.

Below is an example, given as a Stack Overflow response, that reads each character from stdin and reallocates the memory space if required (it’s advised that you ensure reallocation of memory is done as a multiple; such as double the size):

char *getln()
{
    char *line = NULL, *tmp = NULL;
    size_t size = 0, index = 0;
    int ch = EOF;

    while (ch) {
        ch = getc(stdin);

        /* Check if we need to stop. */
        if (ch == EOF || ch == '\n')
            ch = 0;

        /* Check if we need to expand. */
        if (size <= index) {
            size += CHUNK;
            tmp = realloc(line, size);
            if (!tmp) {
                free(line);
                line = NULL;
                break;
            }
            line = tmp;
        }

        /* Actually store the thing. */
        line[index++] = ch;
    }

    return line;
}

There is a very good blog post that covers the steps in detail here: https://brennan.io/2015/01/16/write-a-shell-in-c/. It’s effectively the same implementation (in principle), except for the use of getchar vs getc (the former can only read from stdin, whereas getc can read from any input stream).

#define LSH_RL_BUFSIZE 1024
char *lsh_read_line(void)
{
  int bufsize = LSH_RL_BUFSIZE;
  int position = 0;
  char *buffer = malloc(sizeof(char) * bufsize);
  int c;

  if (!buffer) {
    fprintf(stderr, "lsh: allocation error\n");
    exit(EXIT_FAILURE);
  }

  while (1) {
    // Read a character
    c = getchar();

    // If we hit EOF, replace it with a null character and return.
    if (c == EOF || c == '\n') {
      buffer[position] = '\0';
      return buffer;
    } else {
      buffer[position] = c;
    }
    position++;

    // If we have exceeded the buffer, reallocate.
    if (position >= bufsize) {
      bufsize += LSH_RL_BUFSIZE;
      buffer = realloc(buffer, bufsize);
      if (!buffer) {
        fprintf(stderr, "lsh: allocation error\n");
        exit(EXIT_FAILURE);
      }
    }
  }
}

Notice c variable is declared as an int and not a char, the author of the blog post makes mention of this as being because EOF is an int type

The author then goes on to explain that in more recent releases, there is a much shorter version that can be implemented thanks to the getline function:

char *lsh_read_line(void)
{
  char *line = NULL;
  ssize_t bufsize = 0; // have getline allocate a buffer for us
  getline(&line, &bufsize, stdin);
  return line;
}

Function Prototypes

A compiler will error if you try to call a function before it has been defined. This can be mitigated by utilising function prototypes that let you define the signature of the function up front and defer the definition until a later point in time (sort of like defining an interface type):

// Function prototypes
int Foo(double data_values[], size_t count);
int Bar(double *x, size_t n);

int main(void) {
  int f = Foo(...signature...);
  int b = Bar(...signature...)
}

// Definitions for Foo() and Bar()

Conclusion

So that covers most of what I found interesting about C over the last few days. I’m not planning on writing any C code in the future (why bother when I can get pretty compariable performance + great tooling etc etc with a language like Go).

But regardless, this was a fun little exercise in learning something new. Hope you learnt something too. As always, feel free to ping me on twitter if you feel I’ve missed something.

Understanding Man Pages

Fri, 25 Nov 2016 00:00:00 +0000

Introduction

Your operating system provides manual pages that explain what specific commands do and where they can be located on your computer.

Most of us are at least familiar with opening a terminal and typing:

man <some_command>

But you might be confused, when you see a manual page informing you to check out <another_command>(2). For example, you might be shown the message at the end of a man page that says See also: chmod(2).

If you try that, it won’t work:

man chmod(2)

So let’s understand why that is and what it means.

Sections

So the manual pages are separated into ‘sections’. This is what the number within the parentheses represents (i.e. chmod(2) represents the chmod command from section 2 of the OS manual).

To find where your manual pages are stored:

cd /usr/share/man

A simple ls will show that on my OS (macOS) I have nine manuals:

man1/   man2/   man3/   man4/   man5/   man6/   man7/   man8/   man9/

Each manual section has an introduction page † that explains what the section covers. These (taken from my OS) are as follows:

man1: introduction to general commands (tools and utilities)
man2: introduction to system calls and error numbers
man3: introduction to the C libraries
man4: contains documentation on special files and sockets
man5: introduction to file formats
man6: contains documentation about games and other miscellaneous fun programs
man7: miscellaneous information pages
man8: introduction to system maintenance and operation commands
man9: introduction to system kernel interfaces

† except for manual sections 4 and 6. I had to go to the online reference for macOS in order to find out what they contained

If you wanted to know ‘at a glance’ what commands were available, then you could install the tree command and execute it at the current directory (/usr/share/man) and this would show you something like the following (cut short for brevity):

.
├── man1
│   ├── ab.1
│   ├── accesstool.1
│   ├── addftinfo.1
│   ├── afconvert.1
│   ├── afhash.1
│   ├── afida.1
│   ├── afinfo.1
├── man2
│   ├── accept.2
│   ├── access.2
│   ├── acct.2
│   ├── adjtime.2
│   ├── aio_cancel.2
│   ├── aio_error.2
│   ├── aio_read.2

Sub sections

You’ll notice that most page files within each manual section have an extension that matches the section they’re contained within (e.g. ab.1 for ab command inside manual 1 or aio_read.2 for aio_read command inside manual 2).

What’s interesting is that there are some files, such as httpdstat.d.1m (which is inside manual section 1) that have a different extension (d.1m) or asn1parse.1ssl (1ssl).

By looking back at the online macOS manual pages reference I noticed there were additional sub sections:

Section 1m: contains documentation for tools built on top of DTrace
Section 1ssl: contains documentation on tools that are part of OpenSSL
Section 1tcl: contains documentation on tools that are part of Tcl
Section 3cc: contains documentation on the Common Crypto API
Section 3pcap: contains documentation on the packet capture library, libpcap
Section 3pm: contains documentation on Perl modules
Section 3ssl: contains documentation on OpenSSL C library routines
Section 3tcl: contains documentation on Tcl/Tk C library routines
Section 3x: contains documentation on curses-related C library routines
Section 5ssl: contains documentation on SSL-specific configuration file formats
Section 7ssl: miscellaneous SSL documentation section
Section n: contains documentation about Tcl/Tk
Section ntcl: contains documentation about Tcl/Tk

Accessing different sections

Typically when we type man <some_command>, the man command will search all the sections looking for the specified command.

It stops at the first match it finds. So there could be a scenario where you search for a command and you get the user space (top-level) entry command and not the ‘system call’ version that actually interacts with your OS.

The chmod command is a good example of that scenario.

If you want to see the actual system call manual page for chmod, you need to search inside the specific section (which in this case is 2):

man 2 chmod

Alternatively, you can search multiple specific sections:

man -S 1:2 chmod

The above command will search through only sections 1 and 2, but it will still stop at the first match it finds so the above command is no different in outcome than just man chmod.

But if you know for sure that the command you’re looking for is somewhere within either the system call or C library manual pages, then man -S 2:3 <your_command> would prevent a command from manual section 1 getting matched first.

ℹ️ NOTE

to see the intro page for a section use man <n> intro

Searching by phrase

In case you’re unsure where to find a specific piece of information, you can use the -K flag which will search all manuals for the given search term/phrase. This can be slow, so it’s best to also provide a manual to scope the search down to.

For example, I wanted to lookup refspec in the git manual. I didn’t know where to look so I ran the following lookup:

man -K "refspec" git

This then presented a manual name along with the options of y/n/q.

If I type y then the listed manual page would be opened. Once I ‘quit’ the page (e.g. q) then the next manual page found to contain the phrase refspec would be listed along with the y/n/q options. Anytime you want to stop going through the list of matches, you type q.

Syscalls and C

Fri, 18 Nov 2016 00:00:00 +0000

This post is aimed at explaining the difference between a system call (provided by the Linux kernal) and a wrapper function that has a similar name within one of the C standard libraries.

I have no formal Computer Science (CS) background, I started programming in 1999 and only from 2016 am I starting to learn the C programming language in order to help give me a deeper knowledge of how *nix systems work and other core CS concepts.

See my previous post called ‘Bits and Bytes)’ to see where these learnings are taking me.

What’s the issue?

It can be confusing sometimes knowing where to look for documentation when dealing with C †

† that is if you’re not a systems engineer, and have no CS degree, nor learnt C

As an example, you might learn about the strace command and start investigating what your Ruby application is up to. In doing so you’ll see lots of calls to different functions and you might decide you want to look up the documentation for those functions.

This could be where your first problem arises. You might think “Ruby is written in C, so I’ll look at the C documentation” and then come up with nothing.

So what’s going on?

The key is to remember that the Linux operating system (which your code is very likely running on) is itself written in C. But Linux provides its own set of functions written in C that aren’t part of the C language.

So you might see a function used and wonder why it’s not showing up within the C language documentation. That’s because it’s not part of the C language. The Linux engineers would’ve created the function within Linux so you need to look at the Linux documentation to find out what it does.

e.g https://linux.die.net/man/

Some of these Linux provided functions are known as ‘system calls’. If you visit the above link you’ll see there in ‘section 2’ a list of all system calls.

An alternative (searchable) list of syscalls can be found here: https://filippo.io/linux-syscall-table/

Wrapper functions?

Now what makes this a little more confusing is that the system calls aren’t usually directly accessible. So ‘section 2’ of the Linux documentation may list all the ‘system call’ documentation, but ‘section 3’ lists all library functions including what are referred to as ‘thin wrapper’ functions for the system calls.

For example, Linux uses a separate library that provides a fork function which is a wrapper around the actual system call fork equivalent provided by Linux itself. The wrapper function is then also something other applications written in C can utilise.

This is noted here https://linux.die.net/man/2/intro in the documentation:

A system call is an entry point into the Linux kernel. Usually, system calls are not invoked directly: instead, most system calls have corresponding C library wrapper functions which perform the steps required in order to invoke the system call. Thus, making a system call look the same as invoking a normal library function.

So what do these thin wrapper functions do? Well the docs tell us…

Often the glibc wrapper function is quite thin, doing little work other than copying arguments to the right registers before invoking the system call, and then setting errno appropriately after the system call has returned. Note: system calls indicate a failure by returning a negative error number to the caller; when this happens, the wrapper function negates the returned error number (to make it positive), copies it to errno, and returns -1 to the caller of the wrapper. Sometimes, however, the wrapper function does some extra work before invoking the system call. For example, nowadays there are two related system calls, truncate and truncate64, and the glibc truncate() wrapper function checks which of those system calls are provided by the kernel and determines which should be employed.

Using fork as an example:

Here is the system call docs: https://linux.die.net/man/2/fork
Here is the wrapper docs: https://linux.die.net/man/3/fork

But where do some of the wrapper equivalents come from? Well, one such provider is glibc; which is GNU’s standard C library. Which states:

The C language provides no built-in facilities for performing such common operations as input/output, memory management, string manipulation, and the like. Instead, these facilities are defined in a standard library, which you compile and link with your programs. The GNU C Library, described in this document, defines all of the library functions that are specified by the ISO C standard, as well as additional features specific to POSIX and other derivatives of the Unix operating system, and extensions specific to GNU systems.

Here’s a link also to the standard C library libc if you’re interested.

Direct system call?

What if one of the additional C libraries (libc, glibc etc) don’t provide a wrapper?

Well in these situations you can make a direct system call!

See https://linux.die.net/man/2/syscall which states:

syscall() is a small library function that invokes the system call whose assembly language interface has the specified number with the specified arguments. Employing syscall() is useful, for example, when invoking a system call that has no wrapper function in the C library.

Bits and Bytes

Wed, 16 Nov 2016 00:00:00 +0000

Introduction

So this is a bit of a random journey through some different computer based subjects. Things that I felt I should try to better understand. Some of it will be very basic, but hopefully it’ll be useful to those who are new to tech, and are interested in learning these things (or old dogs like me who should know better).

Bit

The word bit is short for binary digit.

A bit is either a 1 (true) or a 0 (false).

Computers only understand the binary format (i.e. base-2)

We discuss ‘base’ numbers below

Byte

A grouping of eight bits is called a byte.

Read the next section to realise why I mention this tidbit of information

RAM

The word ram is an acronym for random access memory.

It’s non-persistent memory.

Meaning it is lost when your machine is restarted, and persists only for the lifetime of the program using it.

RAM consists of bits, but each segment of memory is actually a grouping of eight bits (which we already know is called a byte).

So in short we would say RAM is made up of bytes.

Bytes are uniquely numbered to allow easy lookup of their contents.

A byte’s unique number is also referred to as its address.

Bit and RAM Visualisation

We can see that each bit has an associated value which is calculated using the power of base-2 (we’ll cover base numbers shortly).

So:

2⁰ = 1
2¹ = 2
2² = 4 (i.e. 2 * 2)
2³ = 8 (i.e. 2 * 2 * 2)
2⁴ = 16 (i.e. 2 * 2 * 2 * 2)
2⁵ = 32 (i.e. 2 * 2 * 2 * 2 * 2)
2⁶ = 64 (i.e. 2 * 2 * 2 * 2 * 2 * 2)
2⁷ = 128 (i.e. 2 * 2 * 2 * 2 * 2 * 2 * 2)

With this information we can identify the value assigned by adding up the numbers associated with the ones and zeros.

So in the image we can see four of the bits are given 1 (on) and the rest are 0 (off), meaning if we add up all the associated values of the bits that are ‘on’ we get 99 (e.g. 64 + 32 + 2 + 1).

Basic Bit Operators

You may have seen bit operators like << or >> and wondered what they mean (generally referred to as ‘bitwise operators’). Essentially they manipulate bits.

Here are some examples:

64 >> 1 = 32
1 << 3 = 8
1 << 7 = 128
1 << 15 = 32768

So if we keep in mind the earlier image of our 8 bits in a byte memory representation, and that each bit had its own value (as described above), we can see (for example) that if we started at 64 and moved to the right by 1 bit, the next bit would be the one assigned the value 32. Hence, 64 >> 1 results in the value 32.

Similarly, if we start with the bit value 1 and move to the left by 15 bits we’ll get back the result of 32768 because although we’ve only shown eight bit spaces above, you can just keep moving to the left (64 * 2 = 128, 128 * 2 = 256, now keep going seven more places until you reach the fifteenth bit… 256 * 2 = 512, 512 * 2 = 1024, 1024 * 2 = 2048, 2048 * 2 = 4096, 4096 * 2 = 8192, 8192 * 2 = 16384, 16384 * 2 = 32768).

For more bitwise operators, refer to these posts: https://wiki.python.org/moin/BitwiseOperators and https://medium.com/learning-the-go-programming-language/bit-hacking-with-go-e0acee258827

Bits and ASCII

ASCII is a set of codes where each ‘code point’ represents a textual character such as a 1, a, z, !, ? etc.

Each code point has an associated binary number. For example, a has the binary number 0110 0001 which if we add up the values associated with those specific bits we’ll find it’ll yield the code point number 97. In ASCII the character a is code point 97.

Bits and Numbers

1 kilobyte (or 1KB) is 1,024 bytes.

1,024 bytes is 8192 bits (8192/8 or 8*1024, whichever you prefer)

The following explanation is taken from “Beginning C” by Apress Publishing…

You might be wondering why we don’t work with simpler, more rounded numbers, such as a thousand, or a million, or a billion. The reason is this: there are 1,024 numbers from 0 to 1,023, and 1,023 happens to be 10 bits that are all 1 in binary: 11 1111 1111, which is a very convenient binary value. So while 1,000 is a very convenient decimal value, it’s actually rather inconvenient in a binary machine—it’s 11 1110 1000, which is not exactly neat and tidy. The kilobyte (1,024 bytes) is therefore defined in a manner that’s convenient for your computer, rather than for you.

So if we add up 512+256+128+64+32+16+8+4+2+1 (notice this takes the existing 8 bit calculation from the above image and continues it for another two bits) we get 1023.

IPs

Here is an example IPv4 IP:

216.27.61.137

IPv4 IPs are expressed in decimal format.

ℹ️ NOTE

IPv6 IPs are eight 4-character hexadecimal numbers,
which represent 16 bits each (for a total of 128 bits)
e.g. 2001:0db8:0a0b:12f0:0000:0000:0000:0001

To translate the above IP into binary form (for the sake of a computer to process it), we could use the above visualisation image to help us. The end result of which would look like this:

11011000.00011011.00111101.10001001

Which breaks down into:

11011000: 128 + 64 + 16 + 8 = 216
00011011: 16 + 8 + 2 + 1 = 27
00111101: 32 + 16 + 8 + 4 + 1 = 61
10001001: 128 + 8 + 1 = 137

This explains why each of the four numbers within the decimal formatted version (i.e. 216.27.61.137) are called octets, as they represent eight bits (or a ‘byte’ as we learned earlier) when viewed in binary form.

This also explains why IPv4 IPs are considered 32-bit numbers, because if you add each of the bits together (i.e. the number of total bits, not the value assigned to each bit) you’ll find there are a total of 32 bits that make up the IP.

Each bit can have two different states (1 or zero), meaning the total number of potential combinations per octet can be either 28 or 256. Meaning each octet can contain a potential value between zero and 255. Meaning if we were to combine the four octets, we could potentially have 4,294,967,296 variations.

We can see the decimal represenation of an IPv4 IP is made up of four base-10 numbers (216, 27, 61, 137). Where each of those four numbers represent the binary equivalent (216=11011000, 27=00011011, 61=00111101, 137=10001001), which is a base-2 representation of a byte (or octet).

If you’re unsure of what base numbers are and how they work, then read on…

Base Numbers

It’s worth quickly covering what base numbers are as they help us understand the other different protocols we use on a regular basis, such as binary and things like IPs.

Any number can be represented in multiple ways using a different base numbering system.

There are many numbering systems, but the typical ones we’re used to are:

Base 10 (Decimal)
Base 2 (Binary)
Base 8 (Octal)
Base 16 (Hexadecimal)

The standard number system we (as humans) are most familiar with is called base-10 and it consists of the following numbers:

0,1,2,3,4,5,6,7,8,9

Notice there are ten numbers, hence it is called the base-10 system

If we were to look at a number like 66, then this would tell us the number is made up of 6 tens and six units.

These numbers (0-9) represent ‘whole numbers’, while in the base-10 system we can also use a decimal point to represent decimal fractions of a number (e.g. 1.2).

Below is an image, credit to Jenny Eather, that helps us visualise this model:

The ‘base number’ is the number of numbers within the system. So base-10 has 10 numbers (0,1,2,3,4,5,6,7,8,9) whereas binary is base-2 because it uses two numbers only (0, 1).

If you want to know the unit each number in a system represents (we’ll use base-10 as the example, thanks to the following visualisation credited to Jenny Eather), then you calculate this using the power of the base number.

So, as per the above visualisation, we can see:

10³: 10*10*10: 1000 (thousands)
10²: 10*10: 100 (hundreds)
10¹: 10: 10 (tens)
10⁰: 1: 1 (unit)

So in practical terms, if you have a number like 75 and want to represent it as base-10:

5 (10⁰: 5 units)
7 (10¹: 7 tens)

Similarly, if we had a number like 675 and want to represent it as base-10:

5 (10⁰: 5 units)
7 (10¹: 7 tens)
6 (10²: 6 hundreds)

You can indicate what base you wish to represent a number like so:

n_b

Where n is the number and b is the base you wish to state it is in.

For example:

75₁₀

This is the number 75 and we’re stating the base it represents is 10.

This is useful when you’ve converted a number like 75 into a different base (let’s say base-8, which would be 113) and you want to give that number in the proper context to someone else. You could write it as 113₈.

Convert Base-10 into Base-²⁄₈

ℹ️ NOTE

the steps are the same for converting to base-2 and base-8

Now let’s consider how to convert the number 75 into another base, like base-8. To do so, follow these steps

divide the number (75) by the desired base (8) (take note of the remainder: 3)
take the result (9) and do the same (i.e. divide by the base and take note of the remainder)
keep doing this until the result of dividing the previous answer by the base is zero
now write out the remainders bottom to top, and that’s the number in base-8

In long form this looks like this:

75 (number) / 8 (base) = 9 (rounded) with a remainder of 3
9 (previous answer) / 8 (base) = 1 (rounded) with a remainder of 1
1 (previous answer) / 8 (base) = 0 (rounded) with a remainder of 1

Meaning 75 evaluated in base-8 would be 113 (all the remainders concatenated together, ‘bottom up’)

Convert Base-10 into Base-16

The algorithm for converting from base-10 into base-2 and base-8 works basically the same for converting into base-16. But there is one caveat whereby a remainder can be in the double digits, and apparently (for reasons I don’t completely understand) we don’t want that, and so the number system was designed to replace the six instances where this can occur (the remainders being: 10, 11, 12, 13, 14, 15) with a alpha-numeric equivalent:

10-A
11-B
12-C
13-D
14-E
15-F

So if we want to convert 110₁₀ into a hexadecimal, the outcome of the algorithm would be:

110 (number) / 16 (base) = 6 (rounded) with a remainder of 14
6 (previous answer) / 16 (base) = 0 (rounded) with a remainder of 6

We know that we need to replace 14 (a double digit remainder) with the letter E (see above mapping).

Meaning 110 evaluated in base-16 would be 6E

Let’s try it again, but with the number 411₁₀:

411 (number) / 16 (base) = 25 (rounded) with a remainder of 11
25 (previous answer) / 16 (base) = 1 (rounded) with a remainder of 9
1 (previous answer) / 16 (base) = 0 (rounded) with a remainder of 1

We know that we need to replace 11 with the letter B.

Meaning 411 evaluated in base-16 would be 19B

Convert Any Base to Base-10

What if you want to convert a base-8 number (let’s say 113, why not) into base-10? The algorithm is to multiple the individual numbers by their associated power of the base and then add the numbers together.

So here are the base-8 powers:

3: 8⁰
1: 8¹
1: 8²

And here is the algorithm:

3 x 8⁰ = 3
1 x 8¹ = 8 (i.e. 1*8)
1 x 8² = 64 (i.e. 1*(8 * 8))
3 + 8 + 64 = 75

If you’re dealing with base-16, then again it’s the same but the difference is you’re translating the letter back into the corresponding number.

Let’s convert 19B from base-16 back into base-10:

B x 16⁰ (11 x 16⁰) = 11
9 x 16¹ = 144
1 x 16² = 256
11 + 144 + 256 = 411

Let’s try one more conversion between base-16 to base-10. The number is 1A4:

4 x 16⁰ = 4
A x 16¹ (10 x 16¹) = 160
1 x 16² = 256
4 + 160 + 256 = 420

CIDR

A CIDR is a range of IP addresses. We can use our understanding of bits, bytes and octets to understand the format of a CIDR.

A CIDR typically resembles something like:

10.0.0.0/n

Where n is given the value 8, 16, 24, or 32 and these represent each of the 8-bit blocks that make up the IP.

If we want an IP range between 10.0.0.0 and 10.255.255.255, we’d specify the CIDR as 10.0.0.0/8.

What 8 states is that the last 8 bits of the 32-bit number is accounted for (this being the 10 we’ve specified in our example). Meaning the rest of the 8-bit segments can be added up to their max of 255 (meaning the last IP in this CIDR range would be 10.255.255.255).

Similarly if we want an IP range between 10.0.0.0 and 10.0.255.255, we’d specify the CIDR as 10.0.0.0/16.

Again, 16 states that the next 8-bits segment of the 32-bit number is now accounted for (this being the 0 we’ve specified in our example 10.0). Meaning the rest of the 8-bit segments can be added up to their max of 255 (meaning the last IP in this CIDR range would be 10.0.255.255).

And so on…

So 10.0.0.0/24 gives us an ip range of 10.0.0.0 to 10.0.0.255 (256 IPs).

Whereas 10.0.0.0/32 gives us an ip range of 1 ip (10.0.0.0 to 10.0.0.0).

ℹ️ NOTE

you can use a tool such as http://www.ipaddressguide.com/cidr to help you generate CIDRs

We can use the earlier byte visualisation table matrix to help us manually calculate a CIDR range.

I’ve reproduced it below with a HTML table:

ℹ️ NOTE

you’ll likely need to scroll to the right to see the start of the 32-bit

IP	10	0	0	1
8 Bit Blocks	8 bits [24-31]	8 bits [16-23]	8 bits [08-15]	8 bits [00-07]
32 Bit #	31	30	29	28	27	26	25	24	23	22	21	20	19	18	17	16	15	14	13	12	11	10	09	08	07	06	05	04	03	02	01	00
Decimal	128	64	32	16	8	4	2	1	128	64	32	16	8	4	2	1	128	64	32	16	8	4	2	1	128	64	32	16	8	4	2	1
Binary	0	0	0	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1

Conclusion

There you go. A whirlwind ride through different basic computer tech topics. As always, if I’ve gotten anything wrong then just let me know on twitter.

Terminal Password Manager

Wed, 19 Oct 2016 00:00:00 +0000

Introduction

I’m guessing that you have an app like 1Password or KeePassX to manage your passwords and other login credentials and you’re now looking for a cheaper alternative, and also one that doesn’t rely on a GUI (although as you’ll see that’s still possible).

If so, then “Pass” might be something of interest to you as it offers you the ability to securely store information via the command line interface (e.g. terminal shell) and is a free solution to that particular problem.

But you need to be aware that it’s not as feature rich as far as something like 1Password is concerned.

For example, 1Password has browser extensions and native apps that allow you to automatically pull your password from 1Password directly into the relevant login fields of the service you’re visiting.

Pass is much simpler than that.

On the plus side, Pass is based on standardised unix technology. Specifically GPG, which can give you confidence in the security mechanisms being utilised.

ℹ️ NOTE

if you need a refresher on encryption and GPG, then I’ll refer you to an earlier blog post of mine that covers the basics on this topic

Installation

Installation on macOS is easy with Homebrew, and let’s face it, if you’re using macOS then Homebrew has become a de facto standard:

brew install pass

Once you’ve installed Pass you’ll probably want to install the shell auto-complete script as well. This requires you sourcing it into your shell profile.

For me on Bash this looks like this:

echo "source /usr/local/etc/bash_completion.d/password-store" >> ~/.bashrc

ℹ️ NOTE

other distros available on the Pass website

At this point you need to initialize Pass:

pass init "AB123C4D"

Just swap AB123C4D for your own GPG id. You can find that information by executing the following GPG command to list out all your available keys:

gpg --list-keys

You should see something like the following output:

pub   1234A/AB123C4D

Your GPG id is the bit after the forward slash: AB123C4D

Example Usage

So here’s the super quick run down on how to use Pass:

pass: displays the structure of your information
pass generate Foo/bar 20: insert new record & auto-generate password
pass insert Foo/bar: insert new record & manually enter password
pass insert Foo/bar -m: insert new record & manually enter multiline data
pass Foo/bar: display first line of data in stdout †
pass -c Foo/bar: copy first line of data into clipboard
pass rm Foo/bar: remove the file bar ∆

∆ to remove the whole directory: pass rm -rf Foo
Once the last file in a directory is removed, so is the directory

So if you executed the second command (generate) and then executed the first command (pass) you would see something like the following:

Password Store
└── Foo
    └── bar.gpg

So we can see we’ve created an arbitrary data structure of Foo as the top level directory, and inside of that a encrypted file called bar.gpg.

Where we’ve specified † is an important point to be aware of, because you’re free to create any ‘structure’ you like. So the suggestion from Pass is to create a single file that contains the following information (just as a guide):

Yw|ZSNH!}z"6{ym9pI
URL: *.amazon.com/*
Username: AmazonianChicken@example.com
Secret Question 1: What is your childhood best friend's most bizarre superhero fantasy?
Phone Support PIN #: 84719

Notice the first line is just the password, which means you can easily copy it to your clipboard using the -c flag (e.g. pass -c Foo/bar). All the remaining information is considered secondary meta data.

Other types of data structuring you could do is to store data in sub-directories, which would make it easier for copying into your clipboard. For example:

Foo/app1/password
Foo/app1/secret-question

Alternatively, you could store the password in one file, and have just a single additional metadata file like so:

Foo/app2
Foo/app2.meta

So app2 is the file containing the password, and app2.meta is the file that contains all the other related information such as secret question/answer key pairs and contact numbers/emails etc.

But to be honest, that last style seems a bit pointless as having a .meta file is still a manual process for copying out data (unless your metadata consists of one additional secret question password, which could be on the first line of that file for easy copying, and then all other data is contact numbers and things like that).

Exporting Data

If you want to automate the migration of data out of a GUI based app such as 1Password, then the Pass community has you covered.

Synchronisation

I wanted to be able to backup my encrypted password store, in case my laptop melted one day. So the simplest solution was to move the directory ~/.password-store into a cloud provider space for synchronisation and then symlinking the directory into my home directory:

ln -s ~/YourCloudOrg/.password-store/ ~/.password-store

Yes this means that your encrypted data is now only as secure as the passphrase around your GPG private key. But I’m fairly confident in both my encryption key and the passphrase around it and so this is an acceptable compromise to make.

Security and Convenience, these two always dance around each other.

Mobile and GUI Applications

I’ve no need for a desktop/laptop GUI, as that’s what the terminal shell is for and I’m happy using that. But if you check the introduction text on the Pass website, you’ll find details on some different community built GUIs that are available.

There are also mobile applications, which I’ve yet to try out because they seem like quite a bit of faff to set-up; and this is the biggest downside to Pass so far and it doesn’t work with synchronisation via a cloud provider.

Instead the Android app expects you to configure your Pass store to be a git repository (something I’ve not covered here). But then that requires you to push the store into a public/private git repo.

Now there’s no reason why I couldn’t do this beause I’ve already conceded that security aspect when I exposed the files by syncing them to a cloud provider (a little less visible than a public git repo for some people), but again, if you’re confident in your key encryption and its passphrase then this might work fine for you.

MultiFactor Authentication

The purpose of multifactor authentication (also known as 2FA: two factor authentication) is to add additional security to the process of accessing a service.

For example, typically you’ll log into a service using a username and password. But if your laptop becomes compromised and you’ve saved your login credentials for a particular service then without 2FA you’ve now given up access to that service and the data it holds.

For some services, such as provided by Google, you can enable 2FA. What this means is that you associate with that service (let’s use Google as the example) another device.

Most of the time the device is a mobile phone, as that is one of the few devices that are usually safely held by the true owner of the Google account being accessed.

With 2FA you’ll be provided a token. You then store this token in a 2FA application (Google has its own “Authenticator” Android app for example). Now every time you go to log into your Google account from a new machine, you’ll be asked to consult the 2FA application (which will give you a random/unique key back). You’ll then be expected to provide Google the 2FA key along with your username and password.

We can do a similar thing using Pass. But instead of a mobile device as the associated device to the Google account, you can associate your laptop running Pass and a desktop equivalent 2FA application (remember, it doesn’t have to be your main laptop, it could be another laptop or computer obviously).

Now this would normally be a bit of a concern for some people. The idea being that 2FA is supposed to help when your ‘device’ is compromised. Hence people associate their mobile as the device for handling 2FA, as it’s less likely to get lost or damaged.

If you have your laptop, which has access to the service, also being the associated 2FA token provider kinda defeats the point.

But because we’re using GPG and Pass, in effect (similar to my comments about sync’ing my Pass store onto a cloud provider), if you’re confident your generated GPG key and its passphrase are solid then you should be less concerned because your key will be near impossible to crack via automation and so if your laptop is compromised it won’t matter as the 2FA application won’t be able to pull the Google token from Pass in order to generate a unique key to access Google (along with your username and password).

One way of achieving this was shared with me by Jake Champion:

brew install oath-toolkit

This will provide you with a oath command, which will be used as your 2FA application. The way it works is that when setting up 2FA on your Google account, you’ll take the provided token and store it in Pass.

Now every time you need to access your account, you can execute the following command and extract the Google token from Pass. Which will generate the key needed to be provided to Google when logging in using your username/password:

oathtool --base32 --totp $(pass 2FA/Amazon)

The above snippet assumes you stored your Google token like so:

pass insert 2FA/Amazon <your_google_token>

That’s all there is to it.

ℹ️ NOTE

see this gist for more details on the Google 2FA setup process

Conclusion

Compared to the pricing of something like 1Password:

$3 per month (forever)
$64 (single licence, not all devices either and not all features)

Then considering I spend the majority of my time working from a terminal shell. It would seem that Pass is a good starting point.

But ultimately I think I’m going to have to explore the git hosted option (with a private repository) for the mobile app setup, just so that I can ensure a little less visibility into my data information structure.

Terminal Debugging Utilities

Mon, 12 Sep 2016 00:00:00 +0000

Introduction

Not all programmers need to get their hands dirty and have to dig deep into what exactly their applications or services are doing at a lower/network level.

This is why programmers work with programming languages, as they provide a nice high-level abstraction layer that protects us from a lot of these concerns.

But regardless if you’re a client-side developer or a server-side developer, at some point your application will start misbehaving and it can be useful to have experience using command-line/terminal based tools to help you debug what’s going on.

That is what I want to briefly cover today: a few select programs that you may find useful to have in your debugging toolbox. I won’t claim to be an expert with any of these, but I’ve had to use these tools at some point or another so I at least know what I have available to me whenever things start going haywire.

I’d encourage you to do follow-up reading once you’ve gone through this post, and to experiment with these tools if there are any in particular that you find interesting.

Agony of Choice

You are going to find that some of the tools I mention, have a lot of crossover behaviour between them.

One of the main points of confusion for people, when given the choice of lots of different utility tools, is: when/why should I use this over another very similar tool?

The answer: it depends (as with everything in life).

Sometimes it can just be a personal preference. You’re happy using tool x, which can sort of do that ‘thing’ you need it to, but maybe tool x isn’t quite as good at showing you the ‘thing’ as tool y (which was designed specifically to solve the ‘thing’ problem); but hey, tool x is good enough at it and it also allows me to inspect y and z problems (which are things I look at the majority of the time).

Just be aware that that there will be crossover functionality, and that on occasions certain tools might only have slight additions that could be useful to you depending on the problem you’re trying to debug.

Deprecated Commands?

It was brought to my attention by Aidy Lewis that tools such as ifconfig and netstat have since been deprecated in favour of other tools. See here for the details of what the replacement tools are.

Prerequisites

TCP and HTTP

The fact is in order to use tools such as tcpdump, telnet or netstat you do need to understand the basics of how the TCP and HTTP protocols work in order to utilise these programs fully.

As far as other tools are concerned, you may need to understand some networking basics. I mean, I know very little, but I know just enough to muddle along when I need to.

I’ll be covering how I’ve used these tools but not really much more beyond that, so you may need to do some additional reading in order to appreciate what these tools offer (outside of my own experience with them).

OSI Model

Finally, before we get going, it’s worth taking a moment to consider the OSI Model. What this model represents are the different layers of a system. From the very real hardware level (e.g. physical cables that make the interwebs work) right up to the software level.

Here is a table matrix that attempts to identify these ‘layers’:

OSI Model
Layer	Protocol data unit (PDU)	Function	Examples
Host layers	7. Application	Data	High-level APIs, including resource sharing, remote file access	HTTP, NFS, FTP, Telnet, SMTP, SSH
6. Presentation	Translation of data between a networking service and an application; including character encoding, data compression and encryption/decryption	S/MIME, TLS
5. Session	Managing communication sessions, i.e. continuous exchange of information in the form of multiple back-and-forth transmissions between two nodes	RPC, SCP, PAP
4. Transport	Segment (TCP) / Datagram (UDP)	Reliable transmission of data segments between points on a network, including segmentation, acknowledgement and multiplexing	TCP, UDP, NBF
Media layers	3. Network	Packet	Structuring and managing a multi-node network, including addressing, routing and traffic control	IPv4, IPv6, ICMP, IPsec, CLNP, DDP
2. Data link	Frame	Reliable transmission of data frames between two nodes connected by a physical layer	IEEE 802.2, L2TP, LLDP, IEEE 802 MAC layers (Ethernet, IEEE 802.11, etc.), PPP, ATM, MPLS
1. Physical	Bit	Transmission and reception of raw bit streams over a physical medium	DOCSIS, DSL, IEEE 802 physical layers (Ethernet, IEEE 802.11, etc.), ISDN, RS-232

ℹ️ NOTE

you’ll find many differing versions of the OSI Model (i.e. the layers described are always the same, but you may see more or less protocols defined depending on what version you look at), this is just one such version copied vertabim from Wikipedia

The reason this is useful, is because you can identify which layer the relevant tools are operating at. Tools like netstat operate at layer four (transport: tcp), whereas telnet operates at layer seven (application: it actually has its own protocol telnet).

When debugging an issue, if you know the problem space is a particular layer of the OSI model, then you’ll have an easier time identifying which tool is best suited to the investigation.

Utilities

There are many different utilities, some provided as built-ins to your OS, others might be GNU flavoured or home grown (e.g. they can be built using a myriad of programming languages) and which you have to download separately.

The ones listed below are a selection of tools I find particularly useful for different scenarios. But they’re not all available on the Mac OS (which is what I use, and which - I’m making a massive assumption - you are likely using too).

I do provide basic installation instructions for some of the tools that aren’t available for the Mac OS (either natively or at all) and for those tools that aren’t available for your OS, I would recommend using Docker for testing them out.

I’d suggest for the purpose of this article to try one of the following if you don’t want/use Mac OS:

docker run -it centos /bin/bash

# or

docker run -it ubuntu /bin/bash

Also, the usage between Mac OS and Linux can vary
Example: top -n on Mac shows only n number of items; Linux runs n number of ticks before stopping
If you notice something different, then it’ll likely be the OS
But ultimately the examples I give are for Mac OS (unless stated otherwise)

With all that out of the way, let’s begin…

top

Summary: displays running processes with cpu and memory utilisation

The top command displays processor activity and also displays tasks managed by the kernel in real-time. If you have a cluster of nodes that are setup to scale up based on either CPU or Memory usage, then your first starting point will be to jump onto a running instance and check the different processes running and what their consumption is.

Or maybe an application on your laptop is running very slowly? You can inspect its CPU consumption to see if it’s doing something odd and maxing out at 99%

Here are some basic commands:

top -user <user>: filter processes by those run by specified user

e.g. top -user $(who | awk 'NR==1 {print $1}')

top -n 10: stops command running after 10 intervals

otherwise runs forever or until <Ctrl-c> (SIGINT “interrupt” signal received)

You can change which column controls the order display (the default being CPU) by typing o and then typing the column name. For example:

+command: order by the COMMAND column (ascending order)
-command: order by the COMMAND column (descending order)

+ is implied (so no need to type it)

The following key strokes can be executed whilst top is running…

z: toggles on/off ability to have currently running processes highlighted

Non-Mac

On the non-Mac version of top you may find that you need the output to show you the complete path to the running program (which can be handy if you don’t recognise the program, and want to know where it’s located):

c: toggles on/off absolute path for the COMMAND

Similarly for non-Mac versions you can dynamically change the refresh rate:

d: change interval for screen/data refresh

Finally one other useful thing for non-Mac versions is the ability to kill a process you don’t like the look of:

k: followed by PID of process you want to kill
1: toggles a breakdown of the CPU usage (rather than showing aggregated value)

I mention the above non-Mac options because when debugging on a remote server, chances are you’ll need some of those details more.

Failing all that you could just install the man pages and double check the available options (or run a Docker container and double check them):

apt-get update && apt-get install man

There are also some pretty fancy alternatives, such as:

brew install htop
npm install -g vtop (node)

ps

Summary: displays snapshot of running processes

The ps command is useful for seeing ‘at a glance’ what process id’s have been provided to different programs and which user started them (as well as the relationship between processes).

Majority of the time you’ll use ps just to identify the process id (pid) so you can then utilise another tool for inspecting the relevant process.

The differences between ps and top are subtle, both display details about active processes (albeit in different formats). But top is continuous whereas ps is just a snapshop.

The ps command also offers some more advanced display options, not just a table matrix, and these can help visualise the parent/children relationship for particular processes.

Here are some useful commands you can try:

ps aux: shows all running processes (even those without a tty or are not owned by you)
ps axjf: shows parent process (ppids) & nested children pids with tree formatting

ℹ️ NOTE

the f option doesn’t work the same on Mac OS (man ps for details)

strace

Summary: monitors interactions between processes, by highlighting what syscalls are being made

So strace is awesome for understanding exactly what your application is doing. If you have (just for example) a Python or Ruby app, then you should know that all the function calls it makes are actually abstractions provided by the language.

Most high-level languages are written in C, and so those abstracted functions will end up calling some internal C functions and those C functions will end up making OS level system calls.

It’s these ‘system calls’ that you’ll end up tracing/tracking with strace.

This utility isn’t available on Mac OS, so you’ll need Linux:

With Ubuntu:

apt-get update && apt-get install strace

With CentOS:

yum install strace

Mac OS has dtrace but it does use quite different commands from strace
Although I have seen articles online that help translate

You can run strace against a new process, like so:

strace ls -l

But you can also attach strace to an already running process! This is great for debugging an application that’s misbehaving out in the wild. You would do that like so:

strace -p <process_id>

So let’s consider a simple example where we want to see what the shell command ls -l is actually doing. First let’s trace it to get an idea of the output you’ll see:

strace ls -l

execve("/bin/ls", ["ls", "-l"], [/* 23 vars */]) = 0
brk(0)                                  = 0x109a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f946a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=32096, ...}) = 0
mmap(NULL, 32096, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f22f9462000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0[\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=134296, ...}) = 0
mmap(NULL, 2238192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f9027000
mprotect(0x7f22f9047000, 2093056, PROT_NONE) = 0
mmap(0x7f22f9246000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
mmap(0x7f22f9248000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS...
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\34\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31168, ...}) = 0
mmap(NULL, 2126336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f8e1f000
mprotect(0x7f22f8e26000, 2093056, PROT_NONE) = 0
mmap(0x7f22f9025000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1840928, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f9461000
mmap(NULL, 3949248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f8a5a000
mprotect(0x7f22f8c14000, 2097152, PROT_NONE) = 0
mmap(0x7f22f8e14000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
mmap(0x7f22f8e1a000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS...
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=252032, ...}) = 0
mmap(NULL, 2347200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f881c000
mprotect(0x7f22f8859000, 2093056, PROT_NONE) = 0
mmap(0x7f22f8a58000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f8618000
mprotect(0x7f22f861b000, 2093056, PROT_NONE) = 0
mmap(0x7f22f881a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\20\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18624, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f9460000
mmap(NULL, 2113760, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f22f8413000
mprotect(0x7f22f8417000, 2093056, PROT_NONE) = 0
mmap(0x7f22f8616000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE...
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f945f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f945d000
arch_prctl(ARCH_SET_FS, 0x7f22f945d840) = 0
mprotect(0x7f22f8e14000, 16384, PROT_READ) = 0
mprotect(0x7f22f8616000, 4096, PROT_READ) = 0
mprotect(0x7f22f881a000, 4096, PROT_READ) = 0
mprotect(0x7f22f8a58000, 4096, PROT_READ) = 0
mprotect(0x7f22f9025000, 4096, PROT_READ) = 0
mprotect(0x7f22f9246000, 4096, PROT_READ) = 0
mprotect(0x619000, 4096, PROT_READ)     = 0
mprotect(0x7f22f946c000, 4096, PROT_READ) = 0
munmap(0x7f22f9462000, 32096)           = 0
statfs("/sys/fs/selinux", 0x7ffca795ea30) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffca795ea30)      = -1 ENOENT (No such file or directory)
brk(0)                                  = 0x109a000
brk(0x10bb000)                          = 0x10bb000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f9469000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 384
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f22f9469000, 4096)            = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1607664, ...}) = 0
mmap(NULL, 1607664, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f22f92d4000
close(3)                                = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost...
ioctl(1, TIOCGWINSZ, {ws_row=53, ws_col=172, ws_xpixel=1895, ws_ypixel=1171}) = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2570, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f9469000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2570
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f22f9469000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en_US.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en_US/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26258, ...}) = 0
mmap(NULL, 26258, PROT_READ, MAP_SHARED, 3, 0) = 0x7f22f9463000
close(3)                                = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 10 entries */, 32768)    = 296
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
open("/usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT 
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22f9462000
write(1, "total 0\n", 8total 0
)                = 8
close(1)                                = 0
munmap(0x7f22f9462000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

OK… so that’s a lot of noise. So let’s see if we can’t quieten it down a bit by using the count flag -c:

strace -c ls -l

total 0
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        10           read
  0.00    0.000000           0         1           write
  0.00    0.000000           0        35        24 open
  0.00    0.000000           0        14           close
  0.00    0.000000           0        12           fstat
  0.00    0.000000           0        25           mmap
  0.00    0.000000           0        14           mprotect
  0.00    0.000000           0         4           munmap
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0         8         8 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         2           getdents
  0.00    0.000000           0         2         2 statfs
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         1           openat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   135        34 total

Now that’s better. We get a good overview of what syscalls have been made by the ls program, how many times and the overall time involved for each type of function.

But as you can see, when tracing a program that only lists the files in a directory (i.e. ls -l), you’re not going to see much in the way of ‘time’. So let’s use lsof (will read more about this later), which can take a few seconds to run:

strace -c lsof

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 66.25    0.018456           7      2479           write
  7.11    0.001980           1      1405           read
  4.44    0.001237           1      2067        12 stat
  3.77    0.001051           1      2074      1021 close
  3.61    0.001007           4       244           alarm
  3.08    0.000857           2       543           munmap
  2.18    0.000608           2       244           rt_sigaction
  2.06    0.000575         575         1           wait4
  1.72    0.000479           1       554           fstat
  1.70    0.000474           1       875        57 readlink
  1.30    0.000363           0       834        20 open
  0.84    0.000235           0       507           lstat
  0.83    0.000230           0       568           mmap
  0.60    0.000167           1       222           openat
  0.42    0.000116           8        14           recvmsg
  0.09    0.000024           0       444           getdents
  0.00    0.000000           0         2           poll
  0.00    0.000000           0        17           lseek
  0.00    0.000000           0        18           mprotect
  0.00    0.000000           0        13           brk
  0.00    0.000000           0         1           ioctl
  0.00    0.000000           0        11        11 access
  0.00    0.000000           0         2           pipe
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0        14           socket
  0.00    0.000000           0         7         6 connect
  0.00    0.000000           0         8           sendto
  0.00    0.000000           0         1           recvfrom
  0.00    0.000000           0         7           setsockopt
  0.00    0.000000           0         1           clone
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         5           uname
  0.00    0.000000           0         1           umask
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         1           getuid
  0.00    0.000000           0         1           getgid
  0.00    0.000000           0         1           geteuid
  0.00    0.000000           0         1           getegid
  0.00    0.000000           0         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.027859                 13191      1127 total

OK that output is a bit more practical, as we can see that the write syscall took approximately 66% of the overall time of the program to run. If this was your own application you might consider that an issue worth delving into more deeply.

If you want to get a rough idea for what some of these system calls mean, then use the below quick guide (otherwise Google is your friend):

Syscall	What it does?
`read`	read bytes from a file descriptor (file, socket)
`write`	write bytes from a file descriptor (file, socket)
`open`	open a file (returns a file descriptor)
`close`	close a file descriptor
`fork`	create a new process (current process is forked)
`exec`	execute a new program
`connect`	connect to a network host
`accept`	accept a network connection
`stat`	read file statistics
`ioctl`	set I/O properties, or other miscellaneous functions
`mmap`	map a file to the process memory address space
`brk`	extend the heap pointer

For debugging with strace I’d suggest using the -c flag first so you get a feel for any particular system calls that stand out as being quite odd.

Maybe you notice a lot of network connections being opened and your app is showing signs of struggling with throughput. So you might then use strace without the -c so you could delve into the specific details.

If you’re debugging on a remote server that’s running a web app, you might attach strace to the running server process and then start manually curling endpoints to see what sort of reaction the server has.

Maybe you only want to look out for certain types of system calls. In that case you can utilise filters (-e). In the following example we’re only interested in the open, connect and access syscalls:

strace -e trace=open,connect,access lsof

You can also use ! to negate the filter (see man strace for more details)

ℹ️ NOTE

if you try to use grep instead of -e then you’ll need to ensure you redirect stderr to stdout or you wont see any output as strace sends to stderr by default; meaning you’d need to execute something like strace uptime 2>&1 | grep open

You might also find that using the -t flag useful for tracking when the call was made:

strace -t -e trace=access lsof

19:01:00 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
19:01:00 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
19:01:00 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)

As you can see strace is a really useful tool when the time comes.

…and the time will come.

By the way, you should be careful with backgrounded processes. If you attach to a backgrounded process running in the same shell instance as your strace execution, then you’ll be locked up.

ℹ️ NOTE

although strace is amazing, you might also want to read this article that discusses the oft-ignored performance overhead of using it in production

lsof

Summary: lists open files

So we actually saw this used earlier when looking at strace. It will display all open files, and any one who knows a bit about operating systems will realise that an open file could in fact be any of the following:

a regular file
a directory
a block special file
a character special file
an executing text reference
a library
a stream or a network file (Internet socket, NFS file or UNIX domain socket)

This means that lsof isn’t as pointless as you may have initially thought. Considering nearly everything in a *nix environment ‘is a file’ (including network sockets).

In order to use this command, let’s see some simple examples:

lsof -u <user>: filter results by specific user
lsof -i TCP:22 -n: lists all running processes on port 22 (ssh)
lsof -i 4: display only IPv4 network files

For example, with that last command (show the IPv4 network files) running on my Mac OS laptop, I see all sorts of processes such as: Slack, Chrome, Spotify etc. Lots of interesting information

This is one of those tools that might not get used very often, but when the right problem occurs it can be a real time saver being able to see what files your machine has open.

netstat

Summary: monitors network traffic

The netstat (network statistics) command is useful because it allows us to see network connections for TCP (both incoming and outgoing), routing tables, and a number of network interface and network protocol statistics.

This gives us an insight into what network communications are going on and whether certain services are talking to the right endpoints, how often and whether their connections are being established or terminated correctly.

Here are some useful examples you can try out:

netstat -a: show both listening and non-listening sockets (for TCP this means established connections)
netstat -l: show just listening sockets
netstat -lt: show just tcp sockets
netstat -lu: show just udp sockets
netstat -aep: extend to show the user and also the pids
netstat -aepn: don’t translate host names (e.g. show ip instead)
netstat -aepT: show host name but don’t truncate it
netstat -ax: show just UNIX domain sockets
netstat -st: shows summary of connections (useful for identifying TCP connection issues)
netstat -lc <n_seconds>: show listening sockets whilst continously refreshing every n seconds
netstat -atepn: nice ‘general’ all-rounder (see below for example output)

  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address               Foreign Address             
  tcp        0      0 0.0.0.0:8126                0.0.0.0:*                   
  tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   
  tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   
  tcp        0      0 127.0.0.1:24220             0.0.0.0:*                   
  tcp        0      0 10.6.4.51:57228             10.6.31.176:6379            
  tcp        0      0 10.6.4.51:57224             10.6.31.176:6379            
  tcp        0      0 10.6.4.51:8080              10.6.8.80:48205             
  tcp        0      0 10.6.4.51:57231             10.6.31.176:6379            
  tcp        0      0 10.6.4.51:57225             10.6.31.176:6379            
  tcp        0      0 10.6.4.51:8080              10.6.6.76:51764             
  tcp       53      0 10.6.4.51:56870             54.231.142.40:443           
  tcp        1      0 127.0.0.1:34704             127.0.0.1:8080              

  State       User       Inode      PID/Program name   
  LISTEN      0          8934       1071/statsd         
  LISTEN      498        10087      1355/puma 2.14.0 (t 
  LISTEN      0          8763       1196/sshd           
  LISTEN      497        9296       1307/ruby           
  ESTABLISHED 498        218757     1355/puma 2.14.0 (t 
  ESTABLISHED 498        218743     1355/puma 2.14.0 (t 
  ESTABLISHED 498        229190     1355/puma 2.14.0 (t 
  ESTABLISHED 498        218766     1355/puma 2.14.0 (t 
  ESTABLISHED 498        218747     1355/puma 2.14.0 (t 
  ESTABLISHED 498        10111      1355/puma 2.14.0 (t 
  ESTABLISHED 497        229141     1307/ruby           
  CLOSE_WAIT  48         226010     20286/httpd

netstat -r: shows routing table (see below for example output)

  Kernel IP routing table
  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  10.6.0.0        *               255.255.248.0   U         0 0          0 eth0
  link-local      *               255.255.0.0     U         0 0          0 eth0
  default         ip-xx-x-x-x.eu- 0.0.0.0         UG        0 0          0 eth0

Effectively, if you’ve any kind of network issues, then this tool can help you potentially identify where it’s coming from or going (or if it’s not coming from or going to the expected source/destination).

Update: an easy way to remember this (thanks Julia Evans - see honorable mentions is “tuna please” netstat -tunapl). You can use lsof -i -P on Mac OS

ifconfig

Summary: configure or review your network interfaces

The ifconfig command is used to configure or review your network interfaces and can help you identify if there is a problem with your network (such as no Ethernet or WiFi or maybe your connections are misbehaving due to misconfiguration).

Because networking is a big topic, and I’m not very good at it, I’ll refer you to the following article which provides a breakdown of typical data ouput: www.aboutlinux.info/2006/11/ifconfig-dissected-and-demystified

If you don’t want to configure a network interface, then running the ifconfig command without any arguments will display all existing network interfaces:

eth0      Link encap:Ethernet  HWaddr 0A:05:1E:A5:6F:FF  
          inet addr:10.6.4.51  Bcast:10.6.7.255  Mask:255.255.248.0
          inet6 addr: fe80::805:1eff:fea5:6fff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:8776319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4212889 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10239965628 (9.5 GiB)  TX bytes:10967533931 (10.2 GiB)
          Interrupt:155 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:341240 errors:0 dropped:0 overruns:0 frame:0
          TX packets:341240 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:31612016 (30.1 MiB)  TX bytes:31612016 (30.1 MiB)

We can see above that we have a single Ethernet card (eth0) and a loop back interface (lo)

In newer Linux OS’ eth<n> is replaced by p2p<n>
And on the Mac OS it becomes en<n>

If you want to find out which interface your WiFi is associated with, then you can use:

$ networksetup -listallhardwareports

Hardware Port: Wi-Fi
Device: en0
Ethernet Address: 38:f9:d3:9e:9f:2b

Hardware Port: Bluetooth PAN
Device: en6
Ethernet Address: 38:f9:d3:95:5c:fc

Hardware Port: Thunderbolt 1
Device: en1
Ethernet Address: 5a:00:5c:52:d9:01

Hardware Port: Thunderbolt 2
Device: en2
Ethernet Address: 5a:00:5c:52:d9:00

Hardware Port: Thunderbolt 3
Device: en3
Ethernet Address: 5a:00:5c:52:d9:05

Hardware Port: Thunderbolt 4
Device: en4
Ethernet Address: 5a:00:5c:52:d9:04

Hardware Port: Thunderbolt Bridge
Device: bridge0
Ethernet Address: 5a:00:5c:52:d9:01

VLAN Configurations
===================

For more information on Network ips and how they’re created (i.e. CIDRs), then you might be interested in this quick gist (but it’s only basic information, so you might have better luck with your own Googling)

iftop

Summary: monitors network traffic and displays table of bandwidth usage

The iftop command is useful for identifying things like “why is our connection so slow?”. It’s equivalent to a tool like top when understanding CPU usage.

The command listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts.

This utility isn’t available by default on any OS, so with Mac OS:

brew install iftop

With Ubuntu:

apt-get update && apt-get install iftop

With CentOS:

yum install wget libpcap -y
wget http://pkgs.repoforge.org/iftop/iftop-0.17-1.el6.rf.x86_64.rpm
rpm -ivh iftop-0.17-1.el6.rf.x86_64.rpm

Below is a simple command to get you started:

sudo iftop -P -i en1

ℹ️ NOTE

use ifconfig to find the interface you’re interested in

I personally find the standard output useful (as per image above), but if you press ? while the program is running you’ll see lots of additional options you can try:

Host display:                          General:
 n - toggle DNS host resolution         P - pause display
 s - toggle show source host            h - toggle this help display
 d - toggle show destination host       b - toggle bar graph display
 t - cycle line display mode            B - cycle bar graph average
                                        T - toggle cumulative line totals
Port display:                           j/k - scroll display
 N - toggle service resolution          f - edit filter code
 S - toggle show source port            l - set screen filter
 D - toggle show destination port       L - lin/log scales
 p - toggle port display                ! - shell command
                                        q - quit
Sorting:
 1/2/3 - sort by 1st/2nd/3rd column
 < - sort by source name
 > - sort by dest name
 o - freeze current order

iptraf

Summary: monitors network traffic (more visual than netstat, but not as detailed)

This utility isn’t available on Mac OS, so you’ll need Linux:

With Ubuntu:

apt-get update && apt-get install iptraf

With CentOS:

yum install iptraf

Because iptraf requires a lot of user interaction to get into different sections of the tool, I’ll have to just leave it up to you to explore.

But you’ll see options like IP traffic monitor, which can be useful viewing. Personally I don’t find iptraf as useful as tools such as iftop. But it’s here for you nonetheless.

tcpdump

Summary: network packet sniffer

tcpdump is a powerful and widely used command-line package analyzer, which is used to capture or filter TCP/IP packets received or transferred over a network on a specific interface.

We can save the output of this program into a pcap file format, that can then be viewed by either tcpdump itself or via an open source GUI based tool such as Wireshark that reads pcap format files and visualises the data.

The following are some examples to help you understand how to execute tcpdump, but be aware that on some hosts you might need to run it with sudo:

tcpdump -D: show available network interfaces
tcpdump -i eth0 -c 5: capture n packets from specific network interface
tcpdump -i eth0 -c 5 -w 0001.pcap: send output to pcap file
tcpdump -r 0001.pcap: read back out the pcap file content for later analysis
tcpdump -n -i eth0 -c 1: converts dns hostnames into ip addresses instead
tcpdump -n -i eth0 -c 10 tcp: capture only tcp packets
tcpdump -n -i eth0 -c 10 port 22 and port 80: capture packets from specific ports
tcpdump -i eth0 src <ip>: capture packets from a specific source ip
tcpdump -i eth0 dst <ip>: capture packets for a specific destination ip
tcpdump -i eth0 src port 80 or dst port 80: filter all HTTP traffic to or from port 80
tcpdump -i eth0 -s 0: include contents of each packet (not just the packet header)
tcpdump -vvv -s 0 -l -n port 53 -XX: watch all DNS traffic (which happens on port 53)

If you run tcpdump on a remote server, you’ll want to use the -w flag to record the data into a pcap file, this is so you can scp the file back to your local machine for later aggregation/analysis with either tcpdump itself (-r) or by importing the pcap file into another tool such as wireshark or tshark

The output of the program may look a little confusing but there is consistent structure you can look out for:

<date_time> <protocol> <src> > <dest>: Flags[<type>] <data>

ℹ️ NOTE

the > always sits between src and dest and indicates the direction of the request

Flags

Flags aren’t always present, depending on the network transfer that has been recorded, but when it is you’ll usually see one of the following:

[U] URG: used to identify incoming data as ‘urgent’. Such incoming segments do not have to wait until the previous segments are consumed by the receiving end but are sent directly and processed immediately
[A] ACK: the receiver will send an ACK that equals the senders sequence number
[P] PSH: push exists to ensure that the data is given the priority (used quite frequently at the beginning and end of a data transfer, affecting the way the data is handled at both ends)
[R] RST: used when a segment arrives that is not intended for the current connection (remote host rejects packet and resets connection)
[S] SYN: initially sent when establishing the classic 3-way handshake between two hosts (one sent by Host A and one sent back with ACK by Host B)
[F] FIN: used to tear down the virtual connections created using the previous flag (SYN)

Data

You’ll typically see more information within the ‘data’ output to indicate information about the packets being passed between hosts.

One such example would be seeing win specified. This is a sliding ‘window’ that represents the size of the packets being sent back and forth.

You’ll also typically see an ACK (acknowledgement packet), and that it won’t be sent until the previous packet has been completely received (this is to help with consistency of specific sequence numbers and the ability to signify packets have been missed).

The win is also used for flow control. An example of this is where one end of the communication is having trouble keeping up with the number of packets and so they modify the window buffer to be a smaller packet size to allow themselves to catch up with their processing of data.

Last Packet?

Although nothing to do with tcpdump, it is worth understanding (for the sake of debugging) that the FIN flag doesn’t necessarily indicate the last packet has just been sent. Here is an example to explain:

Host A receives what it believes to be the last data packet
So Host A sends a FIN to tell Host B it’s closing its connection, along with an ACK to acknowledge Host B’s last data packet
Host B sends an ACK to acknowledge receipt of Host A’s FIN
Host B then sends its own FIN and ACK to close its connection down
Host A sends a final ACK to acknowledge Host B’s last communication was received

wireshark

Summary: network packet sniffer and analyser (gui)

Wireshark is a network protocol analyzer.
It lets you see what’s happening on your network at a microscopic level.

There’s not a lot I can say about Wireshark other than you really will need to know how certain network protocols (such as TCP) work in order to understand the output that’s being recorded.

The output will look very similar to what tcpdump provides (depending on what network traffic you’re recording), and is a really useful tool for easier interrogation of pcap files created by tcpdump.

This utility isn’t available by default on any OS, so with Mac OS:

brew install wireshark --with-qt5

ℹ️ NOTE

sometimes the flags that are available change. So I would suggest running the command brew cat wireshark first to see what’s available first.

With Ubuntu:

apt-get update && apt-get install wireshark

With CentOS:

yum install wireshark

I typically have only ever used Wireshark with pcap files I’ve created via tcpdump so that’s what I’ll demonstrate here. There are two ways to open the pcap file in Wireshark.

execute wireshark from the terminal, which will open the gui, and from the gui interface select “Open Capture File”.
execute wireshark -r /path/to/pcap/file

ℹ️ NOTE

if you just want to use Wireshark to monitor all network traffic, then execute sudo wireshark -i <interface> (use ifconfig or sudo wireshark -D to see what interfaces are available). Once the gui is open it’ll have the specified interface pre-selected, so just double-click on it to start recording its traffic

Every time there is (for example) a HTTP request, there might end up being 200 TCP packets recorded, which can be (as you could imagine) difficult to recognize and make sense of manually.

But this problem can be simplified within Wireshark by clicking on “Statistics” and then “Conversations”, where it will organize all these disparate packets into TCP sessions for you. Thus making analysing the data much easier.

ℹ️ NOTE

we cover ‘filtering’ more in the next section about tshark, but one simple search for a HTTP GET header in your recorded traffic is frame contains "GET"

Docker?

One thing I did notice when trying to use this tool with Docker, was that you must run the container in privileged mode:

docker run --privileged

Otherwise you’ll see:

can't run /usr/sbin/dumpcap: Operation not permitted

Apparently --security-opt seccomp:unconfined is an alternative option.

The following is taken and paraphrased from the Docker website:

Doing this will allow Docker to access all devices on the host as well as set some configuration to allow the container to nearly all the same access to the host as processes running outside the containers on the host.

For more details, please refer to the documentation.

tshark

Summary: network packet sniffer and analyser (cli version of wireshark)

Tshark is a network protocol analyzer, but a terminal based one.
It lets you see what’s happening on your network at a microscopic level.

As mentioned above in the Wireshark section, to get access to tshark on the Mac OS you need to install Wireshark. There are separate installs though for other OS’.

With Ubuntu:

apt-get update && apt-get install tshark

With CentOS:

yum install tshark

I’ve only used tshark a few times and typically I find using the Wireshark gui better for interpretting tcpdump data (which has so far been one of my primary use cases; e.g. debugging traffic occuring on a remote server).

But the benefit of tshark, outside of those of us who love living inside the terminal, is the ability to automate analysis without the need for a gui. This means we can write scripts to process the data we’ve recorded and so becomes a very powerful tool.

Similar to Wireshark, if you want to read in a pcap file then you would use the -r flag to point to its location on your file system:

tshark -r /path/to/pcap/file

Also you can just run tshark and have it monitor all network traffic on your host machine or just a specific interface (using the -i flag). There is also the -D flag for having it show you all available interfaces you can listen on.

The tshark command knows a lot more about what’s going on inside your TCP packets than tcpdump and so you have more options available to you in order to filter out the data you’re interested in:

sudo tshark -i any \
            -R 'http.request.method == "GET"' \
            -T fields \
            -e http.request.method -e http.request.uri -e ip.dst

In the above example we’re ‘explicitly’ listening on all interfaces for network traffic. We then filter out all packets except those which are a HTTP GET request.

Now instead of just showing all the data for the HTTP GET requests that come through we use the -T flag to indicate that we’re interested only in ‘fields’ being displayed. We then can use the -e flag to indicate which subset of ‘fields’ we want to show.

In this case we’ve chosen to display the request method (i.e. GET), the URI that was associated with the GET and the ip address associated with it.

Filtering Example

One very real example I had recently was the need to inspect DNS traffic and to identify the TTL record values returned by the DNS resolutions happening on port 53.

I had used tcpdump to record the relevant traffic:

tcpdump -vvv -s 0 -l -n port 53 -XX -w dns-traffic.pcap

I then was able to pass this into tshark to automate the extraction of useful data, all without having to consult the Wireshark gui (meaning I could do this within the same environment as the data was recorded in if I wanted to):

tshark -r ~/dns-traffic.pcap -T fields -e dns.resp.ttl -e dns.resp.name

One useful trick some people aren’t aware of, is that if you are writing automation scripts with tshark but you’re unsure of how to access specific data fields, then just use Wireshark temporarily to figure out the fields you need.

For example, if you open the pcap in wireshark, you can find the filter you need by selecting the data manually via the UI and then right-click’ing the relevant data field and selecting “Prepare a Filter > Selected”. This will generate the exact value you would assign to the tshark -e flag.

ℹ️ NOTE

the filtering system syntax is called BPF (Berkeley Packet Filter) and you can find documentation here

telnet

Summary: utility for communicating with another host

Telnet is both a tool telnet and a Network Protocol of the same name: Telnet. The telnet program is used for interactive communication to a remote/external host on a given port. Once the connection to the remote host is established, an HTTP request can be sent to the host by typing it in the prompt.

Telnet’s usage nowadays is a little limited due to the massive success of protocols such as SSH, but it can be interesting to play around with (although I’ve never really had much of a ‘need’ for it myself).

ℹ️ NOTE

one such tool that is more useful in this respect is netcat which reads and writes data across network connections, using the TCP or UDP protocols

To use the command you type telnet <host> <port>. Once ‘connected’ to the host you need to provide a command to be executed, for example a GET request.

Below is an example of making a GET request. First we must connect:

telnet www.google.com 80

We will get the following response:

Trying 87.237.19.30...
Connected to www.google.com.
Escape character is '^]'.

From here we can provide our request:

GET #q=cars HTTP/1.1

ℹ️ NOTE

you need to press twice to send the request

From here we get the following response:

HTTP/1.1 302 Found
Location: http://www.google.co.uk/?gws_rd=cr&ei=k7BjV-GbFOLOgAbd3JbADw#q=cars
Cache-Control: private
Content-Type: text/html; charset=UTF-8
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Date: Fri, 17 Jun 2016 08:10:59 GMT
Server: gws
Content-Length: 268
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=80=123; expires=Sat, 17-Dec-2016 08:10:59 GMT; path=/; domain=.google.com; HttpOnly

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/?gws_rd=cr&amp;ei=k7BjV-GbFOLOgAbd3JbADw#q=cars">here</A>.
</BODY></HTML>
Connection closed by foreign host.

If we try a similar request with the BBC site, we’ll see we get a 404 not found:

$ telnet www.bbc.co.uk 80
Trying 212.58.244.66...
Connected to www.bbc.net.uk.
Escape character is '^]'.
GET /news HTTP/1.1

HTTP/1.1 404 Not Found
Content-Type: text/html
Date: Fri, 17 Jun 2016 08:13:29 GMT
Connection: Keep-Alive
Content-Length: 50591

Honorable mentions

A few months after writing this article I stumbled across a similar (but much better) post from Julia Evans. I then realised she has written lots of amazing posts, so you should check them out.

Here are some tools that she mentions that you should consider looking at:

dstat: prints network/disk usage every second
opensnoop: prints files being opened (but without strace perf concerns)
netcat: connect, listen, tunnel with ease
ngrep: provides grep functionality for the network
perf: records a program on an interval for later analysis (also perf top)

Conclusion

There you have it. A whirlwind run down of different terminal based debugging tools. The primary one’s (for me) being:

strace
tcpdump
wireshark
tshark

Obivously they all have their specialisms and unique features. I’ve not even scratched the surface of what they can do. If you know of anything really useful that I’ve missed, then let me know.

GitHub Pull Request Formatting

Mon, 22 Aug 2016 00:00:00 +0000

Introduction

What makes a good Pull Request?

The answer will depend on your team, and to a certain extent your organisation.

Currently I work at BuzzFeed, and prior to that I worked at the BBC.

Both have a large number of engineers and teams, but due to different organisational structures they have differing opinions on what constitutes a good pull request format.

These opinions aren’t happening at the organisational level either, but are very much ‘team’ specific. Teams work differently and so have different needs.

Below I discuss some ideas around what I’ve used in the past and what I’m using today and I’ll leave it as an exercise for the reader to determine what parts they decide to takeaway with them.

Why

Probably the most important part of a pull request is understanding why the change is needed in the first place.

What exactly are you changing and is it even needed?

Are you solving a problem that promotes a real business requirement, or are you just adding a nice feature that doesn’t actually serve to improve the end user’s experience?

Maybe the feature you’re adding is an internal improvement (e.g. refactor, dev tooling etc). That’s fine and I guess it comes down to how your team prioritises its work.

But taking a moment to stop and think about a new code change before you start working on it, is an important step to take.

Size

Pull requests should be small.

A large pull request that touches many different files across a project and has many different side effects, outcomes and responsibilities is extremely difficult to reason about from the perspective of the person reviewing the code.

Small pull requests allow for quicker reviews and merging. It promotes an iterative approach to implementing new features. It also helps to avoid conflicts when merging or rebasing.

We’re also able to ‘fail fast’ if our work priorities change and we need to switch gears or change direction altogether. We’re not trying to solve every problem all at once.

Process

Pull requests should be opened almost immediately, to allow for team feedback and help in direction (depending on what you need).

For me, this typically means making a single, small change and opening a pull request around it.

Once the pull request is open I utilise ‘labels’ to signify the state of the pull request: wip (work in progress), rtr (ready to review), rtm (ready to merge), help (I actively want engineers to chip in early and help me flesh out the design if they see something wrong).

I ensure I have a good pull request description that indicates what the problem is that the pull request solves and how it solves that problem. In markdown that would look something like:

**Problem**: ...
**Solution**: ...

**Notes**: 

...

**Todo**:

- [ ] ...
- [ ] ...
- [ ] ...

**Screen Shots**:

![image alt text](http://some-image)

As you can see above, this is a generic template. I like to have the problem and solution lines to be very concise and not to take up multiple lines.

The ‘notes’ section can be multiple lines so I keep it as an isolated section. This section can also include things like automated code linting or test suite coverage results.

The ‘todo’ section helps me to keep track of what tasks I have in order to complete this pull request, but it also helps other engineers to understand my thought process and see where I might be going before I even get there (allowing the team to ask questions or make suggestions as per the help label).

Finally, the ‘screen shots’ section is useful for those unfamiliar with the side effects of the change to be able to visually identify where the change appears or what it looks like. This isn’t always necessary, depending on the code change being made, but is useful for UI changes.

Communication

It’s important to notify fellow team members that your pull request exists, in order for them to provide appropriate feedback.

You can manually @<username> people within a comment in the pull request, or you could create a team in GitHub and then @<team-name> instead (helps especially for teams who like to rotate members across other teams within an organisation).

Merge Strategies

I’ve written in the past about different ‘git merge strategies’. My preferred way is to git merge --squash, and luckily GitHub’s UI provides a ‘one click’ way to squash merge a pull request.

That’s what I suggest using for merging pull requests into master, but you may have different requirements in your team/organisation, so pick whatever works best.

Conclusion

So this has been my preferred approach to creating a good pull request.

To recap, here is a top level look at the structure and concepts I suggest:

Small pull request
Utilise labels to indicate status
Consistent formatting:
- Gif (optional)
- Problem/Solution
- Notes (optional)
- Todos (optional)
- Screen shot images (optional)
Agree a merge strategy for team consistency

Big O for Beginners

Tue, 28 Jun 2016 00:00:00 +0000

Introduction

When you first start learning algorithms (Binary Search, Quick Sort, Breadth-first Search etc), you’ll quickly realise that in order to take advantage of these algorithms, you need to know how fast they are.

Otherwise, when presented with a programming problem in which you want to select an algorithm to use to solve that problem, how will you know which algorithm is more efficient?

One way to know how fast an algorithm is, would be to use the Big O notation.

Understanding Big O

Big O doesn’t tell you how fast in time (e.g. seconds) an algorithm is. Instead it informs you of the number of operations, and how those operations will grow over time.

Although we’ll see how to calculate the speed of operations later on in this post, the primary benefit is to see ‘at a glance’ the growth of operations as your data becomes larger.

So the O in “Big O” means “Operation”

This means in order for you to really understand Big O you’re going to need to know some maths. Now, this is OK. I’m genuinely terrible at maths, but you’ll see as we go along that it’s not as complicated as you might think.

Effectively there are two math concepts we need to know:

Logarithms
Factorials

You don’t even need to know that much about them. Only the bare minimum is required. So let’s make a start with Logarithms and then move onto Factorials afterwards. Once we understand those two concepts we can go back to Big O and start tying together some examples.

Logarithms

As I said, in order to understand Big O, you’ll need to understand how Logarithms work.

I’m terrible at Math, but luckily the awesome book “Grokking Algorithms: An Illustrated Guide” (which I highly recommend) helped me at least understand the basics of Logarithms; enough so that I could then go on to understand Big O.

In essence the Logarithm notation looks something like:

Log n (х)

ℹ️ NOTE

I had used a nice 𝑛 icon instead of n but on my phone I noticed it wasn’t showing :-/ so I had to change it to something that wasn’t a symbol.

…where the n is what’s called the “base” and х is the number you’re aiming for. But really what we’re interested in is the result of this calculation.

Logarithm Example

Imagine we have the following Logarthim:

Log5(100)

What this is effectively asking is:

“how many times do I need to multiple 5 by itself in order to reach the number 100?”

Let’s find out:

5*5*5 = 125

Looks like the result of our Logarithm would’ve been 3, because there were three 5’s used in order to get to a number that was equal or greater than 100.

In this case the calculation wasn’t exactly equal. By that I mean we went past 100 and ended up at 125. But that’s the essence of how to understand what a Logarithm is asking and how to calculate the result.

So we can see the calculation looks like this:

Log5(100) = 3

The 3 is effectively the worst case number of steps involved when calculating that particular item.

The number 100 in this case represents the number of items we have to execute our algorithm against. So if you were using an algorithm (such as a Binary Search; demonstrated later on below) and it was running over a collection of items, then the length of the items (in the above example Logarithm) would be 100.

We’ll see how this is useful in measuring an algorithm’s speed in a later section of this post. But for now let’s go and understand Factorials…

Factorials

Factorials are one of those things that can come in handy for a number of reasons. But generally they’re really useful for identifying ‘variations’.

Imagine you have three letters:

How many variations of these letters can you produce? So we have A, B, C as one variation. A, C, B would be another variation and then maybe B, A, C would be another etc. But how do you calculate how many variations there are?

The solution is the factorial: n! (where n is the number of items you have).

So in our example we had three items, so that would be written as 3! factorial.

Which really means:

3*2*1 = 6

So that’s six variations you have for three items.

But what if you have 10 items?

10*9*8*7*6*5*4*3*2*1 = 3,628,800

You can see that with a small number of items, the number of operations is massive. This is the total oposite of Logarithms which we looked at earlier (remember its growth of operations stayed consistently good when the collection grew).

Although Factorials serve a useful purpose (the example given in the book “Grokking Algorithms” is one called ‘The Travelling Salesperson’ - which is a problem that required calculating the quickest route the saleperson can take in order to visit n number of cities) they are probably the worst performing algorithm.

So if you see n! you should be wary.

Back to Big O

So now we understand how Logarithms and Factorials work we can come back to the Big O notation and understand that it’s really a simple visual wrapper around these different mathematical calculations.

O(х)

In the above snippet, х is a calculation.

If we were considering Logarithms, then it would look like the following:

O(Log n (х))

If we were considering Factorials, then it would look like the following:

O(n!)

It’s important noting that you’ll never see a specific calculation like O(Log5(100)) or O(3!) in Big O. It’ll always be an abstract version like O(Log n (х)) or O(n!) because Big O notation is a way of talking to other people and having a ‘common language’.

For example:

“Hey Bob, I don’t think we should use the Simple Search algorithm because it’s O(n). We’d be much better off using a Binary Search, as that’s O(Log n (х))”.

See how this gives us a common language. It’s similar to Design Patterns. Patterns can be implemented in a variety of different ways and yet we have common language for easily identifying code that follows a certain pattern (or when we think a particular pattern might be a good solution to a design problem, we have a common language for explaining the solution to someone else without needing to actually implement it first).

Now my mention of ‘Simple Search’ and ‘Binary Search’ might not mean much to you, as you might not know how these algorithms work. So let’s look at these two algorithms next and then after that you’ll hopefully understand why Big O helps us understand the performance of these algorithms.

Simple Search

Simple Search is probably the simplest algorithm you’ll ever learn. You’ll see why in just a moment…

Imagine you have a collection of twelve items (and these items are numbers), which are sorted/in order. You are required to locate a particular item within that collection.

Here is the collection:

[1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 20, 21]

The Simple Search approach is to loop over the collection one item at a time and check whether the current item matches the item you’re looking for.

So if the number you were looking for was 19, then that means you’ll first check 1, nope that’s not it. You’ll then check 3, nope that’s not it either… and keep going until you reach the number you’re looking for.

So what does this algorithm look like in Big O?

O(n)

What does Big O tell us?

Big O is telling us that this algorithm performs in ‘linear’ time. This means the number of operations increase linearly with the number of items in the collection. So in the above example, the worst case number of operations will be 12. But if the collection length was 100 items, then the worst case number of operations would be 100.

So in effect, the bigger the collection, the more expensive this algorithm becomes. With a very small collection it’s fast because of its simplicity, but beyond a small collection it’s a poor performing choice of algorithm. This is what Big O is telling us here.

Binary Search

Consider the same example as before, a collection of 12 items. The Binary Search algorithm is quite straight forward: you set the start and end indexes (usually zero for ‘start’, and the length of the collection for the ‘end’).

Now you locate the middle of the collection and check if the value you’re looking for either matches or is too low/high. If it matches, then hey great you’ve just reduced the number of operations by a large amount compared to the Simple Search.

If the middle item (our ‘guess’) is lower than the actual item we’re looking for, then you reassign the value of ‘start’ to be the middle index (the ‘end’ stays set to the length of the collection). You’ve now reduced the sliding window of items by half.

Alternatively if the middle item was larger than the item we’re looking for, then you reassign the value ‘end’ to be the middle index (the ‘start’ stays set to zero). You’ve again reduced the sliding window of items by half.

From here we ‘rinse/repeat’ until the next ‘middle of the collection’ selection is the item we’re looking for.

This is a much more efficient search algorithm compared to Simple Search.

Let’s see what this looks like in Big O notation…

O(Log n (х))

What does Big O tell us?

Big O is telling us that this algorithm works in ‘log time’, which means the performance improves as the size of the collection increases! You can tell this ‘at a glance’ with Big O syntax (especially as you now know how Logarithms work). This algorithm will perform well across a wide range of collection sizes.

Big O in effect tells us how the operations grow.

So for this particular algorithm example, the worst case number of operations is 4. Take a look at our earlier explanation of Logarithms if you’re unsure why that is, but in a non-abstract sense this would look like: O(Log2(12)). We’re dividing our collection in two (Log2) for each operation ((12)).

This is probably one of the best Big O’s you’ll come across as effectively the performance (and by that I mean the growth of the operations) stays consistently good as the size of the collection increases.

So if you have a collection of 1000 items, then that would result in (worst case) 10 operations required to locate the item you’re looking for:

2*2*2*2*2*2*2*2*2*2 = 1024

How about a collection of a million items? That would be result in (worst case) 20 (yes 20!) operations to find the item you were looking for amongst one million items. That’s incredible.

The Travelling Salesperson

Not much to say here that we haven’t already mentioned earlier when talking about Factorials. This problem is about calculating the number of different routes someone can take in order to ensure they reach all the specified number of cities, and then working out which route was quickest.

In Big O notation this looks like:

O(n!)

What does Big O tell us?

Big O is telling us this algorithm results in ‘factorial time’, and this is one of the worst performing algorithms known to mankind. Apparently it baffles the maths community, in that there isn’t actually a better algorithm to solve this problem.

So as we saw earlier, if there were ten cities to visit and we need to identify the quickest route to visit all ten cities, it would take us approximately 3,628,800 operations to just calculate all the variations, before we could identify which one was quickest.

Calculating Operation Speed

In order to calculate the speed of an algorithm, we need to know the worst case number of operations. This is what Big O in effect gives us (whether it be linear time, log time or factorial time). So how do we calculate the speed based on the number of operations?

First we need to know how many operations can be executed within one second.

The answer to that question is ten operations:

1 second / 0.1 = 10 operations

ℹ️ NOTE

your computer can handle more than 10 operations a second, but as we’ve said before, Big O is about worst case scenarios and so although not realistic; this number does give us a nice base line to work from

We can then calculate the time associated with a number of operations.

Imagine we have the following Big O:

O(Log n (х))

If this was a collection of 16 items, then the non-abstract version would look something like:

O(Log2(16))

We know from our earlier discussions that this would result in a worst case result of 4 operations to find the item we’re searching for.

Remember: 2x2x2x2 = 16
That’s 4 times we multipled 2 by itself to reach 16

We also now know that a single operation takes 0.1 of a second.

So with this in mind we can calculate the speed of O(Log2(16)) as being: 0.4 seconds

0.1 * 4 operations = 0.4

If we were using the Simple Search algorithm (which is O(n)) on a collection of 16 items, then we know that this would take 1.6 seconds to complete:

0.1 * 16 operations = 1.6

Arrays vs Linked Lists

Let’s consider what Big O looks like for the Array and Linked List data structures.

An Array supports ‘index access’, meaning you can jump straight to an Array index. Whereas with Linked Lists you have to traverse the entire list in order to locate a specific item.

ℹ️ NOTE

another difference comes in memory management. Arrays require n number of memory ‘slots’ to be next to each other, whereas Linked List memory can be sparse and spread out due to how it implements its internal chaining of nodes. This is why Arrays are typically considered to be ‘fixed size’ and not easily expanded, because expansion of the Array’s size could potentially require an expensive movement of the Array to a new location in memory in order to faciliate a new index and yet still have memory slots side-by-side

This suggests that Array lookups are O(1), known as ‘constant time’ because the lookup growth stays the same no matter the length of the collection (i.e. it’s constant).

Linked List lookups on the other hand are O(n), which we already know is ‘linear time’ (remember this is how the Simple Search algorithm performed).

But what about new data insertions? Well, with an Array you insert new items at the end of the Array, and because of its index access it would indicate O(1). But if you’re inserting an item into the middle of the Array, then this changes to O(n) because of the internal implementation of Arrays in memory, it means you need to re-order all the following items, which is expensive (hence the side-effect of inserting into the middle is really more like linear time).

Linked List insertions are generally O(1) if inserting at the beginning or end of the list, but more like O(n) + O(1) if inserting into the middle of the list because you have to traverse the list first and then insert your new item.

Selection Sort

The ‘selection sort’ algorithm sorts an unordered list by looping over the list n number of times, and for each loop it identifies either the smallest or largest element (which - smallest/largest - depends on how you’re hoping to sort your list: do you want ascending or descending order). But ultimately you’ll end up constructing a new ‘ordered’ list.

Specifically you loop a number of times to match the collection length. Then you loop the collection looking for smallest/largest item. Then you mutate the collection so it’s smaller by one.

This ends up being O(n₂) (n to the power of 2, or O(n * n)).

ℹ️ NOTE

although you’re looping over the collection multiple times, you are in fact looping over a slightly smaller collection each time. But it’s still considered O(n) for each time you loop over the collection because regardless of how many items are in there you treat it as abstract

So if your collection is 10 items long, then it would calculate as follows:

10 * 10 (i.e. n₂ meaning: 10 to the power of 2) = 100 operations
0.1 * 100 = 10 seconds

Quick Sort

The ‘quick sort’ algorithm achieves the same result as ‘selection sort’, but is much faster. This particular algorithm sorts an unordered list using recusion instead.

Specifically it uses the D&C (Divide and Conquer) approach to problem solving.

The process is as follows:

You pick a ‘pivot’ (a random array index)
Loop the array storing items less than the pivot
Loop the array storing items greater than the pivot
Assuming the ‘less’ and ‘greater’ collections are already sorted
You can now return ‘less’ + pivot + ‘greater’

In reality you’ll use recursion to then sort both the ‘less’ and ‘greater’ arrays using the same algorithm.

This ends up being O(n₂) in the worst case, but can be O(n Log₂ n) in the better case.

The explanation for this, is that the quicksort function takes in a single collection and loops over it. In the process it then splits the collection into three chunks (less, pivot and greater) and it recursively calls itself (i.e. quicksort calls quicksort) on the ‘less’ and ‘greater’ chunks, subsequently looping over those collections as well.

Big O notation helps us to understand the hidden complexity of the algorithm. If you see something like O(n₂), then you know that there are nested loops or some kind of recursion happening in order to cause that.

Now it’s worth being aware that Quick Sort’s performance is dependent on the pivot you choose. Take a look at some of the example implementations of the quick sort algorithm in the reference list below “Gist: Algorithms in Python”.

There you’ll see we have three implementations: one where we pick the first index every time as the pivot, one where we pick the middle index every time and one where we pick an index at random.

Picking an index at random will give you the best chance of high performance. If you, for example, always pick the first index then you’ll have a situation where you’re potentially sorting items unnecessarily.

So in the best case scenario, if your collection is 10 items long, then it would calculate as follows:

10 * 4 (Log₂10 == 2*2*2*2) = 40 operations
0.1 * 40 = 4 seconds

But in the worst case scenario, if your collection was 10 items long, then it could calculate as follows:

10 * 10 = 100 operations
0.1 * 100 = 10 seconds

Conclusion

This has been a very basic introduction to the concept of Big O. Hopefully you’ve found it useful and have a greater appreciation for what Big O offers in the way of understanding the performance of particular algorithms (although we’ve only really looked at a very small selection).

We’ve seen the following Big O types:

O(Log n (x)): log time
O(n!): factorial time
O(n): linear time
O(1): constant time
O(n * n) (also known as O(n₂)): selection sort example
O(n Log n (x)): quick sort example

There are many more algorithms and calculations for Big O, and as I learn them I’ll be sure to update this blog post accordingly. If in the mean time you notice any mistakes, then please feel free to let me know.

References

The Perfect Developer Qualities

Fri, 27 May 2016 00:00:00 +0000

For me the perfect developer (if there is such a person) has these qualities:

Friendly: is respected and liked by all they work with and are always approachable (even in times of stress)
Humble: has great humilty and is not driven by ego
Calm: doesn’t get emotive within discussions (including discussions that are both in their favour and those that aren’t)
Understanding: appreciates that business requirements do change regularly and that there are no perfect scenarios; so is able to adapt to problematic situations in the appropriate manner
Agile: recognises when they are potentially moving down a rabbit hole/time sink/yak shave and will successfully re-evaluate the situation and refocus their attention
Patient: appreciates that no dev is born equal and so varying soft/practical skills will be encountered
Experienced: has a wide ranging skill set with relevant practical experience and most importantly realises the fundamentals of simple code design and recognised patterns
Consistent: by being consistent in their development approach they faciliate easier rotation of developers when resources are restricted/restructured
Wise: knows when and where a particular technology is appropriate to use (and when not appropriate)
Analytical: is pragmatic and can break down complex problems into logical smaller tasks
Articulate: able to communicate clearly their ideas to people of all technical levels
Open: continuous documentation and visibility into design and development progress; ensuring good communication
Reasonable: understands a problem’s sense of proportion; what’s important and what isn’t (e.g. when something is a big deal and worth pressing; and when it isn’t and so not prematurely raising red flags)
Passionate: enjoys their craft; always keen/open to new experiences and learning new things