Intended Audience

I’m assuming you are in the stage of learning Go. Therefore, you should have Go installed on your machine. If not, please refer to the Appendix section for installation links.

Before we get started, I need to get you on the same page. Throughout this entire article, I will be using the go version 1.18.3. So, I’d recommend you either use the version that I use, version 1.11, or any above to grasp everything properly.

What is GOROOT?

GOROOT is the variable that defines where Go SDK is located. This is where Go’s code, compiler, and rest of the required tooling lives. This folder does not hold our source code. The $GOROOT is something similar to /usr/local/go or /opt/homebrew/Cellar/go/1.X.X/bin.

In older versions of Go, we set $GOROOT manually. But in newer versions, we don’t need to set up the $GOROOT variable unless you use different Go versions. If you install Go in a different path, then export the variable $GOROOT in your shell’s default profile (i.e., .zshrc, .profile).

What is GOPATH?

To explain this, we need to travel back in time. Let’s see how things were before Go 1.11.

How old GOPATH works

When Go was first introduced in 2009, Go authors required you to organize the Go code specifically when working with the go tool. A convention to say. Here’s some simplified information I borrowed from the docs.

• Go programmers typically keep all their Go codes in a single workspace.
• A workspace contains many version control repositories (i.e., managed by Git, Bitbucket, etc.).
• Each repository contains one or more packages.
• Each package consists of Go source files in a single directory.
• The path to a package’s directory determines its import path.

Go authors had this notion called a single workspace directory. It is very different from other programming language environments (i.e., C++) in which the project has a separate workspace and can be multiple workspaces closely tied to version-controlled repositories.

But in Golang, a workspace refers to a directory hierarchy with three directories at its root.

To give a rough idea of how a workspace looks in practice, here’s an example:

pkg/
    darwin_arm64/
        golang.org/x/
            image/
                bmp.a            # compiled object
            example/
                stringutil.a     # compiled object
bin/
    hello                        # command executable
    outyet                       # command executable
src/
    golang.org/x/example/
        .git/                    # Git repository metadata
        hello/
            hello.go             # command source
        outyet/
            main.go              # command source
            main_test.go         # test source
        stringutil/
            reverse.go           # package source
            reverse_test.go      # test source
    golang.org/x/image/
        .git/                    # Git repository metadata
        bmp/
            reader.go            # package source
            writer.go            # package source

    ... (many more repositories and packages omitted) ...

Well, now let’s take this convention into practice and understand how it was before back then.

HolUp — Wait a Minute

From this point onwards, we will look at how GOPATH used to be in earlier versions before 1.11. And for that, we need to turn off module-aware mode from environment variables. You can check whether you have it enabled or disabled by using the following command.

go env GO111MODULE

If you don’t see the value as off, I want you to overwrite the default value by executing go env -w GO111MODULE=off to explicitly turn off the module-aware mode. Now we can make sure that we use the semantics before 1.11.

By default, Go sets auto, "" (empty string), or on as values of GO111MODULE depending on the version you are using.

Sometimes GO111MODULE environment variable is a pain in the arse. Gophers rely on this environment variable to change how Go imports packages. As I said, it depends on the version you are using. Go and its module-aware mode semantics change with different Go versions (well, slightly).

GO111MODULE in Go 1.11 and 1.12

GO111MODULE in Go 1.13

GO111MODULE in Go 1.14 and 1.15

Has the same semantics as Go version 1.13.

GO111MODULE in Go 1.16

In this version, if you see GO111MODULE as GO111MODULE="" or GO111MODULE= as an empty string or empty value it simply means GO111MODULE=on. I found out this via a issue comment on golang repo. It applies to both 1.15, 1.16 and beyond.

GO111MODULE in Go 1.17

Has the same semantics as Go version 1.16.

GO111MODULE in Go 1.18

Has the same semantics as Go version 1.17.

However, now go get no longer builds or installs packages in module-aware mode. go get is now dedicated to adjusting dependencies in go.mod. Effectively, the -d flag is always enabled by default. Please refer the go cmd changelist in the 1.18 release notes for more information.

Now that Go module-aware mode is disabled, the packages we develop and install should be in $GOPATH so that the Go build system knows where the imported packages are.

Now go tool expects you to keep your project and source files in GOPATH/src. And go-tool uses pkg/ for compiled packages and the bin/ for executables. This gives you all the necessary files for your development, and go-tool can resolve packages you have imported into your project.

Why learn if it’s obsolete?

Well, I’m glad you asked! Frankly, you don’t need to. Especially if you were a Gopher from the very beginning. But if you’re just starting out, you might run into slight hiccups here and there from time to time because we still have content that refers to older Go $GOPATH behavior. So, it’s always better to know the history.

Okay, now let’s see an example.

We can import third-party packages (i.e., libraries written Go, C, or even C++) or our own custom packages to our programs. For example, consider the below application.

Let’s say we want to create a calculator application. And along the way, publish our calculator operations as a reusable module so that other developers can reuse them. And make the interface and the logic separately in a different package as a driver application. And driver uses a third-party package called chalk to change the output colors.

Sounds easy, right? Let’s see how we can do that.

mkdir -p $GOPATH/src/operations     # create directory
cd  $GOPATH/src/operations          # navigate to the directory
touch operations.go                 # create package file

Open the operations.go file in your editor and paste in the following source.

package operations

func Add(a int, b int) int {
	return a + b
}

func Sub(a int, b int) int {
	return a - b
}

Okay, we have already created our reusable operations above. But how can we practically compile and use it in other projects?

Well, obviously we need another package to do so. But first, we need to execute go install inside the $GOPATH/src/operations package to create a compiled binary to use in other applications.

cd $GOPATH/src/operations && go install

If you navigate to $GOPATH/pkg, you will see that operations.a compiled binary file will be generated in $GOPATH/pkg/{GOOS}_${GOARCH} directory.

├── bin
├── pkg
│   └── darwin_arm64
│       └── operations.a
└── src
    └── operations
        └── operations.go

5 directories, 2 files

Now that we have a binary file, we can actually go ahead and create a new application package called calcapp¹.

mkdir -p $GOPATH/src/calcapp    # create directory
cd $GOPATH/src/calcapp          # navigate to application directory
touch main.go                   # create driver main file

Remember that we want the third-party library called chalk to format our output. So let’s go ahead and install that too.

go get github.com/ttacon/chalk

There is quite a bit happening in the background. First, without Go 11 modules enabled, the package we get from go get should be in $GOPATH so that the Go build system knows where the imported packages are.

├── bin
├── pkg
│   └── darwin_arm64
│       ├── github.com
│       │   └── ttacon
│       │       └── chalk.a
│       └── operations.a
└── src
    ├── calcapp
    │   └── main.go
    ├── github.com
    │   └── ttacon
    │       └── chalk
    │           ├── LICENSE
    │           ├── README.md
    │           ├── chalk.go
    │           ├── chalk_test.go
    │           ├── doc.go
    │           ├── examples
    │           │   └── main.go
    │           └── img
    │               └── chalk_example.png
    └── operations
        └── operations.go

13 directories, 11 files

First, Go fetches the package chalk and then puts its source under $GOPATH/src in a domain/org/package manner. And it installs the package and places its compiled binary in $GOPATH/pkg/${GOOS}_${GOARCH} in the same way as we talked about.

Now we can start writing our driver application.

package main

import (
	"fmt"
	"github.com/ttacon/chalk"
	"operations"
)

func main() {
	log(fmt.Sprintf("Add: %d", operations.Add(2, 2)))
	log(fmt.Sprintf("Subtract: %d", operations.Sub(4, 2)))
}

func log(message string) {
	fmt.Println(
		chalk.Green,
		message,
		chalk.ResetColor,
	)
}

Now we can run the driver application by executing go run main.go and see the output.

$ go run main.go
    Add: 4
    Subtract: 2

Now, if you execute go install and check /bin, you can see that the executable is put. Navigate to $GOPATH/src/calcapp and execute the following.

$ go install && tree $GOPATH

/Users/yasin/go
├── bin
│   └── calcapp
├── pkg
│   └── darwin_arm64
│       ├── github.com
│       │   └── ttacon
│       │       └── chalk.a
│       └── operations.a
└── src
    ├── calcapp
    │   └── main.go
    ├── github.com
    │   └── ttacon
    │       └── chalk
    │           ├── LICENSE
    │           ├── README.md
    │           ├── chalk.go
    │           ├── chalk_test.go
    │           ├── doc.go
    │           ├── examples
    │           │   └── main.go
    │           └── img
    │               └── chalk_example.png
    └── operations
        └── operations.go

13 directories, 12 files
$ ./calcapp  # Works because I have added $GOPATH/bin to $PATH
    Add: 4
    Subtract: 2

With this, we explored all subdirectories src/, pkg/, and bin/ in the root workspace directory. Well, that’s not it. We have a couple of more things left to learn about the old $GOPATH.

Can we place our project outside $GOPATH?

When I started learning Go back in 2017, I didn’t put my go files in $GOPATH. My dumbass didn’t refer the docs properly. But it turns out, it strangely compiled the code, and it worked for one main package. Even when GO111MODULE is disabled or in older versions of Go, we can place our projects outside the go path. This is what I mean:

Here’s a working example

I’m implementing a project named myproject, with a package main and including two files main.go and some_functions.go as follows:

mkdir -p $HOME/Desktop/myproject
cd $HOME/Desktop/myproject
touch main.go some_functions.go

package main

import "fmt"

func Test1() {
	fmt.Printf("hello ")
}

func Test2() {
	fmt.Printf("world!\n")
}

package main

func main() {
	Test1()
	Test2()
}

And run it like this:

$ cd $HOME/Desktop/myproject && go run .
hello world!

And voilà! Surprisingly, the program runs, even when the project is outside $GOPATH. Actually, all projects must be in $GOPATH is due to sub-packages. In the above example, we only had the main package. So, it makes sense why it worked. And one main package is not the case, so it is not confined by this limitation. But when you add another package to the project, things get a bit fussy.

Here’s a problematic example

mkdir -p $HOME/Desktop/myproject2
cd $HOME/Desktop/myproject2 && touch main.go
mkdir functions && cd functions && touch some_functions.go

package functions

import "fmt"

func Test1() {
	fmt.Printf("hello ")
}

func Test2() {
	fmt.Printf("world!\n")
}

package main

import "myproject2/functions" // ❌ Compilation error.
// import "functions" // ❌ Big nope.

func main() {
	functions.Test1()
	functions.Test2()
}

Without the Go module feature enabled, we cannot specify our functions/ package in the main package, so we cannot find out which should be in the import statement “function” location and cannot build this project. The only way to make it work without Go modules was to move the project into $GOPATH/src like so:

$ tree $GOPATH

/Users/yasin/go
├── bin
│       ...( executables ) ...
├── pkg
│       ...( compiled objects ) ...
└── src
    ├── myproject2
    │   ├── functions
    │   │   └── some_functions.go
    │   └── main.go
    |
     ...( more packages ) ...

15 directories, 14 files

package main

import "myproject2/functions" // 🎉

func main() {
	functions.Test1()
	functions.Test2()
}

This was a strictly opinionated approach back in the day, and every Gopher had to follow this convention and maintain their source like this back in the day. Conceptually, this allowed Gophers to link any go code at any instant of time without ambiguity. Well seems pretty reasonable, isn’t it? But, NO!.

Well, what’s the problem then?

The above $GOPATH approach worked well for a cohesive, more extensive monorepos that doesn’t rely on third-party packages².

Imagine you have semantic versioning in third-party packages and including your own. So with time, it will get hairy real quick. Without any versioning for the packages, it led to all this pain of managing different tags in the source.

Because of this, Go authors decided to introduce the GO111MODULE environment variable to handle it. Before Go 1.11, authors didn’t consider shipping the go tool with a package manager. Instead, we used go get to fetch all the sources by using their repository import path and placing them in $GOPATH/src. Since there was no package manager or any versioning, the master branch would represent a stable version of the package.

When Google released Go 1.11, they said: This release adds preliminary support for a new concept called “modules,” an alternative to GOPATH with integrated support for versioning and package distribution. Go 1.11 was released on August 24, 2018.

Gotcha

You might see references to Go Modules as vgo. Basically, it means the same thing. And it implies Versioned Go. Instead of using the $GOPATH for storing a single git checkout of every package, Go Modules stores tagged versions with go.mod keeping track of each package’s version.

However, Go runtime still does use $GOPATH as a download directory for Go packages. To make Google’s saying correct, the Go module does not entirely replace $GOPATH. The go tool uses it for package distribution and versioning. The main goal of go modules is to distribute modules in a much more streamlined way. And now we are no longer confined to GOPATH. So, placing sources under src/ folder is ineffective and is not the best practice when you have module-aware mode enabled.

The interaction between the GOPATH behavior and the Go Modules behavior has become one of Go’s most significant turning points. Finally, now we can learn how it works in practice.

How GOPATH + GO111MODULE works

Now it’s time to see how GOPATH works with Go Modules. I’ll give you a similar example we tried above and modify it to cover the things I mention below.

• How to import locally created modules into a project.
• How to use remote modules installed via go get.
• How to use module sub-packages.

Switch to Go Modules

From this point onwards, we are using the modern approach. We are going to turn on GO111MODULE from environment variables.

go env -w GO111MODULE=on

Also, to increase the clarity of the tutorial, I will be cleaning up my $GOPATH and previous examples from my workspace.

sudo rm -rf ~/Desktop/myproject $GOPATH/* && cd $GOPATH && mkdir -p src pkg bin

Go ahead and create a workspace anywhere³. First, let’s start off with the operations package.

DIR="$HOME/Desktop/go_workspace"                # My root workspace
mkdir -p $DIR/operations && cd $DIR/operations  # Create package directory.
go mod init operations                          # Initialize go module.
touch $DIR/operations/operations.go             # Create the source file.

Paste the following source in operations.go.

package operations

func Add(a int, b int) int {
	return a + b
}

func Sub(a int, b int) int {
	return a - b
}

Then, let’s write the driver application calcapp.

mkdir -p $DIR/calcapp $DIR/calcapp/formatters                     # Create package directory.
cd $DIR/calcapp && go mod init calcapp                            # Initialize go module.
touch $DIR/calcapp/formatters/formatters.go $DIR/calcapp/main.go  # Create source files.

Also, for the driver application, we need a third-party package called chalk.

go install github.com/ttacon/chalk  # Installs the chalk package.
go mod tidy                         # Sync sums

Paste the following source in formatters.go.

package formatters

import (
	"fmt"
	"github.com/ttacon/chalk"
)

func Red(message string) {
	fmt.Println(
		chalk.Red,
		message,
		chalk.ResetColor,
	)

}

func Green(message string) {
	fmt.Println(
		chalk.Green,
		message,
		chalk.ResetColor,
	)
}

Now notice that we have a custom local package called operations. We need to import that package into our calcapp to make it work. So what do we do? To point to the local version of a dependency in Go rather than the one over the web, we use the replace keyword within the go.mod file.

The replace line goes above your require statements, like so:

module calcapp

go 1.17

// Can be github.com/yourname/operations
replace yasint.dev/operations => ../operations

// 💡 The actual semantic version hash will be different in yours.
require github.com/ttacon/chalk v0.0.0-20160626202418-22c06c80ed31
require yasint.dev/operations v0.0.0

And now, when you compile calcapp module using go install, it will use your local code rather than resolve a non-existing web dependency.

Replace Directive

According to the docs, when using replace, you need to make sure that the code you’re pointing to also has a go.mod file. If not, initialize a module using go mod init.

We can safely paste the following source in main.go.

package main

import (
	"calcapp/formatters"
	"flag"
	"fmt"
	"yasint.dev/operations"
)

func main() {

	isSubtraction := flag.Bool("sub", false, "subtraction operation")
	aValue := flag.Int("a", 0, "a value")
	bValue := flag.Int("b", 0, "b value")

	flag.Parse()

	if *isSubtraction {
		formatters.Red(
			fmt.Sprintf(
				"Subtraction: %d",
				operations.Sub(*aValue, *bValue),
			),
		)
	} else {
		formatters.Green(
			fmt.Sprintf(
				"Addition: %d",
				operations.Add(*aValue, *bValue),
			),
		)
	}

}

Did you notice? We can directly import our package by a path like calcapp/formatters and even reference local modules effortlessly! How cool is that? Go mod is more intelligent than this. It can even recursively resolve multiple nested sub-packages.

cd ~/Desktop/go_workspace/calcapp
go install
calcapp -a 10 -b 10     # => Addition: 20
calcapp -sub -a 10 -b 5 # => Subtraction: 5

Conclusion

Now we know with Go module-aware mode enabled, Go projects are no longer confined to $GOPATH. Meaning Go never restricts the structure or the location of Go projects. Go module alleviates versioning and module resolution constraints elegantly. I hope now you have a better understanding of $GOPATH and $GOROOT.

Thanks for reading! Until next time!

Appendix

macOS installation (Apple Silicon)

We can use homebrew to install golang. It will take care which binaries should be installed for different cpu architectures.

Install Golang

brew install golang

Check installation path

Once you installed, brew has a command that you can check where it exactly installed. By executing brew info golang you will get a similar output like the following.

go: stable 1.17.5 (bottled), HEAD
Open source programming language to build simple/reliable/efficient software
https://go.dev/
/opt/homebrew/Cellar/go/1.17.5 (10,825 files, 564.3MB) *
...

Verify go

Check whether the go command is working properly to verify you have it in the path.

go version # Outputs => go version go1.17.5 darwin/arm64

Tip 💡: Execute command -v go to check the command’s path.

Scrumptious bits

Usually when you install a package via brew, it will place the binaries in /opt/homebrew/Cellar/. Then after every folder under that is symlinked to /opt/homebrew/opt and, the go command will be automatically symlinked via /opt/homebrew/bin.

In Apple Silicon based macs we have to append the /opt/homebrew/bin to your $PATH variable to work everything correctly (Kurt B, Stackoverflow).

Other installations

Well, for other systems official Golang installation guide is far more than enough.