How to configure an application?
How to add a command-line option to a program?
How to create and parse environment variables?
Environment variable
Unmarshal
Command-line options
When we write programs for ourselves, we are (often) writing directly in the source code some important options, like, for instance, the port on which our server will listen, the name of the database we are using... But what if you attempt to share your program with others? There is a significant probability that your users will not use the same database name as you are using in the development phase; they will also want to use another port for their server!
\nAnd what about security! Do you want everybody to know your database password? You should not commit credentials in your code.
\nIn this section, we will cover how you can configure your applications.
\n\nAn application is configurable when it offers users the possibility of control the aspects of execution
In many open-source applications, the configuration is handled via key-value data structure
In a real-world application, you often find a class (or a type struct) that exposes the configuration options. This class often parses a file to attribute values to each option. Applications often propose command-line options to adjust configurations.
\n\nTo define a named command-line option, we can use the standard package flag.
Let’s take an example. You are building a REST API. Your application will expose a web server; you want the listening port to be configurable by the user.
\nThe first step is to define the new flag :
\n// configuration/cli/main.go \n\nport := flag.Int("port", 4242, "the port on which the server will listen")The first argument is the flag name; the second one is the default value; the third is the help text. The returned variable will be a pointer to an integer.
\nThere is an alternative syntax available. You first define a variable, then you pass a pointer to that variable as argument of the IntVar function :
var port int\nflag.IntVar(&port, "port", 4242, "the port on which the server will listen")The next step is to parse the flags :
\nflag.Parse()The function Parse has to be called before using the variables that you defined. Internally it will iterate over all the arguments given in the command line and assign values to the flags you have defined.
In the first version (flag.Int) the variable will be a pointer to an integer (*int). In the second version (flag.Intvar) will be of type integer (int). This specificity changes the way you use the variable afterward:
// version 1 : Int\n\nport := flag.Int("port", 4242, "the port on which the server will listen")\nflag.Parse()\nfmt.Printf("%d",*port)\n\n// version 2 : IntVar\n\nvar port2 int flag.IntVar(&port2, "port2", 4242, "the port on which the server will listen") flag.Parse()\nfmt.Printf("%d\\n",port2)The flag package expose functions to add flags that have different types. Here is a list of the types on the left and the flag package’s corresponding functions.
Int64, Int64Var
\nString, StringVar
\nUint, UintVar
\nUint64, Uint64Var
\nFloat64, Float64Var
\nDuration, DurationVar
\nBool, BoolVar
\nTo pass the specified flags to your program you have two options :
\n$ myCompiledGoProgram -port=4242or
\n$ myCompiledGoProgram -port 4242This last usage is forbidden for boolean flags. Instead, the user of your application can set the flag to true by simply writing :
\n$ myCompiledGoProgram -myBoolFlagBy doing so, the variable behind the flag will be set to true.
\nTo get a view of all flags available, you can use the help flag (which is added by default to your program !)
\n$ ./myCompiledGoProgram -help\nUsage of ./myCompiledProgram:\n -myBool\n a test boolean\n -port int\n the port on which the server will listen (default 4242)\n -port2 int\n the port on which the server will listen (default 4242)Some cloud services support configuration via environment variables. Environment variables are easy to set with the export command. To create an environment variable named MYVAR, just type the following command in your terminal1 :
\n$ export MYVAR=valueIf you want to pass environment variables to your application, use the following syntax
\n$ export MYVAR=test && export MYVAR2=test2 && ./myCompiledProgramHere we are setting two environment variables, and then we launch the program myCompiledProgram.
To retrieve the value of an environment variable, you can use the os standard package. The package exposes a function named Getenv. This function takes the name of the variable as argument and returns its value (as a string) :
// configuration/env/main.go \nmyvar := os.Getenv("MYVAR")\nmyvar2 := os.Getenv("MYVAR2")\nfmt.Printf("myvar : '%s'\\n", myvar)\nfmt.Printf("myvar2 :'%s'\\n", myvar2)Another method allows you to retrieve environment variables : LookupEnv. It’s better than the previous one because it will inform you if the variable is not present :
// configuration/env/main.go \nport, found := os.LookupEnv("DB_PORT")\nif !found {\n log.Fatal("impossible to start up, DB_PORT env var is mandatory")\n}\nportParsed, err := strconv.ParseUint(port, 10, 8)\nif err != nil {\n log.Fatalf("impossible to parse db port: %s", err)\n}\nlog.Println(portParsed)LookupEnv will check if the environment variable exists
If it does not exists, the second result (found) will be equal to false
In the previous example, we parse the port retrieved into an uint16 (base 10) with strconv.ParseUint
You can also get all the environment variables with the function Environ(), which will return a slice of strings :
\nfmt.Println(os.Environ())The returned slice is composed of all the variables in the format “key=value”. Each element of the slice is a key-value pair. The package does not isolate the value from the key. You have to parse the slice values.
\n\nThe os package exposes the os.Setenv function :
err := os.Setenv("MYVAR3","test3")\nif err != nil {\n panic(err)\n}os.Setenv takes two arguments :
the name of the variable to set
its value.
Note that this can generate an error. You have to handle that error.
\n\nConfiguration can also be saved in a file and parsed automatically by the application.
\nThe format of the file depends on the needs of the application. If you need a complex configuration with levels and hierarchy (in the fashion of the Windows Registry), then you might need to think about formats like JSON, YAML, or XML.
\nLet’s take the example of a JSON configuration file :
\n{\n "server": {\n "host": "localhost",\n "port": 80\n },\n "database": {\n "host": "localhost",\n "username": "myUsername",\n "password": "abcdefgh"\n }\n}We will place this file somewhere in the machine that hosts the application. We have to parse this file inside the application to get those configuration values.
\nTo parse it, we first have to create a type struct in our program :
\n// configuration/file/main.go \ntype Configuration struct {\n Server Server `json:"server"`\n Database DB `json:"database"`\n}\ntype Server struct {\n Host string `json:"host"`\n Port int `json:"port"`\n}\ntype DB struct {\n Host string `json:"host"`\n Username string `json:"username"`\n Password string `json:"password"`\n}We create three types struct :
\nConfiguration that will group the two other types :
Server that holds the properties of the JSON object server
DB that store the properties of the JSON object database
Those type structs will allow you to unmarshal the JSON configuration file. The next step is to open the file and read its content :
\n// configuration/file/main.go \n\nconfFile, err := os.Open("myConf.json")\nif err != nil {\n panic(err)\n}\ndefer confFile.Close()\nconf, err := ioutil.ReadAll(confFile)\nif err != nil {\n panic(err)\n}Here we are calling os.Open to open the file \"myConf.json\" located for the example in the same directory as the executable. In real life, you should upload this file to an appropriate location on the machine.
We get the whole content of the file by using ioutil.ReadAll. This function returns a slice of bytes. The next step is to use this slice of raw bytes with json.Unmarshal :
// configuration/file/main.go \n\nmyConf := Configuration{}\nerr = json.Unmarshal(conf, &myConf)\nif err != nil {\n panic(err)\n}\nfmt.Printf("%+v",myConf)We create a variable myConf that is initialized with an empty struct Configuration.Then we pass to json.Unmarshal the variable conf that contains the slice of bytes extracted from the file, and (as the second argument) a pointer to myConf.
In the end, our program outputs the following :
\n{Server:{Host:localhost Port:80} Database:{Host:localhost Username:myUsername Password:abcdefgh}}You can then share this variable with the whole application.
\n\nThe module github.com/spf13/viper is at the time of writing very popular among the Go community. It has more than 14.000 stars and more than 1.3k forks2.
This package allows you to simply define a configuration for your application from files, environment variables, command line, buffers... It also supports an interesting feature: if your configuration changes during your application’s lifetime, it will be reloaded.
\n\nLet’s give it a try :
\n// configuration/viper/main.go \n//...\n// import "github.com/spf13/viper"\nviper.SetConfigName("myConf")\nviper.(".")\nerr := viper.ReadInConfig()\nif err != nil {\n log.Panicf("Fatal error config file: %s\\n", err)\n}First, you have to define the name of the config file you want to load :viper.SetConfigName(\"myConf\"). Here we are loading the file named myConf. Note that we do not give viper the extension name of the file. It will retrieve the correct encoding and parse the file.
In the next line, we have to tell viper were to search with the functionAddConfigPath. The point means “look in the current directory” for a file named myConf. The supported extension at the time of writing are :
json,
toml,
yaml,
yml,
properties,
props,
prop,
hcl
Note that we could have added another path to search for config. The AddConfigPath function will append the path to a slice.
When we call ReadInConfig, the package will look for the file specified.
You can then use the configuration loaded :
\nfmt.Println(viper.AllKeys())\n//[database.username database.password server.host server.port database.host]\nfmt.Println(viper.GetString("database.username"))\n// myUsernameWith the AllKeys function, you can retrieve all the existing configuration keys of your application. To read a specific configuration variable, you can use the function GetString.
The GetString function takes as parameter the key of the configuration option. The key is a reflection of the configuration file hierarchy. Here \"database.username\" means that we access the value of the property username from the object database.
The database object also has the property host. To access it, the key is simply \"database.host\".
Viper exposes GetXXX functions for the types bool, float64, int , string , map[string]interface{}, map[string]string, slice of strings, time, and duration. It’s also possible to simply use Get to retrieve a configuration value. The returning type of this last function is the empty interface. I do not recommend it because it can lead to type errors. In our case, we are expecting a string, but if a user puts an integer into the config file, it will return an int. Your program will not work as expected and might panic.
Viper can parse environment variables automatically with a prefix :
\nviper.SetEnvPrefix("myapp")\nviper.AutomaticEnv()With those two lines, each time, you will call a getter method (GetString, GetBool, ...) viper will look for environment variables with the prefix MYAPP_ :
fmt.Println(viper.GetInt("timeout"))will look for the value of an environment variable named MYAPP_TIMEOUT.
Many projects are suffering from a lack of documentation concerning their configuration options.
\nBy documenting every single option available in a separate file, you are lowering your software’s adoption barrier.
\nInside a company, you might not be the person who will handle your application’s deployment. Hence you have to include in your estimates the documentation time. Better documentation means a reduced time to market for your solution.
\nThe interesting study of Ariel Rabkin and Randy Katz from the Berkeley University
There are mentions of configuration options that do not exist in the application’s source code. They noted that sometimes the options are simply commented into the application source code.
Some options exist in the application source code that are not even documented.
The solution is easy: create precise documentation of your configuration options and keep it up to date when the project evolves.
\n\nDuring a study
This figure is impressive and also very interesting. Systems can fail because of configuration mistakes. You cannot protect yourself against users that put the wrong port number in the configuration. But your application can protect itself from wrong configuration.
\nBy detecting it and alerting the user.
By not replacing a fraudulent value with the default one.
// configuration/error-detection/main.go \n//...\ndbPortRaw := os.Getenv("DATABASE_PORT")\ndbPort, err := strconv.ParseUint(port, 10, 16)\nif err != nil {\n log.Panicf("Impossible to parse database port number '%s'. Please double check the env variable DATABASE_PORT",dbPortRaw)\n}In the previous code listing, we are retrieving the value of the environment variable DATABASE_PORT. We then attempt to convert it to an uint16. If an error occurs, we refuse to start the application and inform the user about what’s wrong.
Note here that we tell the user what to do to fix this error. This is a good practice, it only takes you 10 seconds to write, but it might save 1 hour of research to your user.
\nDouble-check also that the program effectively uses all your configuration variables.
\n\nConfiguration variables are often composed of secrets: passwords, access tokens, encryption keys... A potential leak of those secrets may cause harm. None of the solutions above are perfect to store and secure production secrets. Rotation of those secrets is also not easy with the solution evoked before.
\nSome open-source solutions exist to handle this very specific issue. They offer a large set of features to protect, monitor, and audit the usage of your secrets. At the time of writing, it seems that Vault, edited by Hashicorp, is the most popular.3 And it’s written in Go!
\n\nHow to add a command-line option of type string to a program?
How to check if an environment variable exists & get its value in the same call?
How to set an environment variable?
How to configure your application with a file?
How to add a command-line option of type string to a program?
\nvar password stringflag.StringVar(&password,\"password\", \"default\",\"the db password\")// define other flags// parse input flagsflag.Parse()
password2 := flag.String(“password2”,“default”, “the db password”)
How to check if an environment variable exists & get its value in the same call?
\nWith the function LookupEnv from the package os
port, ok := os.LookupEnv(“DB_PORT”) if !ok { log.Fatal(“impossible to start up, DB_PORT env var is mandatory”) }
How to set an environment variable?
\nWith the function SetEnv from the package os
err := os.Setenv(“MYVAR3”, “test3”) if err != nil { panic(err) }
How to configure your application with a JSON file?
\nCreate a specific type with all your config variables.
At bootstrap, open the JSON file
Unmarshal the file.
Application configuration can be done via command-line options.
The standard package flag provide methods to add such options.
var password stringflag.StringVar(&password,\"password\", \"default\",\"the db password\")// define other flags// parse input flagsflag.Parse()
The configuration options are given at program startup via the command line : $ myCompiledGoProgram -password insecuredPass
The configuration is generally loaded at application startup (in the main)
Configuration of an application can also be done via environment variables
To retrieve an environment variable, you can use the os package
dbHost := os.GetEnv(“DB_HOST”)
dbHost, found := os.LookupEnv(“DB_HOST”)
You can also handle configuration via a file (JSON, YAML,.…). The idea is to create a type struct & unmarshal the file contents.
The module github.com/spf13/viper is a popular reference to handle application configuration.
Configuration options should be documented carefully by developers. An up to date documentation is a competitive advantage.
A dedicated secret management solution should handle application secrets.
Previous
\n\t\t\t\t\t\t\t\t\tTemplates
\n\t\t\t\t\t\t\t\tNext
\n\t\t\t\t\t\t\t\t\tBenchmarks
\n\t\t\t\t\t\t\t\t