Generating test data in JSON, CSV and SQL formats using Go and Faker

Sometimes, you may need to generate fake data for use for testing behavior of your systems or for load or volume testing, for example to test that pagination behavior works well with hundreds or hundreds of thousands of records .

In this tutorial, you will learn how to accomplish that using Go using the Faker package to generate data and then see how to export that data as JSON, CSV and as SQL insert statements, ready for inserting into your RDBMS.

Let's get started by creating a new project and initializing a Go module.

$ mkdir datagen

$ cd datagen

$ go mod init datagen

Next, use go get to add the faker package

$ go get -u github.com/go-faker/faker/v4

Now create a new file named main.go which will house the code.

Generating random people

This next part is where it gets interesting and the real action begins. But, first and foremost you must know your domain model or the shape of the data you want to work with.

You must know your domain model or the shape of the data you want to work with in order to generate good test data.

In the examples below, we will pretend we need to create data for a CRM-like product, and for ease of showing the concepts we will only deal with a Person entity, which we will also keep lean and add only a few fields. However, know that the techniques shown here work for types or entities with lots of fields

package main

import (
	"encoding/json"
	"fmt"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

func main() {
	dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
	if err != nil {
		panic(err)
	}
	person := Person{
		FirstName:   faker.FirstName(),
		LastName:    faker.LastName(),
		DateOfBirth: &dateOfBirth,
		Email:       faker.Email(),
		Phone:       faker.Phonenumber(),
		OtherNumbers: []string{
			faker.Phonenumber(),
		},
		Address: faker.GetRealAddress().Address,
	}

	fmt.Printf("%v", person)
	// okay that doesn't look great

	// let's use json
	data, err := json.Marshal(person)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(data))
}

We can run this with the following

$ go run main.go

Generating 1,000 random people

Now, usually you will want to generate a lot of data for stress testing, or testing out different use cases. So let's generate a lot more data - it’s as simple as just adding in a loop.

You can replace the contents of main with the following which generates 1,000 individuals, the limit can be tuned by changing the value of the constant N_PEOPLE

package main

import (
	"encoding/json"
	"fmt"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func main() {
	people := make([]Person, 0)
	for range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	// let's use json
	data, err := json.Marshal(people)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(data))
}

Again, you can run this with go run main.go

What you may have noted is that this writes the result to the Terminal (or rather to stdout). In order to write to a file, we can pipe the output to a JSON file. Running the following will give you a people.json file that contains people records.

$ go run main.go > people.json

Exporting generated data to CSV

Sometimes you may need data for testing import functionality that takes in a CSV file, lets see how to generate CSV file. Replace main with the following:

package main

import (
	"encoding/csv"
	"os"
	"strings"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func (p Person) toRecord() []string {
	return []string{
		p.FirstName,
		p.LastName,
		p.DateOfBirth.Format(time.DateOnly),
		p.Email,
		p.Phone,
		strings.Join(p.OtherNumbers, ";"),
		p.Address,
	}
}

func toRecords(people []Person) [][]string {
	records := make([][]string, 0)
	for _, p := range people {
		records = append(records, p.toRecord())
	}
	return records
}

func main() {
	people := make([]Person, 0)
	for range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	writer := csv.NewWriter(os.Stdout)
	writer.WriteAll(toRecords(people))
}

We can run this and write to CSV file with the following command

$ go run main.go > people.csv

Generating data as SQL insert statements

For most systems, you may be using a relational database of some kind. Here is the same code adapted to generate SQL insert statements that can be used to import data into a people table. Again, this is an example and your table names, columns, etc.. depend on the structure of your data.

package main

import (
	"fmt"
	"os"
	"strings"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	ID           int64
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func quote(unquoted string) string {
	return fmt.Sprintf(`'%s'`, strings.ReplaceAll(unquoted, "'", `''`))
}

func (p Person) toValuesRow() string {
	values := strings.Join([]string{
		quote(p.FirstName),
		quote(p.LastName),
		quote(p.DateOfBirth.Format(time.DateOnly)),
		quote(p.Email),
		quote(p.Phone),
		quote(strings.Join(p.OtherNumbers, ";")),
		quote(p.Address),
	}, ",")

	return fmt.Sprintf("(%s)", values)
}

func main() {
	people := make([]Person, 0)
	for i := range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			ID:          int64(i + 1),
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	sb := strings.Builder{}
	sb.WriteString("INSERT INTO people(first_name, last_name, date_of_birth, email, phone, other_numbers, address) VALUES \n")

	for i, p := range people {
		sb.WriteString(p.toValuesRow())
		if i == len(people)-1 {
			sb.WriteString(";\n")
		} else {
			sb.WriteString(",\n")
		}
	}

	fmt.Fprint(os.Stdout, sb.String())
}

Here is how to run the code to generate the SQL script, using the shell pipe operator again.

$ go run main.go > people.sql

Conclusion

Generating test data is important and typical for testing systems to verify behavior and for load/volume testing. We have seen how to use Go and the Faker package to generate such data and write to JSON, CSV and SQL files. We used a simple example, but the techniques here can be adapted for generating data for different domains and scenarios. The limit is your imagination. You can even adapt and evolve the ideas into CLI tool for your specific projects or teams.

I hope you found this useful.