In this tutorial, you will learn how to accomplish that using Go using the Faker package to generate data and then see how to export that data as JSON, CSV and as SQL insert statements, ready for inserting into your RDBMS.

Let's get started by creating a new project and initializing a Go module.

$ mkdir datagen

$ cd datagen

$ go mod init datagen

Next, use go get to add the faker package

$ go get -u github.com/go-faker/faker/v4

Now create a new file named main.go which will house the code.

Generating random people

This next part is where it gets interesting and the real action begins. But, first and foremost you must know your domain model or the shape of the data you want to work with.

You must know your domain model or the shape of the data you want to work with in order to generate good test data.

In the examples below, we will pretend we need to create data for a CRM-like product, and for ease of showing the concepts we will only deal with a Person entity, which we will also keep lean and add only a few fields. However, know that the techniques shown here work for types or entities with lots of fields

package main

import (
	"encoding/json"
	"fmt"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

func main() {
	dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
	if err != nil {
		panic(err)
	}
	person := Person{
		FirstName:   faker.FirstName(),
		LastName:    faker.LastName(),
		DateOfBirth: &dateOfBirth,
		Email:       faker.Email(),
		Phone:       faker.Phonenumber(),
		OtherNumbers: []string{
			faker.Phonenumber(),
		},
		Address: faker.GetRealAddress().Address,
	}

	fmt.Printf("%v", person)
	// okay that doesn't look great

	// let's use json
	data, err := json.Marshal(person)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(data))
}

We can run this with the following

$ go run main.go

Generating 1,000 random people

Now, usually you will want to generate a lot of data for stress testing, or testing out different use cases. So let's generate a lot more data - it’s as simple as just adding in a loop.

You can replace the contents of main with the following which generates 1,000 individuals, the limit can be tuned by changing the value of the constant N_PEOPLE

package main

import (
	"encoding/json"
	"fmt"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func main() {
	people := make([]Person, 0)
	for range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	// let's use json
	data, err := json.Marshal(people)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(data))
}

Again, you can run this with go run main.go

What you may have noted is that this writes the result to the Terminal (or rather to stdout). In order to write to a file, we can pipe the output to a JSON file. Running the following will give you a people.json file that contains people records.

$ go run main.go > people.json

Exporting generated data to CSV

Sometimes you may need data for testing import functionality that takes in a CSV file, lets see how to generate CSV file. Replace main with the following:

package main

import (
	"encoding/csv"
	"os"
	"strings"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func (p Person) toRecord() []string {
	return []string{
		p.FirstName,
		p.LastName,
		p.DateOfBirth.Format(time.DateOnly),
		p.Email,
		p.Phone,
		strings.Join(p.OtherNumbers, ";"),
		p.Address,
	}
}

func toRecords(people []Person) [][]string {
	records := make([][]string, 0)
	for _, p := range people {
		records = append(records, p.toRecord())
	}
	return records
}

func main() {
	people := make([]Person, 0)
	for range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	writer := csv.NewWriter(os.Stdout)
	writer.WriteAll(toRecords(people))
}

We can run this and write to CSV file with the following command

$ go run main.go > people.csv

Generating data as SQL insert statements

For most systems, you may be using a relational database of some kind. Here is the same code adapted to generate SQL insert statements that can be used to import data into a people table. Again, this is an example and your table names, columns, etc.. depend on the structure of your data.

package main

import (
	"fmt"
	"os"
	"strings"
	"time"

	"github.com/go-faker/faker/v4"
)

type Person struct {
	ID           int64
	FirstName    string
	LastName     string
	DateOfBirth  *time.Time
	Email        string
	Phone        string
	OtherNumbers []string
	Address      string
}

const N_PEOPLE = 1000

func quote(unquoted string) string {
	return fmt.Sprintf(`'%s'`, strings.ReplaceAll(unquoted, "'", `''`))
}

func (p Person) toValuesRow() string {
	values := strings.Join([]string{
		quote(p.FirstName),
		quote(p.LastName),
		quote(p.DateOfBirth.Format(time.DateOnly)),
		quote(p.Email),
		quote(p.Phone),
		quote(strings.Join(p.OtherNumbers, ";")),
		quote(p.Address),
	}, ",")

	return fmt.Sprintf("(%s)", values)
}

func main() {
	people := make([]Person, 0)
	for i := range N_PEOPLE {
		dateOfBirth, err := time.Parse(time.DateOnly, faker.Date())
		if err != nil {
			panic(err)
		}
		person := Person{
			ID:          int64(i + 1),
			FirstName:   faker.FirstName(),
			LastName:    faker.LastName(),
			DateOfBirth: &dateOfBirth,
			Email:       faker.Email(),
			Phone:       faker.Phonenumber(),
			OtherNumbers: []string{
				faker.Phonenumber(),
			},
			Address: faker.GetRealAddress().Address,
		}
		people = append(people, person)
	}

	sb := strings.Builder{}
	sb.WriteString("INSERT INTO people(first_name, last_name, date_of_birth, email, phone, other_numbers, address) VALUES \n")

	for i, p := range people {
		sb.WriteString(p.toValuesRow())
		if i == len(people)-1 {
			sb.WriteString(";\n")
		} else {
			sb.WriteString(",\n")
		}
	}

	fmt.Fprint(os.Stdout, sb.String())
}

Here is how to run the code to generate the SQL script, using the shell pipe operator again.

$ go run main.go > people.sql

Conclusion

Generating test data is important and typical for testing systems to verify behavior and for load/volume testing. We have seen how to use Go and the Faker package to generate such data and write to JSON, CSV and SQL files. We used a simple example, but the techniques here can be adapted for generating data for different domains and scenarios. The limit is your imagination. You can even adapt and evolve the ideas into CLI tool for your specific projects or teams.

I hope you found this useful.

The code for this project is on a GitHub repo: https://github.com/zikani03/spam-detection-with-onnx

Which model to use?

SPAM detection is a very important part of modern digital communications especially if your running platforms that accept User Generated Content (UGC). Implementing SPAM detection is one of the classic machine learning problems, and there are many approaches to doing so.

Fortunately, it is possible to find an open SPAM detection model now on Hugging Face and use it without much ado, even for commercial use. As I was looking around on Huggingface I came across OTIS, from the description of the project it says

Otis is an advanced anti-spam artificial intelligence model designed to mitigate and combat the proliferation of unwanted and malicious content within digital communication channels.

Sounds interesting enough, so I looked to see if there was an ONNX version of this model and was glad to find that the onnx-community organization has exactly that, here.

So the next step was to download the model.onnx and tokenizer.json files and include them in the project. Otis is licensed under BSD 3-Clause license for the curious.

The Controller

The controller isn’t much but here it is for reference, as you can see we have defined our API endpoint at the path: /api/spam/check which is intended to be called via a POST request. We rely on Spring’s internal content negotiation for the request and responses meaning we can expect to be able to send and receive JSON.

@RequestMapping("/api/spam/check")
@RestController
public class SpamCheckerController {
    private final SpamDetectionService spamDetectionService;

    public SpamCheckerController(@Autowired  SpamDetectionService spamDetectionService) {
        this.spamDetectionService = spamDetectionService;
    }

    @PostMapping
    public ResponseEntity<SpamCheckResponse> checkSpam(@RequestBody SpamCheckRequest request) throws Exception {
        return ok(spamDetectionService.detectSpam(request));
    }
}

The Spam Detection Service

The end goal is to have an API that can be called from HTTP client. But In order to separate concerns, we place the inference code for the Spam detection in a class named SpamDetectionService with an appropriate @Service annotation.

Inside this class we leverage the ONNX runtime for Java, passing the paths to the model and tokenizer files to initiate a HuggingFaceTokenizer . Here is the full code of the service:

@Service
public class SpamDetectionService implements AutoCloseable {

    private final HuggingFaceTokenizer tokenizer;
    private final OrtEnvironment env;
    private final OrtSession session;

    public SpamDetectionService(
            @Value("${model.path:-/models/model.onnx}") String modelPath,
            @Value("${tokenizer.path:-/models/tokenizer.json}") String tokenizerPath) throws IOException, OrtException {

        this.env = OrtEnvironment.getEnvironment();
        // Load session options -- no particular settings for GPU or CUDA environments
        OrtSession.SessionOptions options = new OrtSession.SessionOptions();
        options.setInterOpNumThreads(2);

        this.session = env.createSession(modelPath, options);
        this.tokenizer = HuggingFaceTokenizer.builder()
                .optPadding(true) // Add 0s if text is too short
                .optTruncation(true) // Cut off if text is too long
                .optTokenizerPath(Paths.get(tokenizerPath))
                .build();
    }

    public SpamCheckResponse detectSpam(SpamCheckRequest request) throws OrtException {
        long startTime = System.currentTimeMillis();
        var response = this.detectSpam(request.content());
        long endTime = System.currentTimeMillis();
        return new SpamCheckResponse(
                response.label,
                response.confidence,
                request.requestId(),
                endTime - startTime
        );
    }

    private RawResult detectSpam(String text) throws OrtException {
        Encoding encoding = tokenizer.encode(text);
        long[] inputIds = encoding.getIds();
        long[] attentionMask = encoding.getAttentionMask();
        long[] shape = {1, inputIds.length};

        try (OnnxTensor inputTensor = OnnxTensor.createTensor(env, LongBuffer.wrap(inputIds), shape);
             OnnxTensor maskTensor = OnnxTensor.createTensor(env, LongBuffer.wrap(attentionMask), shape)) {

            Map<String, OnnxTensor> inputs = new HashMap<>();
            inputs.put("input_ids", inputTensor);
            inputs.put("attention_mask", maskTensor);
            String tokenTypeIdsName = "token_type_ids";
            String outputName = session.getOutputNames().iterator().next();

            if (session.getInputNames().contains(tokenTypeIdsName)) {
                long[] tokenTypeIds = new long[inputIds.length];
                inputs.put(tokenTypeIdsName, OnnxTensor.createTensor(env, LongBuffer.wrap(tokenTypeIds), shape));
            }

            try (OrtSession.Result results = session.run(inputs)) {
                return formatResults(results, outputName);
            } finally {
                inputs.values().forEach(OnnxTensor::close);
            }
        }
    }

    record RawResult(String label, float[] probs, float cleanProb, float scamProb, float confidence) {}

    private RawResult formatResults(OrtSession.Result results, String outputName) throws OrtException {
            float[][] logitsArray = (float[][]) results.get(outputName).get().getValue();
            float[] rawLogits = logitsArray[0];
            float[] probs = softmax(rawLogits);

            float cleanProb = probs[0] * 100;
            float scamProb = probs[1] * 100;

            int prediction = (probs[1] > probs[0]) ? 1 : 0;

            String label = (prediction == 1) ? "SCAM" : "CLEAN";

            float confidence = (prediction == 1) ? scamProb : cleanProb;

            //return ("Result: " + label + " (" + String.format("%.2f", confidence) + "% confidence)");
            return new RawResult(label, probs, cleanProb, scamProb, confidence);
    }


    public static float[] softmax(float[] logits) {
        float[] probabilities = new float[logits.length];
        float maxLogit = Float.NEGATIVE_INFINITY;
        for (float v : logits) {
            if (v > maxLogit) maxLogit = v;
        }
        float sum = 0.0f;
        for (int i = 0; i < logits.length; i++) {
            probabilities[i] = (float) Math.exp(logits[i] - maxLogit);
            sum += probabilities[i];
        }
        for (int i = 0; i < logits.length; i++) {
            probabilities[i] /= sum;
        }

        return probabilities;
    }

    @Override
    public void close() throws Exception {
        session.close();
        env.close();
        tokenizer.close();
    }
}

You may note that the paths have default values which point to a directory starting with /models that’s because we intend to run this by default from a Docker container.

However, you can customize the paths to these models using the following configuration in a Spring Boot configuration file, e.g. in application.yaml

# application.yaml
model:
  path: "/path/to/models/model.onnx"
tokenizer:
  path: "/path/to/models/tokenizer.json"

Running the service via Docker

The project in the repository uses Jib to build docker image from the Java source code. Run the following command to build the container, by default the created image will be named zikani03/spam-detection-with-onnx

$ ./mvnw clean jib:dockerBuild

Once the build completes successfully you can run a docker container using the following, binding on port 8080 which the API runs at inside the container.

$ docker run -p "8080:8080"  zikani03/spam-detection-with-onnx

Once that’s running, you can then test the SPAM Detection service using your favourite HTTP Client e.g. Postman, Insomnia or even just cURL

$ curl -X POST -H "Content-Type: application/json" -d '{"requestId":"test","content":"Cһeck out our amazinɡ bооѕting serviсe ѡhere you can get to Leveӏ 3 for 3 montһs for just 20 USD.","token":"abc"}' "http://localhost:8080/api/spam/check"

You should get a result similar to this :

{"result":"SCAM","confidence":99.99815368652344,"id":"test","checkDurationMillis":149}

I like to load test things with hey, not bad.

The performance is okay, considering this is all running on CPU and not GPU (which I’m sure you can use with the onnxruntime libraries).

Conclusion

I have been curious about performing Machine Learning with Java for a while and ran into ONNX as I was trying out some Python stuff and got curious if I could leverage ONNX models in Java, and ofcourse you can! Microsoft’s onnxruntime for Java is a great place to start.

Sure, there is a lot more to add to this project to make it a real production-grade service, but I hope I have illustrated how it is possible to do some inference with Java and ONNX models. There are many models out there which you can leverage for different use cases.

I hope you are as excited about doing ML in Java too.

Connecting the gopher to the panda

So in order to get this to work in basi, I first needed to see if we can swap out Chrome/Chromium for LightPanda using the Chrome Developer Protocol in playwright-go, the Go library we use in basi, to talk to browsers to tell them what to do. I saw a demo/examples the showing LightPanda use via Playwright NodeJS , so this should be possible, right?.

Download and run Light Panda

Download LightPanda Browser from GitHub releases and then run it with the CDP services enabled on a port of our choice.

NOTE: the name of the binary you run may be different, depending on your OS and which binary you downloaded. The example below shows for Linux on x86

$ lightpanda-x86_64-linux serve --port 9226 --log_level debug

Implementing the Go program to talk to LightPanda

Create a new Go module project

$ mkdir lightpanda-go-example

$ cd lightpanda-go-example

$ go mod init lightpandagoexample

Next, Paste the following code in a file named main.go

This program starts a connection to LightPanda via Playwright by connecting to the browser using Chrome Devtools Protocol. After it connects, we try to automate a simple flow of going to the Playwright website and checking that the Link to the Community page contains the right text.

package main

import (
    "fmt"
    "log"

    playwrightgo "github.com/playwright-community/playwright-go"
)

func main() {
    err := run()
    if err != nil {
        log.Fatal(err)
    }
}
func run() error {

    err := playwrightgo.Install(&playwrightgo.RunOptions{
        Browsers: []string{"chromium"},
        DryRun:   true,
    })
    if err != nil {
        return fmt.Errorf("could not launch playwright: %w", err)
    }

    pw, err := playwrightgo.Run()
    if err != nil {
        return fmt.Errorf("could not launch playwright: %w", err)
    }

    browser, err := pw.Chromium.ConnectOverCDP("ws://localhost:9226")
    if err != nil {
        return fmt.Errorf("could not launch LightPanda via CDP: %w", err)
    }
    context, err := browser.NewContext()
    if err != nil {
        return fmt.Errorf("could not create context: %w", err)
    }
    page, err := context.NewPage()
    if err != nil {
        return fmt.Errorf("could not create page: %w", err)
    }

    _, err = page.Goto("https://playwright.dev/")
    if err != nil {
        return fmt.Errorf("could not goto: %w", err)
    }

    loc := page.Locator(`a[href="/community/welcome"]`)
    assertions := playwrightgo.NewPlaywrightAssertions()
    assert := assertions.Locator(loc)

    err = assert.ToHaveText("Community")
    if err != nil {
        return fmt.Errorf("could assert community link: %w", err)
    }

    err = browser.Close()
    if err != nil {
        return fmt.Errorf("could not close browser: %w", err)
    }
    err = pw.Stop()
    if err != nil {
        return fmt.Errorf("could not stop Playwright: %w", err)
    }

    return nil

}

Side note: Let’s compare the same thing with playwright-go (directly) vs using basi

For comparison here is how the same test would be written in a file named example.basi and executed with $ basi run example.basi

Goto "https://playwright.dev/"
Find "a[href='/community/welcome']"
ExpectText "Community"

Mandatory for now: Patch playwright-go library

At the time of writing, using playwright-go with lightpanda will only work if you clone playwright-go and checkout to the changes in the pull request I opened which fix an issue with playwright-go’s handling of the response from LightPanda’s implementation of the CDP.

NOTE: this step is mandatory to make this example work until the PR is merged or similar changes applied. Since playwright-go is looking for new maintainers, I can’t say when that will be

Here is how to patch a local copy of playwright-go :-

$ git clone https://github.com/playwright-community/playwright-go

$ cd ./playwright-go

# Checkout to the PR I opened
$ git fetch origin pull/581/head:main-with-cdp-fix

$ git checkout main-with-cdp-fix

Edit your go.mod to add a replace directive and make sure to point the path to the repository we just cloned, above :

replace github.com/playwright-community/playwright-go => /path/to/cloned/playwright-go

Run and build the Go program

You can now run the Go program

$ go mod tidy

$ go run main.go

That should work if you followed the steps. Nice, we just used LightPanda browse for automated e2en browser tests!

The way forward

The Zig community is putting out great projects, from TigerBeetle, Ghostty, more. I was intrigued by LightPanda, a headless web browser and was curious if I could use it from a browser automation tool I an working on that’s intended to make authoring and running e2e browser tests more approachable. It’s definitely possible! Being able to connect to LightPanda from Go via playwright means there’s a path to add support for it to basi and hopefully that means making the testing more lightweight especially for CI.

Currently, LightPanda doesn’t support a few things including Form file uploads and Screenshots. So we cannot replace Chrome/Chromium yet but I’m sure LightPanda will grow into maturity as it gets more use and development. Will definitely be watching the space, maybe I may just learn enough Zig to help out too. Even if that doesn’t happen, it’s always fun to tinker with new technology and see the potential of how one can use it today or in the future.

Hope you found this interesting. Thanks.

This very moment is some weird present-version of the past.

So what if we changed our approach to work and life with the confidence that whatever we do right now may be used or needed in a future however near or far. The consequence of this style of working is that it forces you to become good at somewhat predicting the future. Although the long term future is generally unpredictable, for those of us that are not prophets, near term tasks and activities can be predicted with higher level of confidence

Let me use a couple of examples :-

• Assume in 20 minutes you need to go out of the house to a meeting.Working in this style requires you to set things up to make that upcoming journey easier. You could set your shoes at the door, iron or press your clothes and put them somewhere accessible or within an arms reach from where you will start off. Doing those activities ahead of time will yield value at the time you actually start working on that task. You won't have to think about where to place or find things.

• Another example of that is creating a very rough draft of a contract you need to send next week. The sooner you get to a draft, even one that just contains headers, the easier it will be to re-board (not onboard) that task later on - the friction will be less.

But what you are describing is just preparation, right?

Yes and no. It goes further than what I will call “basic” preparation. What I am proposing here can be called preempting, or an advanced or more rigorous approach to preparation.

Author’s Note: The definition of the word pre-empt and how it’s being used in this article may seem to be at odds, but let’s appropriate this word for our purposes. Bear with me.

Collins Dictionary say’s If you preempt an action, you prevent it from happening by doing something that makes it unnecessary or impossible

Pre-empting in this view, means setting things up so that your future self doesn't have to do extra work to get things done. This concept is powerful once you realize that there are a lot of things you can preempt, and that preempting doesn't always mean getting things into a final state. What I mean by that is that you can preemptively start on a task, an leave it in a state where it is started/running but not exactly complete.

Preempting is like getting a ticket to an event, preparing is what you do when you are about to go to the event.

Leveraging AI to preempt work at the computer

We are fortunate to live in an age where LLMs (Large Language Models) and Agentic AI solutions exist. Even though they may be in their early forms, these tools can help us pre-empt work, especially work that needs to be done at the computer. One of the best ways to use these tools, that I have found, is to use them to get over the initial hurdle of starting something. For example, you can

• Ask ChatGPT or Gemini to give you a draft of a contract or proposal or just the headings you need

• Ask Claude code to scaffold a codebase for you for an idea or to work on the initial implementation of a couple of features

• Ask an image generation model to generate variations on ideas for a Logo or brand.

… and many more

The possibilities, with AI in the mix are endless and the amount of productivity this can unlock is huge. In addition to productivity, I believe being able to preempt work can help reduce anxiety related to working on or completing a task.

Go forth and preempt the world.

Thanks for reading.

Taking advantage of the pipeline mode, a client will wait less for the server, since multiple queries/results can be sent/received in a single network transaction.

In this article we will see how to issue pipelined queries using pgx library in Go

You will need the following

• Postgresql version 14 and above

• Go 1.24+

Example: Refund each customer by 5% of their balance

In this article I present a self contained code example that will 1) created a new table, 2) populate it with customers that have random balances and then, 3) uses pipelined queries to process refund for each client.

Because the queries are pipelined, it means our refund queries don’t have to wait for one customers refund query to complete to send the next one.

Now let’s code! Create a new Go project and copy the following code:

$ mkdir pipelinedqueries

$ cd pipelinedqueries

$ go mod init pipelinedqueries

Copy the following into a new file named main.go

package main

import (
    "context"
    "fmt"
    "math"
    "math/rand"
    "os"
    "time"

    "github.com/go-faker/faker/v4"
    "github.com/jackc/pgx/v5"
    "github.com/jackc/pgx/v5/pgconn"
    "github.com/shopspring/decimal"
)

const (
    sqlCreateUsersTable = `
create table pipeline_customers ( 
    id serial not null primary key, 
    name text not null, 
    balance numeric(18,6) not null default 0, 
    last_active_at timestamptz
);
`
    sqlRefundEachCustomer = `UPDATE pipeline_customers SET balance = balance + $1 where id = $2`
)

type Customer struct {
    ID           int64           `db:"id"`
    Name         string          `db:"name"`
    Balance      decimal.Decimal `db:"balance"`
    LastActiveAt time.Time       `db:"last_active_at"`
}

func createCustomers(conn *pgx.Conn, N int) error {
    _, err := conn.Exec(context.Background(), `DROP TABLE IF EXISTS pipeline_customers;`)
    if err != nil {
        return err
    }
    _, err = conn.Exec(context.Background(), sqlCreateUsersTable)
    if err != nil {
        return err
    }

    for i := 0; i < N; i++ {
        _, err := conn.Exec(context.Background(), `INSERT INTO pipeline_customers(name, balance, last_active_at) VALUES ($1, $2, $3)`,
            faker.Name(),
            decimal.NewFromFloat32(1000.0*float32(rand.Intn(100))*1.31425326),
            time.Now().Add(time.Duration(rand.Intn(72)*-1)*time.Hour),
        )
        if err != nil {
            return err
        }

    }
    return nil
}

func fetchAllCustomers(conn *pgx.Conn) ([]Customer, error) {
    customers := make([]Customer, 0)
    rows, err := conn.Query(context.Background(), `SELECT id,name,balance,last_active_at FROM pipeline_customers order by id asc`)
    if err != nil {
        return nil, err
    }
    for rows.Next() {
        customer := Customer{}
        err := rows.Scan(&customer.ID, &customer.Name, &customer.Balance, &customer.LastActiveAt)
        if err != nil {
            return nil, err
        }
        customers = append(customers, customer)
    }
    return customers, nil
}

// refundAllCustomers uses pgx's support for pipelined queiries to update each customer balance in a pipeline
func refundAllCustomers(conn *pgx.Conn, customers []Customer, injectErrors bool) error {
    ctx := context.Background()
    pipeline := conn.PgConn().StartPipeline(ctx)

    eqb := pgx.ExtendedQueryBuilder{}

    // introduce error for 1% random customers
    nErrors := int(math.Round(float64(len(customers)) * 0.01))
    errorsInjected := 0

    for _, customer := range customers {
        // calculate 5% of the customers balance
        customerRefundAmount, _ := customer.Balance.Mul(decimal.NewFromFloat32(0.05)).Float64()
        fmt.Println("processing customer", customer.Name, customerRefundAmount)
        queryArgs := []any{
            customerRefundAmount,
            customer.ID,
        }
        // introduce error random customers
        if injectErrors && errorsInjected < nErrors {
            if rand.Intn(2) >= 1 {
                queryArgs = []any{
                    fmt.Sprintf("%f%s", customerRefundAmount, "RANDOM_STRING"),
                    customer.ID,
                }
                errorsInjected += 1
            }
        }

        err := eqb.Build(conn.TypeMap(), nil, queryArgs)
        if err != nil {
            return fmt.Errorf("failed to buld query: %v", err)
        }

        pipeline.SendQueryParams(sqlRefundEachCustomer, eqb.ParamValues, nil, eqb.ParamFormats, eqb.ResultFormats)

    }
    err := pipeline.Sync()
    if err != nil {
        return fmt.Errorf("failed to sync pipeline query: %v", err)
    }
    results, err := pipeline.GetResults()
    if err != nil {
        return fmt.Errorf("failed to get pipeline results: %v", err)
    }
    if results == nil && err == nil {
        // no results received from server
        return nil
    }
    // spew.Dump(results)
    _, ok := results.(*pgconn.ResultReader)
    if !ok {
        return fmt.Errorf("failed to read results from pipelined query: %v", err)
    }
    err = pipeline.Close()
    if err != nil {
        return fmt.Errorf("failed to close pipeline: %v", err)
    }
    return nil
}

func printCustomersTable(customers []Customer, n int) {
    if len(customers) < n {
        return
    }

    for i := 0; i < n; i++ {
        customer := customers[i]
        fmt.Printf("Customer: %s \t\t Last Active: %s\n", customer.Name, customer.LastActiveAt.Format(time.DateOnly))
        fmt.Printf("Balance: %s\n---\n", customer.Balance)
    }
}

func main() {
    dbURL := "postgres://user:password@localhost:5432/pipeline_tutorial"
    conn, err := pgx.Connect(context.Background(), dbURL)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
        os.Exit(1)
    }
    // comment out the next few lines to prevent creating new customers on each run
    err = createCustomers(conn, 100) // N = number of customers
    if err != nil {
        panic(fmt.Errorf("createCustomers: %v", err))
    }

    customers, err := fetchAllCustomers(conn)
    if err != nil {
        panic(fmt.Errorf("fetchAllCustomers: %v", err))
    }
    // printCustomersTable(customers, 10)
    err = refundAllCustomers(conn, customers, true)
    if err != nil {
        panic(fmt.Errorf("refundAllCustomers: %v", err))
    }

    fmt.Println("------- AFTER PIPELINE QUERY -------")
    otherCustomers, err := fetchAllCustomers(conn)
    if err != nil {
        panic(fmt.Errorf("fetchAllCustomers: %v", err))
    }
    printCustomersTable(otherCustomers, 10)
}

Side Note on SQL errors in pipeline queries: I have intentionally added error injection to this example code to show you how errors are treated when working with pipelined queries. We inject random error that results in INVALID SQL for 1% of our customers. What you will note when you run the code and the error conditions are satisfied is that the whole pipeline fails - this should give you confidence that SQL or database level errors won’t silently fail during processing pipeline. Logic errors are still something you have to make sure to handle.

The code with the following command:

go run main.go

Let’s take a closer look at the core parts of the function that handles the Postgresql pipeline queries, I’ve stripped out the error injection and annotated the relevant lines :

// refundAllCustomers uses pgx's support for pipelined queries to update each customer balance in a pipeline
func refundAllCustomers(conn *pgx.Conn, customers []Customer, injectErrors bool) error {
    ctx := context.Background()
    pipeline := conn.PgConn().StartPipeline(ctx) // 1 

    eqb := pgx.ExtendedQueryBuilder{} // 2

    for _, customer := range customers {
        // calculate 5% of the customers balance
        customerRefundAmount, _ := customer.Balance.Mul(decimal.NewFromFloat32(0.05)).Float64()
        fmt.Println("processing customer", customer.Name, customerRefundAmount)
        queryArgs := []any{
            customerRefundAmount,
            customer.ID,
        }

        err := eqb.Build(conn.TypeMap(), nil, queryArgs) // 3
        if err != nil {
            return fmt.Errorf("failed to buld query: %v", err)
        }

        // 4 below
        pipeline.SendQueryParams(sqlRefundEachCustomer, eqb.ParamValues, nil, eqb.ParamFormats, eqb.ResultFormats)

    }
    err := pipeline.Sync() // 5
    if err != nil {
        return fmt.Errorf("failed to sync pipeline query: %v", err)
    }
    results, err := pipeline.GetResults() // 6
    if err != nil {
        return fmt.Errorf("failed to get pipeline results: %v", err)
    }
    if results == nil && err == nil {
        // no results received from server
        return nil
    }
    // spew.Dump(results)
    _, ok := results.(*pgconn.ResultReader) // 7
    if !ok {
        return fmt.Errorf("failed to read results from pipelined query: %v", err)
    }
    err = pipeline.Close() // 8
    if err != nil {
        return fmt.Errorf("failed to close pipeline: %v", err)
    }
    return nil
}

Several things are happening in the code above, let’s focus on the interesting bits:

1. Obtain a Pipeline pipeline from the connection by starting the pipeline.

2. Create an extended query builder that’s used to pass the query parameters

3. Build the query with a type mapping and the arguments for our SQL query

4. We send the query into the pipeline using the parameters from the extended query builder from the previous step. Since this is running in pipeline, we can continue sending multiple queries to Postgres without for it to process it (note that we are doing this inside the loop)

5. We then call sync on the pipeline to flush/execute the submitted queries and make the results available

6. We call GetResults() to obtain the results of the pipeline query

7. We check if the pipeline gave us a result we can read from, in this example we ignore the result reader

8. We close the pipeline and release related resources

On performance and resource utilization

Pipeline mode generally consumes more memory on both the client and server as noted in the Postgres documentation - this makes intuitive sense since we are buffering the queries and their parameters as we run the pipeline, especially before flushing/syncing. It is recommended to do appropriate management of the send/receive queue and to, of course, monitor the behaviour of your systems to identify bottlenecks or spikes in resource utilization.

Conclusion

In this article I shared how to use pipelined queries in Postgres from a Go program. Pipelined queries can open up a lot of use cases where you need to do batch operations but the data is coming in stream or whatever other use cases you can think of. As a reminder, pipeline mode generally consumes more memory on both the client and server and it is recommended to do appropriate management of the send/receive queue and monitor the behaviour of your systems to avoid bottlenecks.

Hope you found this interesting.

What is a monorepo?

A monorepo is a workspace that contains multiple projects in one Git repository. According to Semaphore CI "A monorepo is a version-controlled code repository that holds many projects. While these projects may be related, they are often logically independent and run by different teams."

Why would I want to merge my repos?

Adopting monorepos can provide several advantages. Firstly, code sharing and reuse become effortless. Developers on your team can easily find and incorporate existing libraries or functionalities into new projects, reducing redundancy and development time.

Monorepos can also enhance collaboration by fostering a shared codebase as teams gain better visibility into interdependent projects built by other teams, enabling faster identification of potential issues and more streamlined enforcement of coding practices and standards across the entire codebase. This can result in better code quality, easier to apply organization wide standards and practices which may help reducing maintenance overhead in the long run.

Another use case is for performing backups of organization wide codebases. This may be ideal for situations where disaster recovery, compliance or regulatory requirement insist that you keep an offsite backup. Being able to merge the code periodically, and download or upload to some Object Storage is a good use case for merging multiple repositories into a monorepo.

Installing and using git-monorepo

Firstly, you will need to install git-monorepo. More documentation on the tool can be found here

Download the latest release from GitHub: https://github.com/zikani03/git-monorepo/releases

After installation, run the following example command to create a monorepo from two GitHub repositories. This may take a bit of time as the repositories have to be cloned from GitHub to your machine, and then the commit histories replayed one-by-one.

$ git-monorepo init \
    --sources gh:zikani03/git-monorepo,gh:zikani03/articulated \
    --target my-new-monorepo

After the command completes, we can inspect the new repository using Git as usual. You will see that the history is now intertwined as if the two repos had been one from the start!

$ cd my-new-monorepo
$ git log --oneline

In the example above you can see we use the gh shorthand for GitHub to indicate that the repositories we are using are going to be pulled from GitHub. You can also specify the full URL to the repository and the tool will continue to work :

$ git-monorepo init \
    --sources https://github.com/zikani03/git-monorepo,https://github.com/zikani03/articulated \
    --target my-new-monorepo

It also works with other Git hosting services

git-monorepo works well with GitHub but you can also use it clone repositories from any Git server. There are shorthands for GitHub (gh), BitBucket (bb) and GitLab (gl). Small caveat is that we have yet to test thoroughly with private repository servers.

Here is an example of cloning projects from GitLab and Codeberg hosting services:

$ git-monorepo init \
    --sources https://gitlab.com/writeonlyhugo/up-business-theme,https://codeberg.org/201984/dut \
    --target my-new-monorepo

Conclusion

In this article I shared a tool I built named git-monorepo which can be used to merge repositories into a monorepo. Comments and contributions are always welcome!

EmbeddingGemma sounds compelling for it’s relatively small size and potential for local on-device applications.

From their article :

EmbeddingGemma generates embeddings, which are numerical representations - in this case, of text (such as sentences and documents) - by transforming it into a vector of numbers to represent meaning in a high-dimensional space. […] EmbeddingGemma empowers developers to build on-device, flexible, and privacy-centric applications. It generates embeddings of documents directly on the device's hardware, helping ensure sensitive user data is secure.

In this article, we will do something a bit unconventional, we will use EmbeddingGemma to generate embeddings from a Go program. Typically, it is advisable to work with embeddings from language like Python which has better ecosystem and libraries for such things.

Prerequisites

• Go 1.24+

• Ollama v0.11.10+

We will use a similar example to the one in the HuggingFace blog post. In order to follow along with this tutorial you will need to have Ollama installed, you will need version atleast v0.11.10 and pull the embeddinggemma:300m model

$ ollama --version

$ ollama pull embeddinggemma:300m

Once this completes successfully, you will have the EmbeddingGemma model on your machine, ready to interact with via Ollama to generate embeddings.

Let’s get to coding, start by creating a new go project

mkdir embeddings-tutorial
cd embeddings-tutorial
go mod init embeddings-tutorial
touch main.go

Before we can look at the Go code, let’s discuss what it does.

The code below is a small program that interacts with Ollama via the LangChain library. In the program we have a user query or question and a list of facts (think of those as a database) that we can query against. Both the query and facts are hardcoded in the code as strings but they could come from anywhere, a text file, stdin etc.. Once we have the database and the user query we convert them to Vector Embeddings (remember, embeddings are a numeric representation) using Ollama via the llm.CreateEmbedding function.

Once the embeddings are created, we need a way to use them to perform a search which is basically an operation to compare the vector embedding of the query with the embeddings of the information in our “database” to see which fact/statement most closely matches the query. In order to do this, we use the HNSW data structure from the github.com/habedi/hann package. The package implements a few algorithms for working with high-dimensionality data, but we pick HNSW as it is one of the most commonly used approach for working with embeddings:

“HNSW - addresses the challenge of quickly finding items “close” to a query point in large datasets, which is computationally expensive with traditional methods like brute-force comparisons.” - from Milvus.io

Copy the following content into main.go

package main

import (
   "context"
   "fmt"
   "log"

   "github.com/habedi/hann/core"
   "github.com/habedi/hann/hnsw"
   "github.com/tmc/langchaingo/llms/ollama"
)

const modelID = "embeddinggemma:300m"

const query = "Which planet is known as the Red Planet?"

var documents = []string{
   "Venus is often called Earth's twin because of its similar size and proximity.",
   "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
   "Jupiter, the largest planet in our solar system, has a prominent red spot.",
   "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
}

func preprocess(content []string) []string {
   res := make([]string, len(content))
   for idx, item := range content {
       res[idx] = fmt.Sprintf("title: none | text: %s", item)
   }
   return res
}

func main() {
   ctx := context.Background()
   llm, err := ollama.New(
       ollama.WithModel(modelID),
   )

   if err != nil {
       log.Fatalf("failed to connect to ollama: %v", err)
   }
   // our embeddings are
   embeddings, err := llm.CreateEmbedding(ctx, preprocess(documents))
   if err != nil {
       log.Fatalf("failed to generate embedding from ollama: %v", err)
   }

   fmt.Println("generated embeddings successfully, now you gotta use 'em", "dimension", len(embeddings[0]))

   dimension := 768
   m := 4   // maximum number of neighbor connections per node
   ef := 16 // search breadth factor
   distanceFuncName := "cosine"

   index := hnsw.NewHNSW(dimension, m, ef, core.Distances[distanceFuncName], distanceFuncName)
   for idx, embedding := range embeddings {
       err = index.Add(idx, embedding)
       if err != nil {
           log.Fatalf("failed to index embedding in HNSW data structure: %v", err)
       }
   }

   // now that we have built an index, we can query it
   queryTask := fmt.Sprintf("task: search result | query: %s", query)
   queryEmbedding, err := llm.CreateEmbedding(ctx, []string{queryTask})
   if err != nil {
       log.Fatalf("failed to generate query embedding from ollama: %v", err)
   }

   numNeighbors := 3
   neighbors, err := index.Search(queryEmbedding[0], numNeighbors)
   if err != nil {
       log.Fatalf("failed to search for query in index: %v", err)
   }

   fmt.Println("Results: ")
   for _, n := range neighbors {
       fmt.Printf("Relevance: %f \t Content: %s\n", 1-n.Distance, documents[n.ID])
   }
}

Now that we have looked at and understand the code, you can then run the following commands:

$ go mod tidy

$ go run main.go

You will get output that looks something like this

As you can see, the answer we want about Mars is ranked the highest! (has the highest relevance score)

This result is good as it gives us confidence that the query we searched for matched the most relevant items in our “database” - this technique can be used to implement semantic search in applications as well as recommendations based in unstructured input/text.

Warning and/or Disclaimer: Now I should mention that while this works, I wouldn’t advise using this approach for production. You are better off using solutions in the Python ecosystem as they have more robust libraries for this sort of thing.

I hope you found this somewhat interesting.

Trying to understand why something isn’t working can often feel daunting or stressful.

Code is meant for humans.

Author’s note: Drafted this article before AI agents were a thing. I’m confident that code is still meant for humans but I cannot say to what degree.

When people make libraries or build tools, they generally build for other humans to consume (that is, read or maintain) and for machines to execute. Understanding this, we can be sure of a few things

1. People are going to try to use familiar conventions, idioms, words or domain language

2. People are going to try to make things easier to use for either themselves or others

What this leads to is that, for a lot of things, if you can make reasonable guesses about what’s expected to happen you can be right or close to it about 50% of the time (I don’t have hard numbers nor have i done a study on this, just anecdata). And the other time you aren’t on point - you can refer to docs to help align how you understand the library, tool or the world. You will have to correct your assumptions or in another case it could lead to you contributing to a feature to the tool or project (I am thinking particularly of Open Source here) in other cases it could lead to you creating your own spin on the problem and innovating a new approach.

You cannot live on assumptions alone

A most important point to make and emphasize here is that you have to verify your assumptions, usually this will come from using the tool or library and observing it’s output, writing tests, reading the documentation or diving into the source code, if that is available. Assumptions are a first step to help you try to meld your mental model of the thing and how you expect it to work for you.

What happens when AI is writing the code? Should I still make assumptions? Well, kinda.

I’d be remiss if I didn’t talk about how AI fits in or affects this line of thinking. If most of the code you are consuming was generated by AI, it is possible that it may be a bit more difficult to make assumptions since LLMs are …. LLMs. It means you may need to not only make assumptions but also verify more thoroughly as sometimes AI can make mistakes, as we’ve been told countless times by LLM service providers.

I needed a way to hide or mask the names of people, companies and places in the notes to safely show what I was working on. Since we are dealing with a privacy issue I didn’t want to use a remote AI service provider API and I was a bit too lazy to setup and fiddle with a Local LLM for the task (I have comments about working with Local LLMs as I’ve been experimenting with them for a bit at this point, but that’s for another day).

Here is the quick and dirty solution I cobbled up using Go and the prose library which I have used before for Named Entity Recognition. In the example below we will use an excerpt from a BBC News article which contains several named entities.

Follow these steps to get it running in your environment.

NB: This is a Go program and needs Go to be installed

1. Create a new directory and initiate a Go module

mkdir anonymize 
cd anonymize
go mod init anonymize

2. Create a file named main.go and place the following contents

package main

import (
    "flag"
    "fmt"
    "strings"

    "github.com/go-faker/faker/v4"
    "github.com/jdkato/prose/v2"
    "github.com/sergi/go-diff/diffmatchpatch"
)

// Source: https://www.bbc.com/news/articles/crr24eqnnq9o
const rawText = `
Using AI to help write code has increased in popularity as the tech becomes more capable and accessible.

Anthropic says it detected a case of so-called "vibe hacking", where its AI was used to write code which could hack into at least 17 different organisations, including government bodies.

It said the hackers "used AI to what we believe is an unprecedented degree".

They used Claude to "make both tactical and strategic decisions, such as deciding which data to exfiltrate, and how to craft psychologically targeted extortion demands".

It even suggested ransom amounts for the victims. Agentic AI - where the tech operates autonomously - has been touted as the next big step in the space. But these examples show some of the risks powerful tools pose to potential victims of cyber-crime. The use of AI means "the time required to exploit cybersecurity vulnerabilities is shrinking rapidly", said Alina Timofeeva, an adviser on cyber-crime and AI. 
`

var showDiff = false

func init() {
    flag.BoolVar(&showDiff, "diff", false, "Show Diff of the anonymized text")
}

func main() {
    flag.Parse()

    if showDiff {
        dmp := diffmatchpatch.New()
        diffs := dmp.DiffMain(rawText, anonymize(rawText), true)
        fmt.Println(dmp.DiffPrettyText(diffs))
    } else {
        fmt.Println(anonymize(rawText))
    }
}

func anonymize(value string) string {
    if value == "" {
        return value
    }
    doc, err := prose.NewDocument(value)
    if err != nil {
        return ""
    }

    entitiesAsPatterns := make([]string, 0)
    for _, ent := range doc.Entities() {
        entitiesAsPatterns = append(entitiesAsPatterns, ent.Text)
        switch ent.Label {
        case "PERSON":
            entitiesAsPatterns = append(entitiesAsPatterns, faker.Name())
            break
        case "LOCATION":
            entitiesAsPatterns = append(entitiesAsPatterns, faker.MacAddress())
            break
        case "FACILITY":
            entitiesAsPatterns = append(entitiesAsPatterns, strings.ToUpper(faker.Name()))
            break
        case "ORGANIZATION":
            entitiesAsPatterns = append(entitiesAsPatterns, faker.DomainName())
            break
        default:
            entitiesAsPatterns = append(entitiesAsPatterns, "[MASKED]")
        }
    }

    r := strings.NewReplacer(entitiesAsPatterns...)
    return r.Replace(value)
}

3. Run the program, it has two output modes - by default it will just print the text with the names the Named Entity Recognition algorithm finds replaced with fake names, and in the second mode it prints a diff which shows which text was changed.

   Run without diff:

    go run main.go
   

   Run with DIFF output

    go run main.go -diff
   

The output comes out differently each time so you will most likely get a different result. Here is an example of output from the program.

Here is how it looks with DIFF mode:

Feel free to try with your own text and modify the program to maybe anonymize text from a document or a database. Go crazy!

Caveat The prose library works very well with english but may sometimes miss some entities especially titles or positions, and obviously since the rest of the text is maintained the context of your note could give away information to someone who has more information or context, it’s not a perfect solution!

Conclusion

In this short article we have seen how we can write a simple Go program to help maintain privacy in our notes by masking or anonymizing the original names of people or places in unstructured text. We leverage the prose library to detect entities, the faker library to generate fake data and Go’s standard library strings.Replacer for the heavy lifting and add some diffing to ensure that we can tell what was actually replaced.

NB: I never actually got around to showing my friends the app I am working on 😅 I guess I just needed motivation to write and publish this article.

I hope you found this interesting. Thanks for reading.

If you want to go straight to the code, you can find it in this repository: https://github.com/zikani03/store-encrypted-json-go-struct-tutorial

Why and when to encrypt data stored in the database

Firstly, why would we want to store data encrypted in a database?

With the increase of cyber threats and threat actors on the internet, any application you develop must have Security as a top priority. Encryption provides a good level of security for sensitive data such as Personally Identifying Information (PII), Finance information (like Credit Card information) and other sensitive pieces of data. If a threat actor gains access to the encrypted data, you want to make it harder for them to decrypt and get the raw data.

So, when should we encrypt data in our applications? Ideally, every time. Practically, no one encrypts everything in the database at the application level. You can use disk level encryption to secure your whole storage layer but that is out of scope of this article.

What will we be encrypting?

In this article, we will demonstrate the technique using a fictional service that needs to encrypt Customer settings which may contain API Keys to third-party services, something that is common nowadays with SaaS platforms and AI service wrappers which ask you keys for APIs like OpenAI’s or another platform.

We want to keep data securely, especially in a multi-tenant system where each Customer has their own settings, and we want to ensure that data is encrypted so that even a database administrator or anyone with access to the database cannot read that information without a decryption key.

The key structs we will work with are shown below:

type Setting struct {
    CustomerID   int64        `json:"customer_id" gorm:"primaryKey"`
    Key          string       `json:"key" gorm:"key"`
    Data         SettingsData `json:"data" gorm:"column:data_hash;type:text"`
}

type SettingsData struct {
    Version         string `json:"version"`
    AllowNewMembers bool   `json:"allowNewMembers"`
    OpenAIAPIKey    string `json:"apiKey"`
}

Our database table for this would look like this. Note that the data is stored as a text column

CREATE TABLE tenant_settings (
  customer_id integer primary key not null,
  key text not null,
  data_hash text not null
);

Encryption using AES-GCM using symmecrypt

Let’s see how to perform basic encryption of data. We will use ovh/symmecrypt to enable us to encrypt data. Firstly, we need to generate an encryption key, which we have to keep somewhere safe as it will be used for both encryption and decryption.

func generateSecureKey() string {
    b := make([]byte, 32)
    // Read 32 random bytes from the cryptographically secure random number generator
    _, err := rand.Read(b)
    if err != nil {
        fmt.Println("Error generating random bytes:", err)
        return ""
    }
    // Print the random bytes in hexadecimal format
    return fmt.Sprintf("%x", b)
}

After generating and storing our encryption key, we need to pick an algorithm to use for encryption/decryption. For the purposes of this article we chose AES-GCM, although the symmecrypt supports other algorithms as well.

We have to store the key in a keyloader, as shown below:

    secretKey := generateSecureKey()
    kk := keyloader.KeyConfig{
        Identifier: EncryptionKeyCtx,
        Cipher:     "aes-gcm",
        Timestamp:  time.Now().UnixMilli(),
        Sealed:     false,
        Key:        string(secretKey),
    }

    keyList := []configstore.Item{}
    keyList = append(keyList, configstore.NewItem(EncryptionKeyCtx, kk.String(), 1))
    configstore.RegisterProvider("env", func() (configstore.ItemList, error) {
        return configstore.ItemList{
            Items: keyList,
        }, nil
    })

The following snippet shows how to perform the encryption once the key is loaded into the configuration store. As you can see, the Encrypt function expects a slice of bytes and also returns a slice of encrypted bytes. As a result we have to decide on how to encode those bytes in order to print them out to the terminal, in this article we are encoding to Base64.

func HowToEncrypt() {
    k, err := keyloader.LoadSingleKey()
    if err != nil {
        log.Fatalf("failed to load encryption key")
        return
    }

    encrypted, err := k.Encrypt([]byte("Hello World, Please encrypt me!"))
    if err != nil {
        log.Fatalf("failed to encrypt data %v", err)
        return
    }

    fmt.Println("Encrypted, encoded as Base64", base64.StdEncoding.EncodeToString(encrypted))
}

Storing and reading the encrypted data via pgx or database/sql

In order to store our encrypted data to the database via pgx we need to implement the Valuer interface. This interface must be implemented for the structs that we wish to store in encrypted manner. The extra benefit is that this enables this technique to work with other databases using the standard database/sql module.

Implementing the Value() function

This implementation of the Value() function does several key things to make things work:

1. Marshals the value of the struct itself to JSON data

2. Loads the encryption key from the key loader

3. Encrypts the JSON data using the loaded key

4. Encodes and returns the encrypted data as a Base64 string

func (settings *SettingsData) Value() (driver.Value, error) {
    data, err := json.Marshal(settings)
    if err != nil {
        return nil, fmt.Errorf("failed to marshal data %v", err)
    }

    k, err := keyloader.LoadSingleKey()
    if err != nil {
        return nil, fmt.Errorf("failed to marshal data %v", err)
    }

    encrypted, err := k.Encrypt(data)
    if err != nil {
        return nil, fmt.Errorf("failed to marshal data %v", err)
    }

    return base64.StdEncoding.EncodeToString(encrypted), nil
}

As a result of these steps, the database driver will store the value as a Base64 string in the database. That string may be decoded from base64 but because it is encrypted the output will look like garbage. In order to decrypt the data, one must use the same key and algorithm that was used to encrypt it.

Implementing the Scan() function

In order to read the data back and decrypt it we simply have to implement the database Scanner interface. This function does a few things:

1. It tries to read/cast the value to a string

2. It tries to decode the value from Base64 into a slice of bytes

3. Loads a key from the key loader

4. Uses the key to decrypt the data, which is expected to be stored as JSON

5. Unmarshals the JSON into the struct

func (settings *SettingsData) Scan(value interface{}) error {
    stored, ok := value.(string)
    if !ok {
        return errors.New(fmt.Sprint("Failed to unmarshal JSONB value:", value))
    }

    data, err := base64.StdEncoding.DecodeString(stored)
    if err != nil {
        return fmt.Errorf("failed to decode base64 %v", err)
    }

    k, err := keyloader.LoadSingleKey()
    if err != nil {
        return fmt.Errorf("failed to load key %v", err)
    }
    decryptedData, err := k.Decrypt(data)
    if err != nil {
        return fmt.Errorf("failed to decrypt %v", err)
    }
    result := SettingsData{}
    err = json.Unmarshal(decryptedData, &result)
    *settings = result
    return err
}

Storing the encrypted data via Gorm

Similar to how we store the data when using pgx or regular database/sql - using Gorm also requires implementation of a specific interface, GormValuer, that tells the Gorm library how to handle this data when storing in the database.

Implementing the GormValue() function

Below is an implementation of the GormValue function for our SettingsData type:

func (s SettingsData) GormValue(ctx context.Context, db *gorm.DB) (expr clause.Expr) {
    data, err := json.Marshal(s)
    if err != nil {
        db.AddError(fmt.Errorf("failed to marshal data %v", err))
    }

    k, err := keyloader.LoadSingleKey()
    if err != nil {
        db.AddError(err)
        return
    }

    encrypted, err := k.Encrypt(data)
    if err != nil {
        db.AddError(fmt.Errorf("failed to encrypt data %v", err))
        return
    }

    return clause.Expr{SQL: "?", Vars: []interface{}{base64.StdEncoding.EncodeToString(encrypted)}}
}

Where to get (and store) the Encryption Key

This part is left as an exercise for the reader. It’s up to you to decide

• Whether you will store and retrieve the encryption key from Configuration file, environment variables or from a third-party Key Management service like HashiCorp Vault, AWS KMS or Keycloak

Conclusion

In this article, we explored how to securely store Go structs as encrypted JSON in a PostgreSQL database using the AES-GCM algorithm via the symmecrypt library. We demonstrated how to implement encryption and decryption logic compatible with both pgx and the Gorm ORM by leveraging Go's Valuer and Scanner interfaces. This approach ensures sensitive fields like API keys remain protected—even from database administrators—by encrypting data at the application level.

By integrating encryption into your data models, you add a critical layer of security that complements other infrastructure-level protections. While this method requires some setup, it offers strong guarantees for data confidentiality—especially in multi-tenant systems or applications handling sensitive customer information.

For a complete implementation, check out the sample code in the linked repository. And remember: secure key storage is just as important as encryption itself—be deliberate about how and where your keys are managed.

Pre-requisites

You will need to have the following installed to follow along:

• Go 1.23+

• Hurl

• Venom

• Text Editor or IDE (recommend either VSCode or GoLand)

What are cash out / admin fees?

Imagine you are offering a service online and want to get commission off of every transaction, such commissions are usually pushed to customers as administrative fees. Typically these fees are calculated relative to the amount the customer/user pays.

To make this more concrete we will use the concept of mobile money cash-out fee wherein a Customer desires to withdraw an amount from their mobile money wallet and receive cash. Mobile networks typically push the fees onto the customer while splitting that with the agent who fulfils that cash out.

We will use real cash out values from this article: https://temovision.com/tnm-mpamba-withdraw-send-and-bank-charges-malawi/

After reading this article you should be able to

• Build a fairly simple micro-service with Go using standard net/http Server

• Use the golang-malawi/chigoli particularly the servicefee module for calculating service fees

• Test API end-points using Hurl and Venom

Step 1: Project setup and calculating fees from the command-line

Create a new Go module

go mod init example.com/fees-api

Add the servicefee dependency from Golang Malawi

go get github.com/golang-malawi/chigoli/servicefee

First let’s create the logic for calculating fees based on the cash out / withdrawal fee bands.

package main

import (
    "fmt"
    "log"

    "github.com/golang-malawi/chigoli/servicefee"
)

func main() {
    serviceFees := servicefee.NewFixedFee(servicefee.FeeExpressions{
        "x >= 100 &&  x <= 500":        20.0,
        "x >= 501 &&  x <= 1000":       40.0,
        "x >= 1001 &&  x <= 2000":      85.0,
        "x >= 2001 &&  x <= 3000":      150.0,
        "x >= 3001 &&  x <= 5000":      200.0,
        "x >= 5001 &&  x <= 10000":     380.0,
        "x >= 10001 &&  x <= 20000":    750.0,
        "x >= 20001 &&  x <= 40000":    1700.0,
        "x >= 40001 &&  x <= 60000":    2500.0,
        "x >= 60001 &&  x <= 100000":   3750.0,
        "x >= 100001 &&  x <= 200000":  6500.0,
        "x >= 200001 &&  x <= 300000":  9000.0,
        "x >= 300001 &&  x <= 400000":  11000.0,
        "x >= 400001 &&  x <= 500000":  12500.0,
        "x >= 500001 &&  x <= 600000":  13500.0,
        "x >= 600001 &&  x <= 750000":  14500.0,
        "x >= 750001 &&  x <= 1500000": 14500.0,
    })

    total, fee, err := serviceFees.CalculateTotalAndFee(6000)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("Total: ", total, "Fee: ", fee)
}

Let us test this simple program with go run main.go.You should get output similar to the the following:

Total: 6380 Fee: 380

If that worked, we can now go to the next step of implementing our API endpoint.

Step 2: Implementing the API Server

Create a main.go file with the following content

package main

import (
    "encoding/json"
    "errors"
    "io"
    "log"
    "log/slog"
    "net/http"

    "github.com/golang-malawi/chigoli/servicefee"
)

var ServiceFees = servicefee.NewFixedFee(servicefee.FeeExpressions{
    "x >= 100 &&  x <= 500":        20.0,
    "x >= 501 &&  x <= 1000":       40.0,
    "x >= 1001 &&  x <= 2000":      85.0,
    "x >= 2001 &&  x <= 3000":      150.0,
    "x >= 3001 &&  x <= 5000":      200.0,
    "x >= 5001 &&  x <= 10000":     380.0,
    "x >= 10001 &&  x <= 20000":    750.0,
    "x >= 20001 &&  x <= 40000":    1700.0,
    "x >= 40001 &&  x <= 60000":    2500.0,
    "x >= 60001 &&  x <= 100000":   3750.0,
    "x >= 100001 &&  x <= 200000":  6500.0,
    "x >= 200001 &&  x <= 300000":  9000.0,
    "x >= 300001 &&  x <= 400000":  11000.0,
    "x >= 400001 &&  x <= 500000":  12500.0,
    "x >= 500001 &&  x <= 600000":  13500.0,
    "x >= 600001 &&  x <= 750000":  14500.0,
    "x >= 750001 &&  x <= 1500000": 14500.0,
})

type CalculateFeeRequest struct {
    Amount float64 `json:"Amount"`
}

type FeeResponse struct {
    Amount float64 `json:"Amount"`
    Fee    float64 `json:"Fee"`
    Total  float64 `json:"Total"`
}

func CalculateFeesHandler(serviceFees servicefee.Fees) func(w http.ResponseWriter, r *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        request := CalculateFeeRequest{}
        data, err := io.ReadAll(r.Body)
        if err != nil {
            slog.Error("failed to parse request body", "error", err)
            w.WriteHeader(http.StatusBadRequest)
            w.Write([]byte("failed to parse request"))
            return
        }

        err = json.Unmarshal(data, &request)
        if err != nil {
            slog.Error("json.Unmarshal: failed to parse request body", "error", err)
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte("failed to parse request"))
            return
        }

        defer r.Body.Close()

        slog.Info("processing request for amount", "amount", request.Amount)
        total, fee, err := serviceFees.CalculateTotalAndFee(request.Amount)
        if err != nil {
            slog.Error("failed to process request", "error", err)
            if errors.Is(err, servicefee.ErrNoApplicableFee) {
                data, err := json.Marshal(FeeResponse{
                    Amount: request.Amount,
                    Total:  request.Amount,
                    Fee:    0,
                })
                if err != nil {
                    w.WriteHeader(http.StatusInternalServerError)
                    return
                }
                w.WriteHeader(http.StatusOK)
                w.Write(data)
                return
            }
            w.WriteHeader(http.StatusInternalServerError)
            return
        }

        responseData, err := json.Marshal(FeeResponse{
            Amount: request.Amount,
            Total:  total,
            Fee:    fee,
        })
        if err != nil {
            slog.Error("failed to process request got error", "error", err)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        w.Header().Add("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        w.Write(responseData)
    }
}

func main() {
    addr := ":4001"

    http.HandleFunc("/calculate-fees", CalculateFeesHandler(ServiceFees))

    slog.Info("Starting server on port", "address", addr)
    err := http.ListenAndServe(addr, nil)
    if err != nil {
        log.Fatalf("failed to run server got %v", err)
    }
}

We can run this with

$ go run main.go

Then in another terminal we can use cURL to verify it works

$ curl -X POST http://localhost:4001/calculate-fees -d '{"Amount":4000}'
# Output should be something like
{"Amount":4000,"Fee":200,"Total":4200}

Step 3: Adding Tests for our Endpoint

In this section we will implement some tests to ensure that our code works as we expect/anticipate and to catch any bugs or regressions. Tests help developers to validate their code and catch bugs, I highly recommend getting used to writing tests even if you aren’t already using a methodology like TDD (Test Driven Development)

Create a file named main_test.go and add the following content

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestServer(t *testing.T) {
    mu := http.NewServeMux()
    mu.HandleFunc("/calculate-fees", CalculateFeesHandler(ServiceFees))
    testServer := httptest.NewServer(mu)

    testCases := []struct {
        Amount        float64
        CalculatedFee float64
    }{
        {Amount: 3000, CalculatedFee: 150},
        {Amount: 4000, CalculatedFee: 200},
        {Amount: 6000, CalculatedFee: 380},
        {Amount: 20_000, CalculatedFee: 750},
    }

    for _, testCase := range testCases {
        // Create the request data
        body := []byte(fmt.Sprintf(`{"Amount": %f}`, testCase.Amount))

        res, err := http.Post(testServer.URL+"/calculate-fees", "application/json", bytes.NewBuffer(body))
        if err != nil {
            t.Errorf("failed to test server: %v", err)
        }
        defer res.Body.Close()

        data, err := io.ReadAll(res.Body)
        if err != nil {
            t.Errorf("failed to test server: %v", err)
        }
        responseStruct := FeeResponse{}
        err = json.Unmarshal(data, &responseStruct)
        if err != nil {
            t.Errorf("failed to test server: %v", err)
        }

        if responseStruct.Fee != testCase.CalculatedFee {
            t.Errorf("incorrect fee calculated. \n\twant: %f\n\thave: %f", testCase.CalculatedFee, responseStruct.Fee)
        }
    }

    t.Cleanup(func() {
        testServer.Close()
    })
}

Run the tests using the following command

$ go test ./...

Step 4: Functional Testing the API using Hurl and venom

Create a new file named test_basic.hurl in the directory with the following content:

# File: test_basic.hurl
POST http://localhost:4001/calculate-fees 
Content-Type: application/json
{
  "Amount": 6000
}

HTTP 200
[Asserts]
jsonpath "$.Fee" == 380

This file defines a Hurl file that instructs hurl to send a POST request to the /calculate-fee endpoint and checks two things 1) checks that the HTTP response status code is 200 OK and 2) Uses JSON Assertions to verify that the fee was calculated correctly.

Use the following command to run hurl to process the file and perform the tests.

$ hurl test_basic.hurl

As you can see the above file only allows us to test a hardcoded amount and fee, a better approach would be to use variables so that we can inject different amounts and fees.

See the example below

# File: test_variables.hurl
POST http://localhost:4001/calculate-fees 
Content-Type: application/json
{
  "Amount": {{amount}}
}

HTTP 200
[Asserts]
jsonpath "$.Fee" == {{fee}}

We can now test the service with variables passed via the command-line. This allows greater flexibility as we can test more cases without cluttering the hurl file with the same configuration for each test case.

$ hurl --variable amount=4000 --variable fee=200 test_variables.hurl
$ hurl --variable amount=7000 --variable fee=380 test_variables.hurl

Testing using ovh/venom

Venom is an integration testing tool. It has support for various protocols and uses YAML for defining and structuring tests. Although for our basic needs, hurl is sufficient for the tests - we will look at how we can leverage venom with its support for HTTP as well support for running binaries like hurl for performing tests.

Create a new file named test_venom.yaml and paste into it the configuration below:

name: Service fee microservice tests 
description: Tests servicefee microservice
testcases:
  - name: Test that we can shell out to hurl
    steps:
    - type: exec
      command: [ hurl,  test_basic.hurl ]
  - name: Test that fee is calculated
    steps:
    - type: http
      method: POST
      url: http://localhost:4001/calculate-fees
      body: '{"Amount": 6000}'
      assertions:
        - result.body ShouldContainSubstring Fee
        - result.bodyjson ShouldContainKey Fee
        - result.bodyjson.Fee ShouldEqual 380
        - result.statuscode ShouldEqual 200

With venom installed, use the following command to run tests:

$ venom run test_venom.yaml

The output will be something like

          [trac] writing venom.19.log
 • Service fee microservice tests (test_venom.yaml)
        • Test-that-we-can-shell-out-to-hurl PASS
        • Test-that-fee-is-calculated PASS
final status: PASS

Step 5: Building a Docker Image for our service

Now that we have build and tested the API, we will build a Docker image that can be used to run containers either through Docker locally, or a container orchestration platform like Kubernetes.

To create a Docker image, add a Dockerfile in the directory. Place the following into the Dockerfile

FROM golang:1.23-alpine as go-builder
RUN apk add --no-cache git
WORKDIR /go/
COPY . .
RUN go generate -x -v
RUN go build -o /bin/servicefee-microservice 
RUN chmod +x /bin/servicefee-microservice 

FROM alpine
ENV PORT 4001
ENV ADDRESS "0.0.0.0"
ENV CACHE_DURATION "3m"
COPY --from=go-builder /bin/servicefee-microservice  /servicefee-microservice 
CMD ["/servicefee-microservice"]

The above Dockerfile defines a multi-stage Docker build that uses the Alpine linux Operating System as a base image for building the service. The base image has Go installed so we can build the code in that environment the same way as on our local machine/laptop. After building the image, we move onto the next stage (starting at FROM alpine) in which we define the contents of the actual Docker image

With the above Dockerfile, run the following command to build a container image locally

$ docker build -t tutorial/servicefee .

Once this completes successfully, you should have a Docker image you can run as a container

$ docker run -p "4001:4001" -e "PORT=8080" tutorial/servicefee

Next Steps

• Loading Service fee configurations from a File or Remote Service

  The service fees are currently hardcoded in the code, ideally the service would load the definition of the fees from a separate file or a remote configuration service or KMS that can be updated without needing to recompile the code. This will make the service flexible to changing requirements should the fees bands ever need to change.

Conclusion

In this article we have gone through the process of implementing a basic microservice for calculating service fees. Although we used the use-case of cashout fees, the concepts in this article can be applied to different scenarios where you want to calculate a numerical value based on some bands or ranges of values. We have also seen how to use hurl and venom, two useful tools that a modern Software Engineer must have in their toolkit.

I hope you found this somewhat useful.

Please reach out for comments or if you spot some errors or places that need correction. zikani03 <at> gmail [.] com

The simple (maybe naive) interpretation of that phrase is that cattle are more expendable than pets.

See, where I come from cattle are more dear to people than pets. Actually, caring for pets too much is modern luxury. Most people keep dogs for security, not for company. Cats are for dealing with rodents. You own cattle, now that makes you wealthy! The more you own, the richer you are. That doesn’t mean they aren’t expendable, no, but someone who owns cattle will go out of their way to make sure they are bred and fed to grow. They send their sons to herd them at the expense of the child’s education or play time. They’d go on a bike or other form of transport to find feed or pay someone to find feed if the nearby pastures don’t have enough supply for the cattle to feed on.

Thinking of servers as cattle is basically saying that this is a very valuable resource that you must keep alive at all costs.

Contrast that with the western interpretation; that servers can be slaughtered and respawned without too much fan fare or worry. I found it amusing, I still do.

It makes me wonder what an African inspired server orchestration platform would look like.

Some more interesting reads on the topic:

• https://www.engineyard.com/blog/pets-vs-cattle/

• https://www.kubermatic.com/blog/cloud-native-best-practices-2-why-cattle-not-pets/

• https://dev.to/aws-heroes/time-to-rethink-cattle-vs-pets-serverless-5c0j

Mawu oti venda nthawi zambiri amatikumbutsa anyamata ndi amayi a mtawuni kapena kumsika omwe amagulitsa zinthu zosiyanasiyana monga nsapato, katundu wa magetsi ndi zina zambiri. Mavenda timawaona mosiyanabe ndi anthu a bizinesi zokhazikika. Koma tikabwela ku nkhani zokhuza Software, mawu okuti Software Vendor amatipatsa malingaliro a makampani aakulu monga Microsoft, ndi Oracle. Koteleko Software Vendor amakhala wolemekezeka ndithu.

Mkulingalira kwanga, ndikuona ngati mavenda "wambawa" akhonzabe kutiphunzitsa mmene tingapangire kuti nafe tidzafike pa mlingo wa makampani monga Microsoft ndi kumatchedwanso Software Vendor. Tiyeni tiunikile makhalidwe amavenda:

1. Venda samaopa - nthawi zambiri venda amakhala wolimba mtima ndipo samaopa kutsatsa malonda ake. Venda wamanyazi sangagulitse kanthu. Venda amatha kutsatira munthu kamtunda akuyesera kumutsatsa malonda ake.

2. Venda samachita manyazi kuonetsa malonda ake - munthu poti ufike pogula zinthu kwa venda, umakhala kuti malondawo waaona ndi maso, penanso nkuyesa komwe ngati chili chovala kapena nsapato. Venda saopa kuonetsa malonda ake ndipo amati "Kuyesa ndi ulere".

3. Venda amatha kuyendamtunda kuti agulitse - mavenda ena amatha kuyenda mtunda wautali, kutiakapeze makasitomala. Izi si zachilendo ngati munaonapo anyamata ogulitsa madzi kapena mazira. Amatha kuyenda kutali ndi komwe achokela wapansi cholinga agulitse.

4. Venda amayesa zida - Mavenda ambiri ndi ochenjera, ndipo amakonda kuyesa zida makamaka pa nkhani ya mitengo. Mavenda ambiri amangotchula mtengo wawo nadikira kuti munthu unenerere. Akatelo amakhala kuti awonjezerapo pa mtengo omwe anaodera katunduyo, ndipo akufuna kuwinapo. Ngati umatha kunenerera, udzaona kuti venda uja amatha kusitsa mtengo kwambiri kuchoka pomwe anayambira.

   Ineyo nnagulapo lamba mu dera la Lilongwe pa mtengo wa 2 sauzande, koma venda anandigulitsayo adayambira pa 12 sauzande. Sikuti ndimatha kunenelera, koma zinango chitika kuti amafunisitsa kuti apeze ndalama. Tikatelo tiwone mdemanga ina

5. Venda ntchito yake ndi kugulitsa - venda amafuna kugulitsa, akapanda kugulitsa zimakhala kuti zake sizinayende. Ndipo nthawi zambiri tsiku likamatha mumaona ma venda ena amatha kugulitsa zinthu pa mtengo wozizira, cholinga chawo chimakhala choti agulitse basi. Za mawa aziona mawa lo. Khalidwe limeneli ili ndi ubwino ndi kuyipa kwake, koma chachikulu ndichoti mtima ofuna kugulitsa tsiku limenelo ndi omwe umapangitsa venda kuti alimbikire ndipo apeze njira ndi ma kasitomala.

• Vercel for deploying projects

• Fly.io

• Docker (Desktop)

• ... and many other modern tools that generate names for things

I was thinking about this and figured it would be an interesting undertaking to try to generate some names from Chichewa words.

First of all, where to get the words. The internet is full of Chichewa speaking folks nowadays, and I guess I could just scrape a bunch of news sites and social media platforms for Chichewa content, but for this tiny experiment that's overkill. I thought we could mostly make use of this old-but-gold resource by my friend, Edmond Kachale, which he worked on years ago with his colleague Prof. Kevin Scannel.

The name gen algorithm

The algorithm is embarrassingly simple, we just shuffle the list of words and pick the first N-words from the shuffled list and join the words with a hyphen ( - )

Initially, I had tried to come up with something that picked random nouns from the Chichewa project I mentioned above but the output wasn't appealing at all. I tried to change the code to try to include the adjectives, adverbs, pronouns, and nouns in different combinations but it ended up being an exercise in linguistics which I am not qualified to perform.

In the end, I settled on a simple "algorithm" - the algorithm is embarrassingly simple, we just shuffle the list of words and pick the first N-words from the shuffled list and join the words with a hyphen ( - ). I went further to modify the code to add a suffix extracted from a Base64 encoded string to ensure that names remain somewhat unique in the face of collisions due to repetitions of the "random" order we pick words from.

I decided this would work better if I came up with my own list of words for the corpus.

package main;

import java.nio.file.*;
import java.util.*;
import java.security.*;


public class ChichewaNameGen {
    private static SecureRandom rng = new SecureRandom();

    public static void main(String... args) throws Exception {

        rng.setSeed(System.currentTimeMillis());

        var lines = Files.readAllLines(Paths.get("./words.txt"));
        var words = new ArrayList<>(lines);
        var N = Integer.parseInt(args.length >= 1 ? args[0] : "3");
        var k = Integer.parseInt(args.length >= 2 ? args[1] : "10");

        for (int i = 0; i < k; i++) {
            System.out.println(generate(words, N));
        }
    }

    public static String generate(List<String> words, int maxWords) {
        Collections.shuffle(words);    
        var w = String.join("-", words.subList(0, maxWords))
                .replace("'", "").toLowerCase();

        return w + "-" + randomHash(w);
    }

    public static String randomHash(String w) {
        var alpha = Base64.getEncoder()
                .encodeToString((System.currentTimeMillis()+ w)
                        .getBytes()).replace("=", "");

        return (alpha).substring(rng.nextInt(0, alpha.length()-9))
                .substring(0, 9);
    }
}

Creating a Chichewa word list for quirkier phrases

I wanted to have a list of words that generates names that roll off the tongue somewhat nicely or are at least quirky enough to be mildly interesting. I came up with a 100 random but relatable Chichewa words. I then placed these in the file where the code above looks. I tried several runs of tests and found the results were more pleasing.

Below is an example with a 5 random names, with length of 3 words from the list:

$ java ChichewaNameGen.java 3 5  
# output of the random words with a substring from a base64 string
fotokoza-ndemanga-tilipo-XRpbGlwbw
zikwanje-malonje-malemba-WFsZW1iYQ
matumba-chiyambi-mwamva-tbXdhbXZh
basiketi-kufika-lolemba-sb2xlbWJh
kufika-chitedze-makope-1tYWtvcGU

I am confident that those are somewhat quirky phrases. I am not going to try to translate. Okay, I can't resist with the first one ;)

Roughly (fotokoza)-(ndemanga)-(tilipo) -> (explain)-(your-thoughts/opinions)-(we-are-here/around)

How many names can we generate

The number of random names/phrases we can generate greatly depends on the number of words in the dataset. With 100 random words this approach should give more than 700 thousand random project names if we set N = 3, roughly (100 * 99 * 98 ).

The last part of the string is a 9 character long sub-string from a Base64 string and it's this bit that will help greatly with reducing collisions if we ever need to generate more names for example in some kind of service with millions of users. That should make us safe-ish from running out of random names in most applications of this kind of thing. Collisions would still occur, but we have more of a leeway for generating new names/phrases without running out of options quickly.

Conclusion

This was an interesting little experiment, I was pleased with the results and plan on using this somewhere in a future project. I might come back to this idea later and try to use the Chichewa dataset properly to create words that are built up grammatically but for now a random list of words works. Here is a link to the Gist with the 100 random chichewa words

Thanks to Seth, Chimwemwe and Walter for reviewing drafts of this post.

However, Hibernate also allows you to create custom implementations of a generator which you can use to create a custom ID generation scheme - which can be useful for generating Transaction IDs for example.

In this article, I will show you how to implement such a custom generator that will be used to generate user-friendly, somewhat predictable, IDs. The IDs will have the following form, FAN-00YY<Month-Letter><Day-Letter>#<counter> which would generate a value that looks something like thisFAN-0023JL#3642. Of course, you can amend the solution presented here to match your own needs for the format of the IDs.

Let's get coding. To enable us to test the solution easily, we will use hibernate withing a Spring Boot project - so create a new Spring Boot project with Web, JPA and Postgres as dependencies. Done? Okay, great.

Let's assume we have an SQL table to keep track of Fans (whether the electric kind or the eccentric kind is up to you). As you can see, we have defined its id column as text and as a primary key which means we must provide unique values for that field every time we want to insert/create a record.

CREATE TABLE fans (
  id text not null primary key,
  name text not null,
  created_at timestamptz
);

The associated entity class may look like this:

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;

import java.time.OffsetDateTime;

@Entity
@Table(name = "fans")
public class Fan {
    @Id
    @FanId
    private String id;

    @Column(name = "name")
    private String name;

    @Column(name="created_at")
    private OffsetDateTime createdAt;

    // getters and setters omitted...
}

The attentive reader will observe that we have added a non-standard annotation @FanId to the id field. This is our custom annotation that will be used to drive the custom ID generation. Lets create the custom annotation as shown below:-

import org.hibernate.annotations.IdGeneratorType;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import static java.lang.annotation.ElementType.FIELD;
import static java.lang.annotation.ElementType.METHOD;

@IdGeneratorType( FanIdGenerator.class )
@Retention(RetentionPolicy.RUNTIME)
@Target({ FIELD, METHOD })
public @interface FanId {
    boolean usePersistentCounter() default false;
}

The ID Generator class

This is the main class that handles the heavy lifting of the ID generation process. It is implemented as a class that implements the BeforeExecutionGenerator which means we can perform some actions before lifecycle events like Entity#save.

The generator defines an interface CounterService to which it delegates the generation of an automatically incrementing counter which forms the last part of our custom ID. In the constructor, we check if the annotation we are processing is set to use persistent counters, if so - we use a specific implementation of the counter service that stores the counters in the database - more on that below.

As you can see in the code below, we specify that we only want this class to be executed on INSERT operations/events.

import org.hibernate.HibernateException;
import org.hibernate.engine.spi.SharedSessionContractImplementor;
import org.hibernate.generator.BeforeExecutionGenerator;
import org.hibernate.generator.EventType;
import org.hibernate.generator.EventTypeSets;
import org.hibernate.id.factory.spi.CustomIdGeneratorCreationContext;

import java.lang.reflect.Member;
import java.time.OffsetDateTime;
import java.util.EnumSet;

import static org.hibernate.internal.util.ReflectHelper.getPropertyType;

public class FanIdGenerator implements BeforeExecutionGenerator {

    private static final CounterService DEFAULT_IN_PROCESS_COUNTER = new InMemoryCounterService();
    private static final char[] CHARS = new char[]{
            'A', 'B', 'C', 'D', 'E', 'F', 'G',
            'H', 'I', 'J', 'K', 'L', 'M', 'N',
            'O', 'P', 'Q', 'R', 'S', 'T', 'U',
            'V', 'W', 'X', 'Y', 'Z', '1', '2',
            '3', '4', '5', '6', '7', '8', '9'
    };

    private final CounterService counterService;

    public FanIdGenerator(
            FanId config,
            Member idMember,
            CustomIdGeneratorCreationContext creationContext) {

        final Class<?> propertyType = getPropertyType(idMember);

        if (config.usePersistentCounter()) {
            counterService = new PersistentCounterService();
        } else {
            counterService = DEFAULT_IN_PROCESS_COUNTER;
        }

        if (!String.class.isAssignableFrom(propertyType)) {
            throw new HibernateException("Unanticipated return type [" + propertyType.getName() + "] for FanId conversion");
        }
    }

    private static String formatYear(int year) {
        return String.format("00%d", year % 100);
    }

    private static char formatMonth(int dayValue) {
        return CHARS[dayValue - 1];
    }

    private static char formatDay(int monthValue) {
        return CHARS[monthValue - 1];
    }

    @Override
    public Object generate(SharedSessionContractImplementor session, Object owner, Object currentValue, EventType eventType) {

        OffsetDateTime now = OffsetDateTime.now();
        return String.format("FAN-%s%s%s#%s",
                formatYear(now.getYear()),
                formatMonth(now.getMonthValue()),
                formatDay(now.getDayOfMonth()),
                counterService.nextInteger(session, now)
        );
    }

    @Override
    public EnumSet<EventType> getEventTypes() {
        return EventTypeSets.INSERT_ONLY;
    }

    interface CounterService {
        int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate);
    }
}

The ID generation scheme was inspired by some code shared and implemented by a friend for a health information system used widely in Malawi, the code for that is here.

Integer Counter generator using in-memory counter

With this simplest implementation, the counter service uses a variable to keep track of the values generated. We use an AtomicInteger for this to enable some kind of thread safety, in case that's required. The implementation is fairly straightforward:-

import org.hibernate.engine.spi.SharedSessionContractImplementor;

import java.time.OffsetDateTime;
import java.util.concurrent.atomic.AtomicInteger;

class InMemoryCounterService implements FanIdGenerator.CounterService {
    private static final AtomicInteger id = new AtomicInteger(1000);

    @Override
    public int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate) {
        return id.incrementAndGet();
    }
}

The above approach works but has the limitation that the counter of the IDs will be reset every time the service is restarted which may lead to errors since that introduces the potential of generating duplicate keys (remember we want the IDs to be unique to use them as primary keys). We can do better by storing the counter's latest value in the database, indexed by date - this way we can be sure that we can always continue from where we left off despite restarts.

Integer Counter Generator using a value stored in a database table

Below is the implementation of the CounterService interface, named PersistentCounterService, which stores counter values in a database table to allow us to fetch easily the latest counter value to try to avoid collisions. Each date has its counter that gets incremented every time we receive a request to generate an ID for that day.

As you can see in the code, the table that keeps track of the counters is named fan_id_counters and we try to create it every time, this allows our ID Generator to be re-usable without any upfront configuration - the only implication is that the user connecting to the database should have appropriate permissions to create tables.

You could always change the implementation to remove the create call and manually create your counter-tracking table ahead of time.

import org.hibernate.engine.spi.SharedSessionContractImplementor;

import java.time.LocalDate;
import java.time.OffsetDateTime;

public class PersistentCounterService implements FanIdGenerator.CounterService {
    private static final int DEFAULT_COUNTER_START = 1000;
    private static final String SQL_CREATE_COUNTER_TABLE = """
            CREATE TABLE IF NOT EXISTS fan_id_counters (
                entry_date date primary key not null,
                counter_value integer not null default 1000
            );
            """;
    @Override
    public int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate) {
        final var today = LocalDate.now();
        var statelessSession = session.isStatelessSession() ? session.asStatelessSession() : session.getSession();
        statelessSession.createNativeMutationQuery(SQL_CREATE_COUNTER_TABLE)
                .executeUpdate();

        var list = statelessSession.createNativeQuery("SELECT counter_value FROM fan_id_counters WHERE entry_date = :date FOR UPDATE LIMIT 1;", Integer.class)
                .setParameter("date", today)
                .setFetchSize(1)
                .getResultList();
        if (list.isEmpty()) {
            statelessSession.createNativeMutationQuery("INSERT INTO fan_id_counters(entry_date, counter_value) VALUES (:date , :counter)")
                    .setParameter("date", today)
                    .setParameter("counter", DEFAULT_COUNTER_START + 1)
                    .executeUpdate();
            return DEFAULT_COUNTER_START;
        }

        statelessSession.createNativeMutationQuery("UPDATE fan_id_counters SET counter_value = counter_value + 1 WHERE entry_date = :date ;")
                .setParameter("date", today)
                .executeUpdate();

        return list.get(0);
    }
}

Let's test this thing

Now that we have the ID generator setup, we can implement a simple controller to allow us to create Fan entities to ensure that the generator does its job and also do a little bit of load testing to ensure that the generator doesn't trip up when there is a large number of requests, as one use-case would be to use it to generate Transaction IDs or Transaction Reference numbers for example.

First, let's create a simple controller that allows us to list and create Fans

@RestController
@RequestMapping("/api/v1/fans")
public class FanController {

    private final FanRepository fanRepository;

    public FanController(FanRepository fanRepository) {
        this.fanRepository = fanRepository;
    }

    @GetMapping
    public ResponseEntity<List<Fan>> createFan(Pageable pageable) {
        return ResponseEntity.ok(fanRepository.findAll(pageable).toList());
    }

    @PostMapping
    public ResponseEntity<Fan> createFan(@RequestParam("name") String name) {
        Fan fan = new Fan();
        fan.setName(name);
        fan.setCreatedAt(OffsetDateTime.now());
        fanRepository.saveAndFlush(fan);
        return ResponseEntity.ok(fan);
    }
}

We can use the following cURL request to create a Fan

curl -X POST "http://localhost:8080/api/v1/fans?name=James"

This could return something like this

{
    "id": "FAN-0023JL#1000",
    "name": "James",
    "createdAt": "2023-08-12T21:04:44.135036Z"
},

Load testing the ID Generator

We can now use a load testing tool like Hey to load test the server - which will send about 1000 requests with the command below.

hey -n 1000 -m POST "http://localhost:8080/api/v1/fans?name=James"

Here is a screenshot that shows a sample run on my environment, not so bad :) -

We can check in the database and verify that each fan has a uniquely generated ID.

Conclusion

In this article, we have seen how to implement a custom ID generator with Hibernate that uses a custom format that includes an auto-incrementing counter that has two implementations, one in-memory and another that stores the most recent counter in a database table indexed by calendar day. You can adapt the code here to implement your ID generator which can be useful for things like Transaction IDs or Transaction References that are somewhat more user friendly.

Thank you for reading.

Order results but select a specific first-row

This trick allows you to write a query with an order-by clause which allows you to choose which row should come first based on some condition, despite the presence of another order-by clause.

See the example below which will always return the company with ID = 2 as the first row despite how the other order by clause is written.

CREATE TABLE companies_test (
  id serial not null,
  company_name text not null,
  created_at timestamp default now()
);

INSERT INTO companies_test (company_name)
VALUES ('Company A'), ('Company B'),('Company C');

select id, company_name, created_at 
from companies_test
order by (id = 2) desc, created_at asc

Using string aggregates to return arrays for related records

The string_agg function is a good function in Postgres that allows you to aggregate columns after a group by operations.

Let's say you have a database of individuals who share the same first and last names and you want to capture all the dates of birth linked to that name in the database, (sorry if this is a quirky example, could only think of it as it's something I faced) you could do something like this:

CREATE TABLE IF NOT EXISTS individuals_test (
  id serial not null,
  first_name text not null,
  last_name text not null,
  date_of_birth date,
  created_at timestamp default now()
);

INSERT INTO individuals_test (first_name, last_name, date_of_birth)
VALUES ('John', 'Doe', '1970-01-01'), ('John', 'Doe', '2003-10-01'),('John', 'Doe', '1993-04-01');

 SELECT 
   concat(first_name, ' ', last_name) as full_name, 
   string_agg(date_of_birth::text, ';') as dates_of_birth 
 FROM individuals_test 
 GROUP BY concat(first_name, ' ', last_name) ORDER BY full_name

The result will look something like

| full_name | dates_of_birth                   |
| --------- | -------------------------------- |
| John Doe  | 1970-01-01;2003-10-01;1993-04-01 |

Checking if a record exists with a true or false result

Most times we have some parts in our code where we would just like to know if a record exists. Here is a pattern for a query that always guarantees a result that will be either true or false depending on the result of the subquery - this way you don't have to check if the resultset from your query is empty or not:

SELECT exists (select from users where username = :username);

CHECK constraint to validate a JSON column

Some days you may find yourself storing JSON in a database column - now the thing about JSON is that you will have to deal with a Schema somewhere, usually in your application code. Here is a simple approach to check that the JSON has at least the required top-level keys as a simple validation to ensure incorrectly formatted JSON isn't entered in your columns. You can go crazy with this and validate against a whole JSON Schema - take a look at pg_jsonschema by the Supabase team

CREATE TABLE IF NOT EXISTS user_app_settings (
  username text not null,
  settings jsonb
);

ALTER TABLE user_app_settings 
ADD CONSTRAINT check_settings_fields 
     CHECK (settings::jsonb ?& array['cover_photo', 'current_theme', 'toolbar_x', 'toolbar_y', 'version']);

 -- this insert will work because the JSON is well-formed according to our rules
 INSERT INTO user_app_settings
 VALUES ('zikani03', '{"version": "v1.0-beta", "cover_photo": "https://some.site/pic/zikani03", "current_theme":"nord", "toolbar_x": 354.5, "toolbar_y": 100}'::jsonb);

 -- this insert below should fail since we don't have the "version" field 
 INSERT INTO user_app_settings
 VALUES ('john', '{"no_version": "v1.0-beta", "cover_photo": "https://some.site/pic/john", "current_theme":"nord", "toolbar_x": 354.5, "toolbar_y": 100}'::jsonb);

Conclusion

In this article, I shared a few tricks you can use in your projects that use Postgresql, though some of them would also likely work in other DBMSs if you found the equivalent functions. I hope to share more as I keep learning...

Thanks for reading.

This very thing happened to me recently and I wanted to document solutions to the problem, I will tell you which I went with at the end.

Let's create a new Spring Boot project to get a better picture of the issue at hand.

Let's add a Person class that looks like this. Notice that we have thrown in Hibernate Validator annotations to validate the data, like the good backend developers we are.

import com.fasterxml.jackson.annotation.JsonProperty;
import jakarta.validation.constraints.Email;
import jakarta.validation.constraints.NotEmpty;
import org.hibernate.validator.constraints.Length;

public class Person {

    @NotEmpty
    @Length(min = 2, max = 120)
    @JsonProperty("full_name")
    private String fullName;

    @Email
    @JsonProperty("email")
    private String email;

    @NotEmpty
    @Length(min = 8, max = 50)
    @JsonProperty("phone_number")
    private String phoneNumber;


    @JsonProperty("country_iso")
    private String countryISO;

    // getters and setters omitted ...
}

And a Spring Boot controller that just validates the data and returns it

// imports omitted for brevity

@RestController
@RequestMapping("/people")
public class PeopleController {

    @PostMapping
    public ResponseEntity<Person> handlePost(@Validated @RequestBody Person request) {
        return ResponseEntity.ok(request);
    }
}

We want our clients to send us data that looks like this

{
  "full_name": "John Banda",
  "phone_number": "",
  "email": "john.banda@example.com",
  "country_iso": "MWI"
}

Now the client sends a request with invalid data (let's assume it's missing the full_name and phone_number fields) and the API, like any good API, returns a good ol' 400 Bad Request to let them know the error of their ways. To figure out how they may have screwed that request up, the client checks the response body and discovers the following:

{
    "timestamp": "2023-09-27T20:22:29.627+00:00",
    "status": 400,
    "error": "Bad Request",
    "path": "/people"
}

From the status code, they can guess it's an error on their end, but the response doesn't show if it's a validation error or something else wrong with the request data. Now the client isn't sure if they sent the request with the rightly named fields in the first place or if it's our API that's gone off the Rails (if there is a pun here, it is intended ;).

Sending validation error response

We would rather give the client a bit more information about why their request is bad. A good way to do that is to catch the validation errors and return a 400 BadRequest with actual details about the validation errors. Let's create a ValidationErrorsResponse class with the following code:

public class ValidationErrorsResponse {
    private int code;
    private String message;
    private Map<String, List<String>> validationErrors;

    public ValidationErrorsResponse(int code, String message) {
        this.code = code;
        this.message = message;
        this.validationErrors = new HashMap<>();
    }

    public void put(String field, String error) {
        validationErrors.computeIfAbsent(field, s -> new ArrayList<>()).add(error);
    }

    // getters and setters omitted ...   
}

Now we will need to register an Exception handler for the validation error which we can register against a MethodArgumentNotValidException

public class ExceptionHandlers {

    @ResponseStatus(HttpStatus.BAD_REQUEST)
    @ExceptionHandler(MethodArgumentNotValidException.class)
    public final ResponseEntity<ValidationErrorsResponse> handleExceptions(MethodArgumentNotValidException ex) {
        final List<FieldError> fieldErrors = ex.getBindingResult().getFieldErrors();

        ValidationErrorsResponse errorResponse = new ValidationErrorsResponse(
                HttpStatus.BAD_REQUEST.value(),
                HttpStatus.BAD_REQUEST.getReasonPhrase()
        );

        for (FieldError fieldError : fieldErrors) {
            errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());
        }

        return ResponseEntity.badRequest().body(errorResponse);
    }
}

It's at this point that if we try to send a POST with an empty or invalid object {} to the /people endpoint, we will get a response similar to the one below, which as you can see shows the fields in camelCase form.

{
    "code": 400,
    "message": "Bad Request",
    "validationErrors": {
        "fullName": [
            "must not be empty"
        ],
        "phoneNumber": [
            "must not be empty"
        ]
    }
}

While this works and in other situations would be sufficient, it is not great for us because the client will now get confused about the naming of the fields; the validation error response shows the fields named in camelCaseForm whereas the request data is expected to be sent in snake_case_form. The very problem we want to avoid!

We have some possible solutions to fix this issue, explained below.

Solution 1: Rename the class fields to the snake case form

So one quick solution to this problem is the naming of the fields, you can just change it to match the @JsonProperty value. This has the downside that your linter or co-worker may shout at you for naming fields "out-of-standard" but fixes the issue without any over-engineering required. The con is that you have to name all the fields in all your other API objects in this way, so don't forget!

public class Person {

    @NotEmpty
    @Length(min = 2, max = 120)
    @JsonProperty("full_name")
    private String full_name;

    @Email
    @JsonProperty("email")
    private String email;

    @NotEmpty
    @Length(min = 8, max = 50)
    @JsonProperty("phone_number")
    private String phone_number;


    @JsonProperty("country_iso")
    private String country_iso;

    // getters and setters omitted ...
}

Solution 2: Improve the exception handler to maintain standards

Another approach to this is to do a little "over-engineering" to solve the problem in a way that maintains your strongly held opinions and standards while ensuring that the client will never be confused about what's what. This approach involves putting a bit more code in the exception handler to check if the field we are dealing with has a @JsonProperty annotation and then uses the value() from that annotation as the field name when building up validation errors, if we can't find the field, we fall back to sending the regular Bean name of the field/property.

We don't have to make any changes to the names of the fields on the Person or any other class. Here is how we could solve the problem with this approach:-

    @ResponseStatus(HttpStatus.BAD_REQUEST)
    @ExceptionHandler(MethodArgumentNotValidException.class)
    public final ResponseEntity<ValidationErrorsResponse> handleExceptions(MethodArgumentNotValidException ex) {
        final List<FieldError> fieldErrors = ex.getBindingResult().getFieldErrors();

        ValidationErrorsResponse errorResponse = new ValidationErrorsResponse(
                HttpStatus.BAD_REQUEST.value(),
                HttpStatus.BAD_REQUEST.getReasonPhrase()
        );

        for (FieldError fieldError : fieldErrors) {
            try {
                var beanClazz = ex.getTarget().getClass();
                var field = beanClazz.getDeclaredField(fieldError.getField());
                var jsonPropertyAnnotation = field.getAnnotation(JsonProperty.class);
                if (jsonPropertyAnnotation != null) {
                    errorResponse.put(jsonPropertyAnnotation.value(), fieldError.getDefaultMessage());
                } else {
                    LoggerFactory.getLogger(getClass()).debug("Failed to get annotated field at {}", fieldError.getField());
                    errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());
                }
            } catch (NoSuchFieldException e) {
                LoggerFactory.getLogger(getClass()).error("Failed to get annotated field", e);
                errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());
            }
        }

        return ResponseEntity.badRequest().body(errorResponse);
    }

Solution 3: Use a consistent naming convention (i.e. camelCase)

Another option is to update your API data fields + docs to use camelCasedFields as a standard and avoid compromising on Java bean standards and introducing inconsistencies. This is probably not the most practical solution if your API is already used by clients in production unless you manage the change/upgrade to a new version very well. You may also face resistance from API clients who insist on dealing with fields named in snake_case_form.

There is no code for this section because the solution requires communication and coordination with your teams and clients, which you cannot code around. YMMV.

Thanks for reading.

Ohh, I promised to tell you which solution I went with. I went for Solution 1 as a quick fix but after reviewing the other API objects + endpoints in our code, I applied Solution 2 which helped avoid renaming a whole bunch of API object classes. Solution 2 works well with simple flat objects and isn't robust for nested structures. Will write a follow-up soon.