• Vercel for deploying projects

• Fly.io

• Docker (Desktop)

• ... and many other modern tools that generate names for things

I was thinking about this and figured it would be an interesting undertaking to try to generate some names from Chichewa words.

First of all, where to get the words. The internet is full of Chichewa speaking folks nowadays, and I guess I could just scrape a bunch of news sites and social media platforms for Chichewa content, but for this tiny experiment that's overkill. I thought we could mostly make use of this old-but-gold resource by my friend, Edmond Kachale, which he worked on years ago with his colleague Prof. Kevin Scannel.

The name gen algorithm

The algorithm is embarrassingly simple, we just shuffle the list of words and pick the first N-words from the shuffled list and join the words with a hyphen ( - )

Initially, I had tried to come up with something that picked random nouns from the Chichewa project I mentioned above but the output wasn't appealing at all. I tried to change the code to try to include the adjectives, adverbs, pronouns, and nouns in different combinations but it ended up being an exercise in linguistics which I am not qualified to perform.

In the end, I settled on a simple "algorithm" - the algorithm is embarrassingly simple, we just shuffle the list of words and pick the first N-words from the shuffled list and join the words with a hyphen ( - ). I went further to modify the code to add a suffix extracted from a Base64 encoded string to ensure that names remain somewhat unique in the face of collisions due to repetitions of the "random" order we pick words from.

I decided this would work better if I came up with my own list of words for the corpus.

package main;import java.nio.file.*;import java.util.*;import java.security.*;public class ChichewaNameGen {    private static SecureRandom rng = new SecureRandom();    public static void main(String... args) throws Exception {        rng.setSeed(System.currentTimeMillis());        var lines = Files.readAllLines(Paths.get("./words.txt"));        var words = new ArrayList<>(lines);        var N = Integer.parseInt(args.length >= 1 ? args[0] : "3");        var k = Integer.parseInt(args.length >= 2 ? args[1] : "10");        for (int i = 0; i < k; i++) {            System.out.println(generate(words, N));        }    }    public static String generate(List<String> words, int maxWords) {        Collections.shuffle(words);            var w = String.join("-", words.subList(0, maxWords))                .replace("'", "").toLowerCase();        return w + "-" + randomHash(w);    }    public static String randomHash(String w) {        var alpha = Base64.getEncoder()                .encodeToString((System.currentTimeMillis()+ w)                        .getBytes()).replace("=", "");        return (alpha).substring(rng.nextInt(0, alpha.length()-9))                .substring(0, 9);    }}

Creating a Chichewa word list for quirkier phrases

I wanted to have a list of words that generates names that roll off the tongue somewhat nicely or are at least quirky enough to be mildly interesting. I came up with a 100 random but relatable Chichewa words. I then placed these in the file where the code above looks. I tried several runs of tests and found the results were more pleasing.

Below is an example with a 5 random names, with length of 3 words from the list:

$ java ChichewaNameGen.java 3 5  # output of the random words with a substring from a base64 stringfotokoza-ndemanga-tilipo-XRpbGlwbwzikwanje-malonje-malemba-WFsZW1iYQmatumba-chiyambi-mwamva-tbXdhbXZhbasiketi-kufika-lolemba-sb2xlbWJhkufika-chitedze-makope-1tYWtvcGU

I am confident that those are somewhat quirky phrases. I am not going to try to translate. Okay, I can't resist with the first one ;)

Roughly (fotokoza)-(ndemanga)-(tilipo) -> (explain)-(your-thoughts/opinions)-(we-are-here/around)

How many names can we generate

The number of random names/phrases we can generate greatly depends on the number of words in the dataset. With 100 random words this approach should give more than 700 thousand random project names if we set N = 3, roughly (100 * 99 * 98 ).

The last part of the string is a 9 character long sub-string from a Base64 string and it's this bit that will help greatly with reducing collisions if we ever need to generate more names for example in some kind of service with millions of users. That should make us safe-ish from running out of random names in most applications of this kind of thing. Collisions would still occur, but we have more of a leeway for generating new names/phrases without running out of options quickly.

Conclusion

This was an interesting little experiment, I was pleased with the results and plan on using this somewhere in a future project. I might come back to this idea later and try to use the Chichewa dataset properly to create words that are built up grammatically but for now a random list of words works. Here is a link to the Gist with the 100 random chichewa words

Thanks to Seth, Chimwemwe and Walter for reviewing drafts of this post.

As a young Malawian bachelor (some rude people would say senior bachelor), I often find myself at the kitchen sink doing dishes. Not only because it's the one skill I am confident at in the kitchen when I compare myself to my house mate, particularly his cooking skills, but also because it's the one activity I have found that somehow helps me think a lot more than other activities.

There are few places in a house where one can retreat to have some time to think. Many of us retreat to the revered W.C. 🚽, others will walk about in a garden if they have one, others laze around the couch or if they are fancy on a Hammock, still more others lay in bed looking up at the ceiling as they gather their thoughts. (Side note: I intend to try to be fancy one day because of this talk by Rich Hickey).

As for me, I long for dirty dishes and some time at the kitchen sink.

The Kitchen Sink Theory

Why is the Kitchen Sink so amenable to thinking? I have thought about this a bit (no doubt at the Kitchen Sink) but can only offer a theory: Washing dishes is an activity that has a predictable, well-defined process and outcome that it can be done almost mindlessly, but not completely. You cannot mindlessly handle a knife, soapy ceramic plate or glass tumbler. Even though it's a somewhat mindless activity, it leaves one room to spend time thinking about other things throughout the activity. Washing dishes is one of the most delightful activities because it leaves you with a clean kitchen sink, countertop, clean dishes, and a sense of completion - if done right.

What to think about at the Kitchen Sink ?

I tend to think about about life, relationships, programming, work related issues or new ideas. One limitation of working the mind at the kitchen sink is that my hands are arrested by the very dishes, cutlery, water and dish washing liquid and I cannot write down what I am thinking of immediately. Taking a break to write would disrupt the flow of cleaning and thinking so I usually delegate that to later, if I don't get distracted by another thought by the time I am leaving the kitchen. Most of the time the thoughts do stick or lead to some action; whether finding something to read, research on, write down or code up.

There aren't enough dirty dishes, I need an alternative.

Despite the fact that this works for me, the most glaring limitation is that once you are done washing dishes, that's the end of the session. It makes no sense to wash dishes for the sake of washing or just as an excuse to think (though I get tempted to do so every now and then).

Therefore I am looking for an alternative activity, something similarly mindless, that will not wait on me to have dirty dishes at my disposal.

What would you suggest ?

Thank you to my lovely fiance for helping me edit this post. Thanks for being patient when I'm doing dishes.

Markdown is a popular authoring format and has gained wide adoption in many different places. Markdown is generally converted to HTML, but there do exist tools like Pandoc which can additionally convert markdown to different formats like .docx documents, PDF etc...

In the Go ecosystem, a popular library for working with Markdown is goldmark - we which we will use to process user-submitted markdown text and render it to a PDF file. You can imagine this being used for basic authoring of content including letters, articles, simple newsletters, basic reports etc...

Markdown PDF generator backend

For us to process Markdown, we will rely on the popular library goldmark which has some community-built extensions, the one we are particularly interested in is the goldmark-pdf library which will allow us to render Goldmark's Markdown AST into PDF. Firstly, we will create a bare-bones (i.e. not production-grade) web server using the standard library's net/http.

The complete server code is shown below:

package mainimport (    "bytes"    "context"    "encoding/base64"    "fmt"    "io/ioutil"    "net/http"    "os"    "text/template"    pdf "github.com/stephenafamo/goldmark-pdf"    "github.com/yuin/goldmark")func main() {    markdown := goldmark.New(        goldmark.WithRenderer(            pdf.New(                pdf.WithTraceWriter(os.Stdout),                pdf.WithContext(context.Background()),                pdf.WithImageFS(os.DirFS(".")),                pdf.WithHeadingFont(pdf.GetTextFont("Arial", pdf.FontCourier)),                pdf.WithBodyFont(pdf.GetTextFont("Arial", pdf.FontCourier)),                pdf.WithCodeFont(pdf.GetCodeFont("Arial", pdf.FontCourier)),            ),        ),    )    mux := http.NewServeMux()    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {        w.Header().Add("Cache-Control", "cache")        indexHtmlTemplate := template.Must(template.ParseFiles("index.html"))        indexHtmlTemplate.ExecuteTemplate(w, "index.html", nil)    })    mux.HandleFunc("/generate", func(w http.ResponseWriter, r *http.Request) {        source, err := ioutil.ReadAll(r.Body)        if err != nil {            http.Error(w, err.Error(), http.StatusInternalServerError)            return        }        var buf bytes.Buffer        if err := markdown.Convert([]byte(source), &buf); err != nil {            http.Error(w, "failed to generate pdf", http.StatusInternalServerError)            return        }        base64Encoded := base64.StdEncoding.EncodeToString(buf.Bytes())        dataURL := fmt.Sprintf("data:application/octet-stream;base64,%s", base64Encoded)        w.Write([]byte(dataURL))    })    http.ListenAndServe(":8001", mux)}

The frontend - adding an in-browser editor

We are going to complete our simple service by adding a frontend to it which is going to be a simple Markdown Editor with a button for Generating and Downloading the PDF. We will use EasyMDE for this - I went through several different options for an editor and at the moment I like EasyMDE, especially for how easy it is to add extra functionality to the Editor. Will will be extending EasyMDE to enable downloading the generated PDF by adding a button with a PDF icon for the user to generate the PDF.

The complete code for the index.html page is shown below

<!DOCTYPE html><html lang="en"><head>    <meta charset="UTF-8">    <meta http-equiv="X-UA-Compatible" content="IE=edge">    <meta name="viewport" content="width=device-width, initial-scale=1.0">    <title>Markdown to PDF</title>    <base href="/">    <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css" rel="stylesheet"        integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">    <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/styles/default.min.css">    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/js/bootstrap.bundle.min.js"        integrity="sha384-pjaaA8dDz/5BgdFUPX6M/9SUZv4d12SUPF0axWc+VRZkx5xU3daN+lYb49+Ax+Tl"        crossorigin="anonymous"></script>    <script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/highlight.min.js"></script>    <script>hljs.initHighlightingOnLoad();</script>    <script defer src="https://use.fontawesome.com/releases/v5.4.1/js/all.js"        integrity="sha384-L469/ELG4Bg9sDQbl0hvjMq8pOcqFgkSpwhwnslzvVVGpDjYJ6wJJyYjvG3u8XW7"        crossorigin="anonymous"></script>    <script src="https://unpkg.com/axios@0.26.0/dist/axios.min.js"></script>    <link rel="stylesheet" href="https://unpkg.com/easymde@2.18.0/dist/easymde.min.css">    <script src="https://unpkg.com/easymde@2.18.0/dist/easymde.min.js"></script></head><body>    <div class="content">        <div id="app">            <textarea id="editor"></textarea>            <a id="downloadEl" href="#" style="display: none;" download="Document.pdf">Download PDF</a>        </div>    </div>    <script type="text/javascript">        let downloadEl = document.getElementById('downloadEl')        new EasyMDE({            autoDownloadFontAwesome: false,            toolbar: [            "bold", "italic", "heading", "|", "quote", 'strikethrough', 'code', 'heading','|', 'undo','redo'            ,'|', { // Separator                name: "generate-pdf",                action: (editor) => {                    axios.post("/generate", editor.value())                    .then(response => {                        let pdfDataURL = response.data                        if (pdfDataURL.indexOf("base64") > 0) {                            downloadEl.setAttribute('href', pdfDataURL)                            downloadEl.click()                        }                    })                },                className: "fa fa-file-pdf",                title: "Download PDF",            }, '|', {                name: "link",                action: 'https://blog.nndi.cloud',                className: "fa fa-globe",                title: "Go to the NNDI Blog"            }],            element: document.getElementById('editor'),            initialValue: '## Welcome to Markdown to PDF using Go.\nType some markdown in here and then click the PDF button to render the markdown to PDF'        });    </script></body></html>

With those two bits of code done, we can now build and run our service. Make sure you copy the Go code into a file named main.go and the HTML code in a file named index.html in the same directory - you can then run the following commands:

$ go mod init markdown-pdf$ go mod tidy$ go run main.go

The last command should start a server on http://localhost:8001 visit that in your browser and try it out.

Thanks for reading.

However, Hibernate also allows you to create custom implementations of a generator which you can use to create a custom ID generation scheme - which can be useful for generating Transaction IDs for example.

In this article, I will show you how to implement such a custom generator that will be used to generate user-friendly, somewhat predictable, IDs. The IDs will have the following form, FAN-00YY<Month-Letter><Day-Letter>#<counter> which would generate a value that looks something like thisFAN-0023JL#3642. Of course, you can amend the solution presented here to match your own needs for the format of the IDs.

Let's get coding. To enable us to test the solution easily, we will use hibernate withing a Spring Boot project - so create a new Spring Boot project with Web, JPA and Postgres as dependencies. Done? Okay, great.

Let's assume we have an SQL table to keep track of Fans (whether the electric kind or the eccentric kind is up to you). As you can see, we have defined its id column as text and as a primary key which means we must provide unique values for that field every time we want to insert/create a record.

CREATE TABLE fans (  id text not null primary key,  name text not null,  created_at timestamptz);

The associated entity class may look like this:

import jakarta.persistence.Column;import jakarta.persistence.Entity;import jakarta.persistence.Id;import jakarta.persistence.Table;import java.time.OffsetDateTime;@Entity@Table(name = "fans")public class Fan {    @Id    @FanId    private String id;    @Column(name = "name")    private String name;    @Column(name="created_at")    private OffsetDateTime createdAt;    // getters and setters omitted...}

The attentive reader will observe that we have added a non-standard annotation @FanId to the id field. This is our custom annotation that will be used to drive the custom ID generation. Lets create the custom annotation as shown below:-

import org.hibernate.annotations.IdGeneratorType;import java.lang.annotation.Retention;import java.lang.annotation.RetentionPolicy;import java.lang.annotation.Target;import static java.lang.annotation.ElementType.FIELD;import static java.lang.annotation.ElementType.METHOD;@IdGeneratorType( FanIdGenerator.class )@Retention(RetentionPolicy.RUNTIME)@Target({ FIELD, METHOD })public @interface FanId {    boolean usePersistentCounter() default false;}

The ID Generator class

This is the main class that handles the heavy lifting of the ID generation process. It is implemented as a class that implements the BeforeExecutionGenerator which means we can perform some actions before lifecycle events like Entity#save.

The generator defines an interface CounterService to which it delegates the generation of an automatically incrementing counter which forms the last part of our custom ID. In the constructor, we check if the annotation we are processing is set to use persistent counters, if so - we use a specific implementation of the counter service that stores the counters in the database - more on that below.

As you can see in the code below, we specify that we only want this class to be executed on INSERT operations/events.

import org.hibernate.HibernateException;import org.hibernate.engine.spi.SharedSessionContractImplementor;import org.hibernate.generator.BeforeExecutionGenerator;import org.hibernate.generator.EventType;import org.hibernate.generator.EventTypeSets;import org.hibernate.id.factory.spi.CustomIdGeneratorCreationContext;import java.lang.reflect.Member;import java.time.OffsetDateTime;import java.util.EnumSet;import static org.hibernate.internal.util.ReflectHelper.getPropertyType;public class FanIdGenerator implements BeforeExecutionGenerator {    private static final CounterService DEFAULT_IN_PROCESS_COUNTER = new InMemoryCounterService();    private static final char[] CHARS = new char[]{            'A', 'B', 'C', 'D', 'E', 'F', 'G',            'H', 'I', 'J', 'K', 'L', 'M', 'N',            'O', 'P', 'Q', 'R', 'S', 'T', 'U',            'V', 'W', 'X', 'Y', 'Z', '1', '2',            '3', '4', '5', '6', '7', '8', '9'    };    private final CounterService counterService;    public FanIdGenerator(            FanId config,            Member idMember,            CustomIdGeneratorCreationContext creationContext) {        final Class<?> propertyType = getPropertyType(idMember);        if (config.usePersistentCounter()) {            counterService = new PersistentCounterService();        } else {            counterService = DEFAULT_IN_PROCESS_COUNTER;        }        if (!String.class.isAssignableFrom(propertyType)) {            throw new HibernateException("Unanticipated return type [" + propertyType.getName() + "] for FanId conversion");        }    }    private static String formatYear(int year) {        return String.format("00%d", year % 100);    }    private static char formatMonth(int dayValue) {        return CHARS[dayValue - 1];    }    private static char formatDay(int monthValue) {        return CHARS[monthValue - 1];    }    @Override    public Object generate(SharedSessionContractImplementor session, Object owner, Object currentValue, EventType eventType) {        OffsetDateTime now = OffsetDateTime.now();        return String.format("FAN-%s%s%s#%s",                formatYear(now.getYear()),                formatMonth(now.getMonthValue()),                formatDay(now.getDayOfMonth()),                counterService.nextInteger(session, now)        );    }    @Override    public EnumSet<EventType> getEventTypes() {        return EventTypeSets.INSERT_ONLY;    }    interface CounterService {        int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate);    }}

The ID generation scheme was inspired by some code shared and implemented by a friend for a health information system used widely in Malawi, the code for that is here.

Integer Counter generator using in-memory counter

With this simplest implementation, the counter service uses a variable to keep track of the values generated. We use an AtomicInteger for this to enable some kind of thread safety, in case that's required. The implementation is fairly straightforward:-

import org.hibernate.engine.spi.SharedSessionContractImplementor;import java.time.OffsetDateTime;import java.util.concurrent.atomic.AtomicInteger;class InMemoryCounterService implements FanIdGenerator.CounterService {    private static final AtomicInteger id = new AtomicInteger(1000);    @Override    public int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate) {        return id.incrementAndGet();    }}

The above approach works but has the limitation that the counter of the IDs will be reset every time the service is restarted which may lead to errors since that introduces the potential of generating duplicate keys (remember we want the IDs to be unique to use them as primary keys). We can do better by storing the counter's latest value in the database, indexed by date - this way we can be sure that we can always continue from where we left off despite restarts.

Integer Counter Generator using a value stored in a database table

Below is the implementation of the CounterService interface, named PersistentCounterService, which stores counter values in a database table to allow us to fetch easily the latest counter value to try to avoid collisions. Each date has its counter that gets incremented every time we receive a request to generate an ID for that day.

As you can see in the code, the table that keeps track of the counters is named fan_id_counters and we try to create it every time, this allows our ID Generator to be re-usable without any upfront configuration - the only implication is that the user connecting to the database should have appropriate permissions to create tables.

You could always change the implementation to remove the create call and manually create your counter-tracking table ahead of time.

import org.hibernate.engine.spi.SharedSessionContractImplementor;import java.time.LocalDate;import java.time.OffsetDateTime;public class PersistentCounterService implements FanIdGenerator.CounterService {    private static final int DEFAULT_COUNTER_START = 1000;    private static final String SQL_CREATE_COUNTER_TABLE = """            CREATE TABLE IF NOT EXISTS fan_id_counters (                entry_date date primary key not null,                counter_value integer not null default 1000            );            """;    @Override    public int nextInteger(SharedSessionContractImplementor session, OffsetDateTime onDate) {        final var today = LocalDate.now();        var statelessSession = session.isStatelessSession() ? session.asStatelessSession() : session.getSession();        statelessSession.createNativeMutationQuery(SQL_CREATE_COUNTER_TABLE)                .executeUpdate();        var list = statelessSession.createNativeQuery("SELECT counter_value FROM fan_id_counters WHERE entry_date = :date FOR UPDATE LIMIT 1;", Integer.class)                .setParameter("date", today)                .setFetchSize(1)                .getResultList();        if (list.isEmpty()) {            statelessSession.createNativeMutationQuery("INSERT INTO fan_id_counters(entry_date, counter_value) VALUES (:date , :counter)")                    .setParameter("date", today)                    .setParameter("counter", DEFAULT_COUNTER_START + 1)                    .executeUpdate();            return DEFAULT_COUNTER_START;        }        statelessSession.createNativeMutationQuery("UPDATE fan_id_counters SET counter_value = counter_value + 1 WHERE entry_date = :date ;")                .setParameter("date", today)                .executeUpdate();        return list.get(0);    }}

Let's test this thing

Now that we have the ID generator setup, we can implement a simple controller to allow us to create Fan entities to ensure that the generator does its job and also do a little bit of load testing to ensure that the generator doesn't trip up when there is a large number of requests, as one use-case would be to use it to generate Transaction IDs or Transaction Reference numbers for example.

First, let's create a simple controller that allows us to list and create Fans

@RestController@RequestMapping("/api/v1/fans")public class FanController {    private final FanRepository fanRepository;    public FanController(FanRepository fanRepository) {        this.fanRepository = fanRepository;    }    @GetMapping    public ResponseEntity<List<Fan>> createFan(Pageable pageable) {        return ResponseEntity.ok(fanRepository.findAll(pageable).toList());    }    @PostMapping    public ResponseEntity<Fan> createFan(@RequestParam("name") String name) {        Fan fan = new Fan();        fan.setName(name);        fan.setCreatedAt(OffsetDateTime.now());        fanRepository.saveAndFlush(fan);        return ResponseEntity.ok(fan);    }}

We can use the following cURL request to create a Fan

curl -X POST "http://localhost:8080/api/v1/fans?name=James"

This could return something like this

{    "id": "FAN-0023JL#1000",    "name": "James",    "createdAt": "2023-08-12T21:04:44.135036Z"},

Load testing the ID Generator

We can now use a load testing tool like Hey to load test the server - which will send about 1000 requests with the command below.

hey -n 1000 -m POST "http://localhost:8080/api/v1/fans?name=James"

Here is a screenshot that shows a sample run on my environment, not so bad :) -

We can check in the database and verify that each fan has a uniquely generated ID.

Conclusion

In this article, we have seen how to implement a custom ID generator with Hibernate that uses a custom format that includes an auto-incrementing counter that has two implementations, one in-memory and another that stores the most recent counter in a database table indexed by calendar day. You can adapt the code here to implement your ID generator which can be useful for things like Transaction IDs or Transaction References that are somewhat more user friendly.

Thank you for reading.

Order results but select a specific first-row

This trick allows you to write a query with an order-by clause which allows you to choose which row should come first based on some condition, despite the presence of another order-by clause.

See the example below which will always return the company with ID = 2 as the first row despite how the other order by clause is written.

CREATE TABLE companies_test (  id serial not null,  company_name text not null,  created_at timestamp default now());INSERT INTO companies_test (company_name)VALUES ('Company A'), ('Company B'),('Company C');select id, company_name, created_at from companies_testorder by (id = 2) desc, created_at asc

Using string aggregates to return arrays for related records

The string_agg function is a good function in Postgres that allows you to aggregate columns after a group by operations.

Let's say you have a database of individuals who share the same first and last names and you want to capture all the dates of birth linked to that name in the database, (sorry if this is a quirky example, could only think of it as it's something I faced) you could do something like this:

CREATE TABLE IF NOT EXISTS individuals_test (  id serial not null,  first_name text not null,  last_name text not null,  date_of_birth date,  created_at timestamp default now());INSERT INTO individuals_test (first_name, last_name, date_of_birth)VALUES ('John', 'Doe', '1970-01-01'), ('John', 'Doe', '2003-10-01'),('John', 'Doe', '1993-04-01'); SELECT    concat(first_name, ' ', last_name) as full_name,    string_agg(date_of_birth::text, ';') as dates_of_birth  FROM individuals_test  GROUP BY concat(first_name, ' ', last_name) ORDER BY full_name

The result will look something like

| full_name | dates_of_birth                   || --------- | -------------------------------- || John Doe  | 1970-01-01;2003-10-01;1993-04-01 |

Checking if a record exists with a true or false result

Most times we have some parts in our code where we would just like to know if a record exists. Here is a pattern for a query that always guarantees a result that will be either true or false depending on the result of the subquery - this way you don't have to check if the resultset from your query is empty or not:

SELECT exists (select from users where username = :username);

CHECK constraint to validate a JSON column

Some days you may find yourself storing JSON in a database column - now the thing about JSON is that you will have to deal with a Schema somewhere, usually in your application code. Here is a simple approach to check that the JSON has at least the required top-level keys as a simple validation to ensure incorrectly formatted JSON isn't entered in your columns. You can go crazy with this and validate against a whole JSON Schema - take a look at pg_jsonschema by the Supabase team

CREATE TABLE IF NOT EXISTS user_app_settings (  username text not null,  settings jsonb);ALTER TABLE user_app_settings ADD CONSTRAINT check_settings_fields      CHECK (settings::jsonb ?& array['cover_photo', 'current_theme', 'toolbar_x', 'toolbar_y', 'version']); -- this insert will work because the JSON is well-formed according to our rules INSERT INTO user_app_settings VALUES ('zikani03', '{"version": "v1.0-beta", "cover_photo": "https://some.site/pic/zikani03", "current_theme":"nord", "toolbar_x": 354.5, "toolbar_y": 100}'::jsonb); -- this insert below should fail since we don't have the "version" field  INSERT INTO user_app_settings VALUES ('john', '{"no_version": "v1.0-beta", "cover_photo": "https://some.site/pic/john", "current_theme":"nord", "toolbar_x": 354.5, "toolbar_y": 100}'::jsonb);

Conclusion

In this article, I shared a few tricks you can use in your projects that use Postgresql, though some of them would also likely work in other DBMSs if you found the equivalent functions. I hope to share more as I keep learning...

Thanks for reading.

This very thing happened to me recently and I wanted to document solutions to the problem, I will tell you which I went with at the end.

Let's create a new Spring Boot project to get a better picture of the issue at hand.

Let's add a Person class that looks like this. Notice that we have thrown in Hibernate Validator annotations to validate the data, like the good backend developers we are.

import com.fasterxml.jackson.annotation.JsonProperty;import jakarta.validation.constraints.Email;import jakarta.validation.constraints.NotEmpty;import org.hibernate.validator.constraints.Length;public class Person {    @NotEmpty    @Length(min = 2, max = 120)    @JsonProperty("full_name")    private String fullName;    @Email    @JsonProperty("email")    private String email;    @NotEmpty    @Length(min = 8, max = 50)    @JsonProperty("phone_number")    private String phoneNumber;    @JsonProperty("country_iso")    private String countryISO;    // getters and setters omitted ...}

And a Spring Boot controller that just validates the data and returns it

// imports omitted for brevity@RestController@RequestMapping("/people")public class PeopleController {    @PostMapping    public ResponseEntity<Person> handlePost(@Validated @RequestBody Person request) {        return ResponseEntity.ok(request);    }}

We want our clients to send us data that looks like this

{  "full_name": "John Banda",  "phone_number": "",  "email": "john.banda@example.com",  "country_iso": "MWI"}

Now the client sends a request with invalid data (let's assume it's missing the full_name and phone_number fields) and the API, like any good API, returns a good ol' 400 Bad Request to let them know the error of their ways. To figure out how they may have screwed that request up, the client checks the response body and discovers the following:

{    "timestamp": "2023-09-27T20:22:29.627+00:00",    "status": 400,    "error": "Bad Request",    "path": "/people"}

From the status code, they can guess it's an error on their end, but the response doesn't show if it's a validation error or something else wrong with the request data. Now the client isn't sure if they sent the request with the rightly named fields in the first place or if it's our API that's gone off the Rails (if there is a pun here, it is intended ;).

Sending validation error response

We would rather give the client a bit more information about why their request is bad. A good way to do that is to catch the validation errors and return a 400 BadRequest with actual details about the validation errors. Let's create a ValidationErrorsResponse class with the following code:

public class ValidationErrorsResponse {    private int code;    private String message;    private Map<String, List<String>> validationErrors;    public ValidationErrorsResponse(int code, String message) {        this.code = code;        this.message = message;        this.validationErrors = new HashMap<>();    }    public void put(String field, String error) {        validationErrors.computeIfAbsent(field, s -> new ArrayList<>()).add(error);    }    // getters and setters omitted ...   }

Now we will need to register an Exception handler for the validation error which we can register against a MethodArgumentNotValidException

public class ExceptionHandlers {    @ResponseStatus(HttpStatus.BAD_REQUEST)    @ExceptionHandler(MethodArgumentNotValidException.class)    public final ResponseEntity<ValidationErrorsResponse> handleExceptions(MethodArgumentNotValidException ex) {        final List<FieldError> fieldErrors = ex.getBindingResult().getFieldErrors();        ValidationErrorsResponse errorResponse = new ValidationErrorsResponse(                HttpStatus.BAD_REQUEST.value(),                HttpStatus.BAD_REQUEST.getReasonPhrase()        );        for (FieldError fieldError : fieldErrors) {            errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());        }        return ResponseEntity.badRequest().body(errorResponse);    }}

It's at this point that if we try to send a POST with an empty or invalid object {} to the /people endpoint, we will get a response similar to the one below, which as you can see shows the fields in camelCase form.

{    "code": 400,    "message": "Bad Request",    "validationErrors": {        "fullName": [            "must not be empty"        ],        "phoneNumber": [            "must not be empty"        ]    }}

While this works and in other situations would be sufficient, it is not great for us because the client will now get confused about the naming of the fields; the validation error response shows the fields named in camelCaseForm whereas the request data is expected to be sent in snake_case_form. The very problem we want to avoid!

We have some possible solutions to fix this issue, explained below.

Solution 1: Rename the class fields to the snake case form

So one quick solution to this problem is the naming of the fields, you can just change it to match the @JsonProperty value. This has the downside that your linter or co-worker may shout at you for naming fields "out-of-standard" but fixes the issue without any over-engineering required. The con is that you have to name all the fields in all your other API objects in this way, so don't forget!

public class Person {    @NotEmpty    @Length(min = 2, max = 120)    @JsonProperty("full_name")    private String full_name;    @Email    @JsonProperty("email")    private String email;    @NotEmpty    @Length(min = 8, max = 50)    @JsonProperty("phone_number")    private String phone_number;    @JsonProperty("country_iso")    private String country_iso;    // getters and setters omitted ...}

Solution 2: Improve the exception handler to maintain standards

Another approach to this is to do a little "over-engineering" to solve the problem in a way that maintains your strongly held opinions and standards while ensuring that the client will never be confused about what's what. This approach involves putting a bit more code in the exception handler to check if the field we are dealing with has a @JsonProperty annotation and then uses the value() from that annotation as the field name when building up validation errors, if we can't find the field, we fall back to sending the regular Bean name of the field/property.

We don't have to make any changes to the names of the fields on the Person or any other class. Here is how we could solve the problem with this approach:-

    @ResponseStatus(HttpStatus.BAD_REQUEST)    @ExceptionHandler(MethodArgumentNotValidException.class)    public final ResponseEntity<ValidationErrorsResponse> handleExceptions(MethodArgumentNotValidException ex) {        final List<FieldError> fieldErrors = ex.getBindingResult().getFieldErrors();        ValidationErrorsResponse errorResponse = new ValidationErrorsResponse(                HttpStatus.BAD_REQUEST.value(),                HttpStatus.BAD_REQUEST.getReasonPhrase()        );        for (FieldError fieldError : fieldErrors) {            try {                var beanClazz = ex.getTarget().getClass();                var field = beanClazz.getDeclaredField(fieldError.getField());                var jsonPropertyAnnotation = field.getAnnotation(JsonProperty.class);                if (jsonPropertyAnnotation != null) {                    errorResponse.put(jsonPropertyAnnotation.value(), fieldError.getDefaultMessage());                } else {                    LoggerFactory.getLogger(getClass()).debug("Failed to get annotated field at {}", fieldError.getField());                    errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());                }            } catch (NoSuchFieldException e) {                LoggerFactory.getLogger(getClass()).error("Failed to get annotated field", e);                errorResponse.put(fieldError.getField(), fieldError.getDefaultMessage());            }        }        return ResponseEntity.badRequest().body(errorResponse);    }

Solution 3: Use a consistent naming convention (i.e. camelCase)

Another option is to update your API data fields + docs to use camelCasedFields as a standard and avoid compromising on Java bean standards and introducing inconsistencies. This is probably not the most practical solution if your API is already used by clients in production unless you manage the change/upgrade to a new version very well. You may also face resistance from API clients who insist on dealing with fields named in snake_case_form.

There is no code for this section because the solution requires communication and coordination with your teams and clients, which you cannot code around. YMMV.

Thanks for reading.

Ohh, I promised to tell you which solution I went with. I went for Solution 1 as a quick fix but after reviewing the other API objects + endpoints in our code, I applied Solution 2 which helped avoid renaming a whole bunch of API object classes. Solution 2 works well with simple flat objects and isn't robust for nested structures. Will write a follow-up soon.

Although I appreciate that it will most likely not be implemented in a lot of places it did give me an excuse to try something sort of new to me; I had never built a standalone React library, so it was a good excuse to finally do that.

Enter react-frostedglass

You may forgive the corny name. react-frostedglass is a React library that enables developers to hide potentially sensitive information when the window is not in focus.

It uses a Layout Effect to apply a blur to components when the window/tab is not in focus. Of course, this does not apply to all the components automatically, rather you as the user of the library have to wrap your components in a FrostedContext provider and use the "frosted" variants of common HTML components.

Example Usage

Let's create a simple react project with Vite to see how the library can be used, first create a Vite project and then install the library using the following commands:

$ npm create vite@latest app -- --template react$ cd app$ npm install https://github.com/zikani03/react-frostedglass.git

Once that is done, replace the App.jsx with the following content:

import { useState } from 'react'import { FrostedContext, useFrostedEffect } from '@zikani03/react-frostedglass'import './App.css'import Account from './Account'function App() {  const focusCheckInterval = 500;  const [blurSize, isFrosted] = useFrostedEffect(focusCheckInterval, '0.3em')  return (    <FrostedContext.Provider value={{ blurSize: blurSize }}>      <Account />    </FrostedContext.Provider>  )}export default App

As you can see above, we use the useFrostedEffect hook and we pass it two parameters, the interval (in milliseconds) to be used to check if the window is in focus and the size of the blur - a valid value for this is anything you can use in the filter: blur() CSS property.

Now create a new component in a file named Account.jsx. Use the code listed below:

import {FrostedDiv, FrostedSpan, FrostedLabel, withFrost} from '@zikani03/react-frostedglass'export default function Account() {    const user = {        email: 'user@example.com',        phone: '+265-xxx-xxx-xxx',        address: 'Address 1, City, Country'    };    return (        <div>            <div className="field">                <strong>Email</strong>                <p>                    <FrostedSpan>{user.email}</FrostedSpan>                </p>            </div>            <div>                <strong>Phone</strong>                <FrostedDiv>{user.phone}</FrostedDiv>                <strong>Address</strong>                <FrostedDiv>{user.address}</FrostedDiv>            </div>            <section {...withFrost({ blurSize: '0.3em' })}>                Some section that's frosted by default...            </section>        </div>    )}

As you can see above, the Account component uses FrostedDiv, FrostedSpan for the content that we want to blur when the window is out of focus

Let's see how Frosted components look

We can now run the development server to see how all this looks in action

$ npm run dev

When we focus on the page we will see everything displaying as normal, except the section element we set to be blurred by default:

Now try opening or focusing on another window and you will observe that the elements become "frosted". How cool is that? 🥶

Using the library

As of the time of writing, I have not published a package to NPM. So if you want to use the library you can install the module directly from GitHub using npm

npm install --save https://github.com/zikani03/react-frostedglass.git

Observations and lessons learned

As mentioned previously, one of my objectives was to learn or at least figure out how to go about creating a standalone React library from scratch. I am glad I was able to accomplish that and here are a few observations:

• Coming up with a nice API/library surface is not always easy, I enjoy creating developer tools and although I have written a few libraries in various languages I was reminded that picking the right abstractions, and naming things (functions, class etc...) requires lots of thought.

• I came to appreciate the use of Context Providers and Layout Effect hooks better

• I was able to configure Vite for building the library "from scratch", that is without depending on a template repository or some kind of boilerplate.

• Related to the point above, I was able to customize the TypeScript compilation options - of course, I had done this before for another JavaScript library I built (you should check that out too, by the way!)

Conclusion

In this article, we have seen how you can use react-frostedglass, a React library I created, to blur components when the screen loses focus. Although this might not be the most practical approach to the problem, it did allow me to figure out how to publish a standalone React library and appreciate the process of building one.

Thank you for reading.

However, for a glutton - on personal projects and when tinkering, this is not enough. They want to be able to know for sure what the differences are and what attributes make two or more solutions different. Sometimes, they may publish these comparisons - this can be seen as a benefit of such gluttony, a public service - saving some other soul some time at the expense of your own.

Note: Maybe gluttony is not the right word choice but it's all I could think of.

Unfortunately for the glutton, software is growing at a rapid rate, and developers keep coming up with new ideas, spin-offs and takes of their own on problems or tools that have been done before. So many damn GitHub repositories. If you don't limit yourself on what you consume you may find yourself feeling anxiety as a result of FOMO (Fear Of Missing Out) which can affect you negatively in the long run. Maybe this is why the old timers and sages retreat to their small set of tools they know very well and ignore the rest of the world only to take a peek now and then until something catches their attention enough to bring them out of the cocoon of "what I have works well enough". I don't know.

Is this what they mean when they say software will eat the world? /s

func WithToUpperCaseCallback(db *gorm.DB, table string, fields ...string) {    cb := UpperCaseCallback(fields...)    db.Callback().Create().Before("*").Register("create:uppercase:" + table, cb)     db.Callback().Update().Before("*").Register("update:uppercase:" + table, cb)}func UpperCaseCallback(fields ...string) func(db *gorm.DB) {    return func(db *gorm.DB) {        if db.Statement.Schema == nil {            return        }        // Only process statements that operate on a table        if db.Statement.Schema.Table == "" {            return        }        for _, argField := range fields {            arr := strings.SplitN(argField, ".", 2)            tableName, desiredFieldName := arr[0], arr[1]            if tableName != db.Statement.Schema.Table {                continue            }            field := db.Statement.Schema.LookUpField(desiredFieldName)            if field == nil {                continue            }            if db.Statement.ReflectValue.Kind() != reflect.String {                continue            }            if fieldValue, isZero := field.ValueOf(db.Statement.Context, db.Statement.ReflectValue); !isZero {                if actualStr, ok := fieldValue.(string); ok {                    db.AddError(field.Set(db.Statement.Context, db.Statement.ReflectValue, strings.ToUpper(actualStr)))                }            }        }    }}

You can register the call back as shown below - this will enable gorm to convert your fields to upper case (in this case first_name and last_name on the "people" table) whenever the entity is saved or updated by gorm, automatically.

type Person struct {    ID        int64  `gorm:"`    FirstName string `gorm:""`    LastName  string `gorm:""`}func main() {    db, err := gorm.Open(postgres.Open("..."), &gorm.Config{})    if err != nil {        panic("failed to connect database")    }    db.Callback().Create().Before("*").Register("create:uppercase", UpperCaseCallback("people.first_name", "people.last_name"))    db.Callback().Update().Before("*").Register("update:uppercase", UpperCaseCallback("people.first_name", "people.last_name"))    // or using utility function    WithToUpperCaseCallback(db, "people", "first_name", "last_name")    // the code below would result in first_name and last_name being capitalized to "JOHN" and "BANDA", respectively before being saved    person := &Person{FirstName: "john", LastName: "banda"}    tx = s.db.Save(person)    if tx.Error != nil {        panic("failed to save")    }}

Conclusion

As you can see, it is pretty straightforward to augment GORM with callbacks to achieve the behavior we wanted. However, use this judiciously, as it adds a layer of "magic" to your codebase - one day you may be left scratching your head as to why some fields are appearing as uppercase when you didn't input them as such.

Funny enough, I have written various similar versions of this functionality in different languages/frameworks, e.g. one for JDBI/Java is [here](https://github.com/nndi-oss/jdbi-utils/blob/main/jdbi3-utils/src/main/java/cloud/nndi/oss/jdbi/customizers/CapitalizeCustomizer.java)

I hope you found this useful.

It relies on NodeJS, which is typically okay for my project since am most likely using something like Vue or React with Node/npm build system anyways. We only need one extra devDependency: ftpdeploy

$ npm install --save-dev ftp-deploy

Once we have that installed, we can now add the config file for it in ftpdeploy.js which can be setup like below, obviously you would change according to your site details.

import FtpDeploy from "ftp-deploy";import { cwd } from 'node:process';import { join } from 'node:path';const ftpDeploy = new FtpDeploy();const config = {    user: "mycpaneluser",    // Password optional, prompted if none given    password: null,    host: "ftp.example.com",    port: 21,    localRoot: join(cwd(), "./dist"),    remoteRoot: "/someapp.example.com/",    // include: ["*", "**/*"],      // this would upload everything except dot files    include: ["*", "dist/*", ".*"],    // e.g. exclude sourcemaps, and ALL files in node_modules (including dot files)    exclude: [        // "dist/**/*.map",        "node_modules/**",        "node_modules/**/.*",        ".git/**"    ],    // delete ALL existing files at destination before uploading, if true    deleteRemote: false,    // Passive mode is forced (EPSV command is not sent)    forcePasv: true,    // use sftp or ftp    sftp: false,};ftpDeploy    .deploy(config)    .then((res) => console.log("finished:", res))    .catch((err) => console.log(err));

This script can then be run using NodeJS via node ftpdeploy.jsOr you can add that to your package.json under scripts, something like:

  "scripts": {    ...    "deploy": "npm run build && node ftpdeploy.js"  },

Which makes deploying or pushing updates as simple as

$ npm run deploy

That's it, I guess ideally, one should have a nice CI/CD process and better tooling for deploying real-world projects but this setup serves me well for some small toy projects, especially on utilizing my CPanel hosting plan.

Notes

• I recommend trying GitHub actions for a CI/CD process that can deploy sites when you push a tag/release.

• My friend, Walter, reminded me that we had agreed to atleast publish one post - this post probably wasn't going to get published if it wasn't for his reminder. Checkout his blog https://ntumbuka.me/

Enter promptato a simple command-line tool that periodically prompts you without cluttering your to-do list app.

./promptato -every 15m -f prompts-1.txt -reload

How it works

The program prompts you to think about something every interval and sends a Desktop Notification with the prompt as the text. The prompt items are chosen at random. The source of the prompts is a text file, where each prompt or question is on a separate line. You can also specify how often you want the prompts, the default is every 42 minutes.

For example, my prompts text file looks like this:

And I get desktop notifications every few minutes like this:

After running it like so

./promptato -every 15m -f prompts-1.txt -reload

Pro-tip: If you are feeling adventurous, you can trigger your own anxiety by having a prompt file with just one line "Where are you with that task?" and run it every 5 minutes - which could simulate the behavior of some micro-managing managers.

Building and Using

To use this you will need to have Go installed and use the following steps to create your build of the program. I may create a GitHub repository and pre-built some binaries for it if people are interested, but the following instructions can help you get it up and running for yourself.

Step 1: Create a go project and create an empty file named main.go

$ mkdir promptato$ cd promptato$ go mod init promptato$ touch main.go

Step 2: Copy the full code below into main.go

Step 3: Get dependencies and compile, if you are on windows this will create a promptato.exe and on other OS it will create a binary (in this case one named promptato)

$ go mod tidy$ go build

The full code

The full program is below, please note it's a very simple no frills, offline program - it could be made better (see below for ideas), below is the Minimum Useable Thing:

package mainimport (    "flag"    "io/ioutil"    "log"    "math/rand"    "strings"    "time"    "github.com/gen2brain/beeep"    "github.com/go-co-op/gocron"    "github.com/mroth/weightedrand")var every stringvar promptFile stringvar reloadFile boolfunc init() {    flag.StringVar(&every, "every", "42m", "Every specifies the duration and supports formats like 5s, 42m, 1h ")    flag.StringVar(&promptFile, "f", "./prompts.txt", "Which file to load prompts from")    flag.BoolVar(&reloadFile, "reload", false, "Whether to reload prompts file on each run...")}func main() {    rand.Seed(time.Now().UTC().UnixNano()) // always seed random!    flag.Parse()    preloadedChoices := loadChoices(promptFile)    s := gocron.NewScheduler(time.UTC)    s.Every(every).Do(func() {        var chooser *weightedrand.Chooser        if reloadFile {            choices := loadChoices(promptFile)            chooser, _ = weightedrand.NewChooser(choices...)        } else {            chooser, _ = weightedrand.NewChooser(preloadedChoices...)        }        result := chooser.Pick()        err := beeep.Notify("Promptato", result.(string), "question.png")        if err != nil {            log.Fatalf("failed to notify user, got %v", err)        }    })    // starts the scheduler and blocks current execution path    s.StartBlocking()}func loadChoices(sourceFile string) []weightedrand.Choice {    data, err := ioutil.ReadFile(sourceFile)    if err != nil {        log.Fatalf("failed to load choices file, got %v", err)        return nil    }    lines := strings.Split(string(data), "\n")    choices := make([]weightedrand.Choice, len(lines))    for _, line := range lines {        trimmed := strings.TrimSpace(strings.TrimPrefix(line, "- "))        choices = append(choices, weightedrand.NewChoice(trimmed, 1))    }    return choices}

Ideas for improvement

This code represents a small useable thing but could be improved further with:

• making it a background service

• having a way to weight the prompts (already using weightedrand library, just needs a way to define weights)

• Having some kind of jitter to randomize the schedule of the prompts

• Allowing users to interact with the prompts to indicate if they don't want it to come up again (this would necessitate re-implementing this with maybe a GUI toolkit)

Anyways, I may or may not implement these ideas. For now, I just needed this for reminding me of some things as the year comes to an end as I meditate on the next year.

Conclusion

In this post, we have seen how a simple command-line tool built with Go can help you remember to think of things without having to add that as a task to your to-do app. I hope you found this at least mildly interesting.

What the heck is this dblstd ?

dblstd is a command-line tool that aims to helps developers/teams check their project structure to verify that some required files/directories are included in a project. The goal is to enforce standard project structures for teams.

The dblstd-action

The dblstd action basically uses dblstd to check the structure of the repository and can be configured to run on every pull request or tag. It's a simple action that literally doesn't have any code. It just clones the dblstd binary and runs it on the repo with option for which shape file to use to check the repo.

You can checkout the action here: https://github.com/zikani03/dblstd-action

Background

I work on/off on a Java toy project, articulated, which scrapes articles from a few Malawian news sites. I use it mostly to play with different technologies, ideas and to try to keep up with the local news. One of the features of that project is performing basic Named Entity Recognition which I decided to build as a separate service in Go after facing issues with Java libraries for similar functionality, but that's a story for another day.

Building an AWS Lambda Function for NER

I decided to re-implement that service as a Lambda function because it seemed like the kind of thing that would make sense to have a lambda function for. As I was going about that, it struck me that the service could be made more useful if it was generic enough to be used with any website.

The result is a project dubbed neria-lambda which can be deployed on AWSs Lambda platform. You send a request with a URL to a website and a jQuery-like selector which specifies which DOM elements (i.e. part of the page) to extract the text from. After the text is fetched, the service uses prose to extract Named Entities from the text, which basically means it detects names of people or places from the text. (Side note: turns out the prose library was archived by the maintainer at some point)

Here is an example in Insomnia

Deploying the Lambda Function

So as much as I love having things in production, I don't like bleeding money so I won't share the Function URLs public URL - this section will explain how you can deploy your own instance of the lambda on AWS.

Step 1: Clone the repository, compile the go program and create a zip file named main.zip. The instructions are below:

$ git clone https://github.com/zikani03/neria$ cd neria/neria-lambda$ GOOS=linux GOARCH=amd64 go build -o main main.go$ zip main.zip main

NOTE: Windows users should use the build.ps1 powershell script for the process

Step 2: Login to the AWS console and create a Lambda function

Step 3: Once the Lambda Function is created, upload the main.zip from step 1.

Step 4: Create a Function URL - which will enable you to access the lambda via a public URL (of course you can configure it to be accessible to certain IAM roles, etc.. - I'm assuming you know your way around this AWS stuff)

Step 5: Once you create the Function URL you can test the lambda with cURL, Postman or Insomnia with the following example request:

{    "Url": "https://www.nyasatimes.com/chilima-says-malawi-is-a-best-investment-place-in-sadc-region-and-beyond/",    "Selector": "#content div.nyasa-content",    "Text": ""}

Experience of deploying a Lambda Function URL

I found the experience of deploying a Lambda Function URL an interesting one, it was easy to follow the AWS documentation and was especially useful to reference their samples from the GitHub repos.

What I found really interesting is how deployment of the function is done via uploading a zip (with an option to get that from S3). I expected a Heroku-like experience for the process but was reminded that sometimes simple approaches go a long way.

Conclusion

In this article I described my experience building a useful Lambda function which you can also deploy for your own use. The project could improve in how it does named entity recognition but since I'm using a library that's since been archived I can't really say much on when and how. It was an interesting experience and one that's given me a couple of ideas (one for a PaaS platform).

Thanks for reading. Feel free to share feedback on Twitter - @zikani03

MVC is dead, long live MVC!

In projects where the frontend is completely handled by a Single Page Application initial data may be fetched using an API call on the components "onMount" event. However, there are still a lot of (MVC) projects which just sprinkle in JavaScript in a Server Side Rendered page for extra functionality like client-side validation.

Please note by server-side rendered here I mean that the page is being rendered by say, PHP (regular PHP, Twig, Blade template etc..), Rails templates, Java with JSP, Thymeleaf, Velocity etc.. , .NET with ASP.NET pages, Razor templates, Django templates or whatever language/framework you fancy that allows you to render to HTML via some template engine.

In these kinds of projects it is not uncommon to find code that mixes in the template rendering logic with the JavaScript code on the page in a way that results in hard to read or maintain code.

NOTE: I am using PHP in these examples but I have seen this pattern in Spring Boot projects, and the solution can be applied to any framework that offers server side rendering of templates

Below is an example of code you may find in some projects when passing data from server-side template to JavaScript. (Raise your hand if you have done something like this before, I see you).

<script type="text/javascript">    const individuals = [        <?php foreach($people as $p): ?>        { "firstName": <?php echo '"' . $p->firstName . '"' ?>, "lastName": <?= '"' . $p->lastName . '"' ?> },        <?php endforeach ?>    ];    const selectOptions = {        <?php echo '"' . ORGANIZER_CONSTANT . '"'?> : "Organizer",        <?php echo '"' . SPEAKER_CONSTANT . '"'?> : "Speaker",        <?php echo '"' . ATTENDEE_CONSTANT . '"'?> : "Attendee",        <?php echo '"' . SPONSOR_CONSTANT . '"'?> : "Sponsor"    }</script>

Which would render to something like:

<script>    const individuals = [        { "firstName": "John", "lastName": "Banda" },        { "firstName": "Mary", "lastName": "Banda" },        { "firstName": "Joseph", "lastName": "Gondwe" },        { "firstName": "Jane", "lastName": "Phiri" },    ];    const selectOptions = {        "organizer": "Organizer",        "speaker": "Speaker",        "attendee": "Attendee",        "sponsor": "Sponsor"    }</script>

This example is trivial, but I have seen situations where lots of data was serialized this way. As you can see this approach of serializing data is messy and can indeed make code hard to follow and also lead to subtle bugs:

A better way to pass data to JavaScript

We can do better by using an approach that I think is better, cleaner, separates the concerns between server-rendered template and JavaScript as well as opening up possibilities for improving testing the JavaScript code.

The approach basically works as follows:

1. Put all the data you need on the JavaScript side into a "root" object(s) within the server-side template (note: you can use an array in PHP, a Map in Java/C# or similar languages, hashmap/dictionary in Python etc..)
2. Encode the data as JSON using the server-side template capabilities (using json_encode in PHP, objectMapper#writeAsString or gson#toJson in a Java / Spring Boot codebase, json.dumps or to_json in Python etc..)
3. Use JSON.parse to decode the JSON into a global object under the toplevel window, typically named the same as the root object from step 1

See the example below...

<!-- Step 1--><?php     $pageData = [        "individuals" => $people,        "selectOptions" => [            ORGANIZER_CONSTANT => "Organizer",            SPEAKER_CONSTANT => "Speaker",            ATTENDEE_CONSTANT => "Attendee",            SPONSOR_CONSTANT => "Sponsor"        ],    ];?><!-- Step 2 and Step 3 --><script type="text/javascript" id="pageDataScript">window.__pageData = JSON.parse(<?php echo json_encode($pageData) ?>);const individuals = window.__pageData.individuals;const selectOptions = window.__pageData.selectOptions;</script>

It seems like an obvious approach in hindsight.

If you are using a template engine like Twig you can use the raw function to avoid the JSON data from being rendered using HTML escapes.

<script type="text/javascript" id="pageDataScript">window.__pageData = JSON.parse( {{ json_encode(pageData) | raw }} );</script>

Benefits of this approach.

1. Cleaner - better separation of concerns as the server side code doesn't 'bleed' into the JavaScript code too much.
2. Easier to reason about the structure of the data passed to the JavaScript handling code
3. Makes it easy to introduce a library like VueJS or React as you could just fetch the data from the __pageData object "onMount"

Extension of the approach

An extension to the above approach enables you to render/write the data separately in a script element and then use another script element to read the data. You will note in the example below that the first <script> element has type application/json - this allows us to place data in there that the Browser safely ignores, and will not execute as JavaScript.

This may seem like overkill but could provide you with flexibility for testing if you have issues with getting the data from the backend since you can easily place random JSON that follows the proper structure in the initialPageData element.

<script type="application/json" id="initialPageData"><?php echo json_encode($pageData) </script><script type="text/javascript" id="pageDataScript">window.__pageData = JSON.parse(document.getElementById("initialPageData").innerHTML);</script>

Conclusion

In this article we have seen that with a few changes it's possible for us to make it cleaner to pass data from server-side rendered template to JavaScript to enhance the experience on pages of your web applications.

Thanks for reading. If you have questions, comments or suggestions, reach out on Twitter @zikani03.

This article will go over how to embed a basic PDF viewer in a Wails application starting from the Wails' React template. It is not meant to be a comprehensive tutorial so I won't go into styling the PDF reader or anything.

I have put the complete code in a repository here

NOTE: this article is using Wails v2.0.0-beta.35

Setup Wails Project

$ wails init -n pdf-in-wails -t react

Install React and React-PDF

$ cd pdf-in-wails$ cd frontend$ npm install --save react react-pdf fast-base64$ cd ..

Read a Document from the File System with Go

We are going to use the Go backend to read the PDF from the file system. In order to do so we will use the Wails' runtime package which provides a function to Open a file dialog and then use the standard ioutil.ReadAll function to read the file into a byte slice.

In the app.go file in the project add this content

package mainimport (    "context"    "fmt"    "io/ioutil"    "github.com/wailsapp/wails/v2/pkg/runtime")// App structtype App struct {    ctx context.Context}// NewApp creates a new App application structfunc NewApp() *App {    return &App{}}// Greet returns a greeting for the given namefunc (a *App) Greet(name string) string {    return fmt.Sprintf("Hello %s, It's show time!", name)}// startup is called at application startupfunc (a *App) startup(ctx context.Context) {    // Perform your setup here    a.ctx = ctx}func (a *App) OpenAndGetPDFData() ([]byte, error) {    filters := []runtime.FileFilter{        {DisplayName: "PDF Documents (*.pdf)", Pattern: "*.pdf"},    }    filePath, err := runtime.OpenFileDialog(a.ctx, runtime.OpenDialogOptions{        Filters:         filters,        ShowHiddenFiles: false,    })    if err != nil {        return nil, err    }    return ioutil.ReadFile(filePath)}

Register the OnStartup function in main.go

 // ... other lines ignoredfunc main() {    // Create an instance of the app structure    app := NewApp()    // Create application with options    err := wails.Run(&options.App{        Title:     "pdf-in-wails",        Width:     1024,        Height:    768,        Assets:    assets,        OnStartup: app.startup,        Bind: []interface{}{            app,        },    })    if err != nil {        println("Error:", err)    }}

Adding a button to trigger the action with React

Since Wails User Interfaces are driven by JavaScript, we need to add some way to triggger the function we just created.

You will notice the import for OpenAndGetPDFData - this function binding is generated automatically by Wails when you run wails build or wails dev or you can run it directly with wails generate module.

Replace the App.jsx with the following.

import './App.css';import {useState} from 'react';import {Document, Page, pdfjs} from 'react-pdf';import {toBytes} from 'fast-base64';import {OpenAndGetPDFData} from "../wailsjs/go/main/App";function App() {    pdfjs.GlobalWorkerOptions.workerSrc = `//cdnjs.cloudflare.com/ajax/libs/pdf.js/${pdfjs.version}/pdf.worker.min.js`;    const [pdfDocument, setPdfDocument] = useState(null)    const [numPages, setNumPages] = useState(null);    const [pageNumber, setPageNumber] = useState(1);    function onDocumentLoadSuccess({ numPages }) {      setNumPages(numPages);    }    async function openPDF() {        try {            const pdfFile = await OpenAndGetPDFData()            const pdfBytes = await toBytes(pdfFile)            setPdfDocument({data: pdfBytes })        } catch(e) {            console.log("Failed to open pdf", e)        }    }    return (        <div id="App">            <button className="btn" onClick={openPDF}>Open PDF</button>            <Document file={pdfDocument}                     onLoadSuccess={onDocumentLoadSuccess}                     onLoadError={(error) => alert('Error while loading document! ' + error.message)}                    onSourceError={(error) => alert('Error while retrieving document source! ' + error.message)}>                <Page pageNumber={pageNumber} />            </Document>            <p>                Page {pageNumber} of {numPages}            </p>        </div>    )}export default App

Wails sends []byte as a Base64 encoded string

One thing you will note is that we are using the toBytes function from the fast-base64 library to get the data of the PDF file on the JavaScript side. This is because Wails serializes the byte slice as a Base64 encoded string when sending the data from the Go side. Other data structures are serialized as JavaScript objects appropriately.

Run it

You can then run the application using Wails dev which will startup a webserver with hot reloading and open a New Window with a running instance of the application.

In order to build the application as a binary you need only to run wails build

$ wails dev

Here are screenshots from my build of the same

With a PDF opened

Conclusion

Wails is an interesting technology for building desktop applications which is worth a look if you want to keep using web technologies to build desktop applications and as an alternative to something like Electron. We have seen how we can embed a PDF viewer in Wails easily using React PDF library (though you can do it without React).

Thanks for reading.

UPDATE: So it seems I missed the memo and didn't know that the Fiber team already had the functionality in v2. So please see this page for more on THE RIGHT WAY

Serving files from embed.FS in Fiber not straightfoward

One of the problems that I ran into is serving static assets from an embedded file system or embed.FS. While Fiber does allow for serving static assets it only allows you to pass the path to a directory which is read from the "real" FileSystem.

This may not be ideal as deploying the program now requires both the binary and the files to be placed/uploaded on a server (NOTE: we are assuming we aren't deploying using containers which could solve this problem easily).

What is embed.FS anyways?

The embed package, a feature of Go since sometime in Go 1.16, enables us to pack files in the binary - this is useful as we can just ship the binary and not worry about forgetting to copy & upload the right files. For more information on //go:embed see this blog post on the JetBrains GoLand blog.

So we want to be able to serve files that are embedded in the Go program itself. How can we do that?

Using rebed to work with embedded file system alongside Fiber

Enter rebed; a library that enables us to read and write the contents of the embed.FS. We can use rebed to write the embedded files to a directory and let Fiber serve them straight from that directory. This gives us the benefit of embedding while also enabling us to have the files on the file system where they can be served via an another web-server like NGINX.

Here is how we could go about that:

package mainimport (    "embed"    "github.com/gofiber/fiber/v2"    "github.com/soypat/rebed")// Let's assume you an app created by the likes of create-react-app// which leaves it's built files in frontend/dist//go:embed frontend/distvar assets embed.FSfunc main() {    app := fiber.New()    // I prefer this form with www parent directory    // as it avoids overwriting the existing frontend/dist directory    // if you happen to run the program in the same working directory    rebed.Write(assets, "www")    app.Static("/", "www/frontend/dist")    err := app.Listen(":3000")    if err != nil {        panic(err)    }}

I feel the only icky part of this is that in your app.Static call you have to pass the full path of the embedded file system, e.g. note www/frontend/dist in the example, as well as the fact that you may have to clean up at some point if you don't want that directory to be on the server during re-deploy or something.

Can we have this another way?

~The other option I considered is to send a PR to the Fiber project to modify how the Static function handles files but I don't think it will get accepted as that may also require changing how fasthttp (the http library beneath Fiber) serves Static files.~

~So for now, you can either copy the files you want to serve manually or embed them and let rebed recreate them for you when the program is running or find another clever way maybe using Fiber's adaptor package somehow.~

UPDATE: Use the right way

Thanks for reading.

The Fiber project has support for a lot of template engines but they don't have one for Scriggo. I like Scriggo because it looks a bit like Jinja (Python), Twig (PHP) and Pebble (Java) template engines which I have used extensively recently.

Fortunately, Fiber allows you to implement your own Template Engine without much hassle. In this article I will share a simple implementation of a simple Scriggo template engine that you can use with Fiber.

A Simple Server side rendered site in Go

We are going to create a simple server side rendered site with Go using the Scriggo engine. So first of all create a new project, e.g.:

$ mkdir example-site$ cd example-site$ go mod init example.com/site

Copy the code for the engine (find it at the bottom of this article) into a file named scriggo_engine/scriggo_engine.go so you can import the Engine.

Now, place the following server code in a file named main.go in your module directory:

package mainimport (    "log"    "github.com/gofiber/fiber/v2"    "example.com/site/scriggo_engine")type Server struct {    app *fiber.App}func (s *Server) NotImplemented(ctx *fiber.Ctx) error {    return nil}func NewServer() *Server {    engine := scriggo_engine.New("templates", ".html")    s := &Server{        app: fiber.New(fiber.Config{            Views: engine,        }),    }    s.app.Get("/", s.indexPage)    return s}type Contact struct {    Name   string    Phone  string    Email    string}func (s *Server) indexPage(ctx *fiber.Ctx) error {    contacts := []Contact{        {Name: "John Phiri", Email: "john@example.com", Phone: "(+265) 999 123 456"},        {Name: "Mary Phiri", Email: "mary@example.com", Phone: "(+265) 999 123 456"},        {Name: "Jane Phiri", Email: "jane@example.com", Phone: "(+265) 999 123 456"},    }    return ctx.Render("index", fiber.Map{        "contacts": &contacts,    })}func (s *Server) Start(bind string) error {    return s.app.Listen(bind)}func main() {    server := NewServer()    err := server.Start("localhost:3000")    if err != nil {        log.Fatalf("Failed to run, got error %v", err)    }}

In a file named templates/base.html

<!DOCTYPE html><html lang="en"><head>    <meta charset="UTF-8">    <meta http-equiv="X-UA-Compatible" content="IE=edge">    <meta name="viewport" content="width=device-width, initial-scale=1.0">    <title>{{ Title() }} - My Website</title></head><body>    <nav>        <h1>My Website</h1>        <ul>            <li><a href="/home">Home</a></li>            <li><a href="/about">About</a></li>            <li><a href="/contact">Contact</a></li>        </ul>    </nav>    {{ Content() }}    {{ Footer() }}</body></html>

In a file named templates/index.html

{% extends "base.html" %}{% macro Title %}Home{% end %}{% macro Content %}<section id="app-content">    <div class="contacts">        {% for contact in contacts %}        <li><a href="/contact/{{ contact.Name }}">{{ contact.Phone}}</a> ({{ contact.Email }})</li>        {% end %}    </div></section>{% end %}{% macro Footer %} <footer>(c) My Website</footer>{% end %}

Project Structure

Your directory should look something like this

example-sitemain.gogo.modgo.sumscriggo_engine|    scriggo_engine.gotemplates    base.html    index.html

Running the code

First we are going to make sure go modules are tidied and downloaded via go mod tidy and then we can run the main.go

$ go mod tidy$ go run main.go

Now go to http://localhost:3000 you should see a server-side rendered HTML page!

Registering Scriggo in-built functions

Typically in template engines we may want to call some functions on our data. Scriggo comes with a lot of built-in functions and using them is relatively straight-foward.

Register the function via addFunc

import "github.com/open2b/scriggo/builtin"// ..engine.AddFunc("base64", builtin.Base64 )// ..

Use the function in your templates

<div>{{ base64("Hello") }}</div>

Scriggo Template Engine for Fiber

Here is the implementation of the engine code, it has not been battle-tested so if there are bugs or ways to improve it, feel free to reach out on Twitter.

package scriggo_engineimport (    "fmt"    "io"    "net/http"    "os"    "path/filepath"    "strings"    "sync"    "github.com/gofiber/fiber/v2"    "github.com/gofiber/template/utils"    "github.com/open2b/scriggo"    "github.com/open2b/scriggo/native")// Engine structtype Engine struct {    // views folder    directory string    // http.FileSystem supports embedded files    fileSystem http.FileSystem    // views extension    extension string    // layout variable name that incapsulates the template    layout string    // determines if the engine parsed all templates    loaded bool    // reload on each render    reload bool    // debug prints the parsed templates    debug bool    // lock for funcmap and templates    mutex sync.RWMutex    // template funcmap    funcmap native.Declarations    // templates    templates map[string]string    // scriggo filesystem    fsys scriggo.Files}// New returns a Scriggo render engine for Fiberfunc New(directory, extension string) *Engine {    engine := &Engine{        directory: directory,        extension: extension,        layout:    "embed",        funcmap:   make(native.Declarations),    }    engine.AddFunc(engine.layout, func() error {        return fmt.Errorf("layout called unexpectedly.")    })    return engine}func NewFileSystem(fs http.FileSystem, extension string) *Engine {    engine := &Engine{        directory:  "/",        fileSystem: fs,        extension:  extension,        layout:     "embed",        funcmap:    make(native.Declarations),    }    engine.AddFunc(engine.layout, func() error {        return fmt.Errorf("layout called unexpectedly.")    })    return engine}// Layout defines the variable name that will incapsulate the templatefunc (e *Engine) Layout(key string) *Engine {    e.layout = key    return e}// Delims sets the action delimiters to the specified strings, to be used in// templates. An empty delimiter stands for the// corresponding default: {{ or }}.func (e *Engine) Delims(left, right string) *Engine {    fmt.Println("delims: this method is not supported for scriggo")    return e}// AddFunc adds the function to the template's function map.// It is legal to overwrite elements of the default actionsfunc (e *Engine) AddFunc(name string, fn native.Declaration) *Engine {    e.mutex.Lock()    e.funcmap[name] = fn    e.mutex.Unlock()    return e}// Reload if set to true the templates are reloading on each render,// use it when you're in development and you don't want to restart// the application when you edit a template file.func (e *Engine) Reload(enabled bool) *Engine {    e.reload = enabled    return e}// Debug will print the parsed templates when Load is triggered.func (e *Engine) Debug(enabled bool) *Engine {    e.debug = enabled    return e}// Load parses the templates to the engine.func (e *Engine) Load() error {    // race safe    e.mutex.Lock()    defer e.mutex.Unlock()    e.templates = make(map[string]string)    e.fsys = scriggo.Files{}    // Loop trough each directory and register template files    walkFn := func(path string, info os.FileInfo, err error) error {        // Return error if exist        if err != nil {            return err        }        // Skip file if it's a directory or has no file info        if info == nil || info.IsDir() {            return nil        }        // Skip file if it does not equal the given template extension        if len(e.extension) >= len(path) || path[len(path)-len(e.extension):] != e.extension {            return nil        }        // Get the relative file path        // ./views/html/index.tmpl -> index.tmpl        rel, err := filepath.Rel(e.directory, path)        if err != nil {            return err        }        // Reverse slashes '\' -> '/' and        // partials\footer.tmpl -> partials/footer.tmpl        name := filepath.ToSlash(rel)        // Remove ext from name 'index.tmpl' -> 'index'        name = strings.TrimSuffix(name, e.extension)        // name = strings.Replace(name, e.extension, "", -1)        // Read the file        // #gosec G304        buf, err := utils.ReadFile(path, e.fileSystem)        if err != nil {            return err        }        e.fsys[filepath.ToSlash(rel)] = buf        e.templates[name] = filepath.ToSlash(rel)        // Debugging        if e.debug {            fmt.Printf("views: parsed template: %s\n", name)        }        return err    }    // notify engine that we parsed all templates    e.loaded = true    if e.fileSystem != nil {        return utils.Walk(e.fileSystem, e.directory, walkFn)    }    return filepath.Walk(e.directory, walkFn)}// Render will execute the template name along with the given values.func (e *Engine) Render(out io.Writer, template string, binding interface{}, layout ...string) error {    if !e.loaded || e.reload {        if e.reload {            e.loaded = false        }        if err := e.Load(); err != nil {            return err        }    }    templatePath, ok := e.templates[template]    if !ok {        return fmt.Errorf("render: template %s does not exist", template)    }    opts := &scriggo.BuildOptions{        Globals: e.funcmap,    }    // Register the variables as Globals for Scriggo's type checking to work    for key, value := range binding.(fiber.Map) {        opts.Globals[key] = value    }    // Build the template.    tmpl, err := scriggo.BuildTemplate(e.fsys, templatePath, opts)    if err != nil {        return err    }    return tmpl.Run(out, nil, nil)}

What's next?

1. Support for embedded filesytems (embed.FS) - I suspect it's already possible but the API doesn't look good for doing so yet.

2. Releasing to the public - I am thinking of either releasing this as a module that others can consume or sending a Pull Request to the Fiber team if they would be interested, or I may just make it a Gist. We will see. Right now the proof of concept works well enough for me to play with on toy projects.

I may update this article later on

The idea behind the tool is to allow you to do this:

$ in build cmake ..

Instead of

$ mkdir -p build$ cd build$ cmake ..

Adding a feature to in

When I found this tool I knew I had to add another feature to it to accomplish something I have needed for a while; a command-line tool to run the same command in multiple project directories at once.

I sent in a Pull-Request to add glob support to enable in to be used to run a command in multiple directories. I was pleased to see the PR merged 🎉 So this functionality is available to anyone else that comes across in.

So, let's see what we can do with this new feature:

Compile all maven projects

$ in "./**/*pom.xml" mvn clean compile

View last N commits for all projects in a directory

This works if the directory has subdirectories that are Git repositories.

$ in "./*" git log --oneline -n 3

Pull refs for all git repos in a directory

Like the previous example, this works if the directory has subdirectories that are Git repositories.

$ in "./*" git pull --all

Run go fmt on all directories containing Go files

$ in "./**/*.go" go fmt

Creating GitHub actions workflow directories in multiple directories

With Powershell

$ in ".\oss\*" powershell "mkdir .github/workflow"

With Bash

$ in "./oss/*" mkdir -p .github/workflow

Installing and using in

In order to use in you will currently need to build it locally. Fortunately, the command does not have external dependencies, at the time of writing, so all you will need is the Go toolchain. Download Go here

$ git clone https://github.com/xyproto/in$ cd in$ go build

Personally, I build the binary as indir as I think that makes more sense despite being more characters and also works much better with Powershell :). If you want to use that similar approach you can build it like so:

$ go build -o indir main.go

Or like this on Windows

$ go build -o indir.exe main.go

When you build it like that you can place the command on your $PATH and call the command as follows:

$ indir "./**/*pom.xml" mvn clean compile

Conclusion

in is a great tool which has saved me the trouble of writing multiple single-use case commands. Open-Source software is awesome and contributing to something that I needed and use feels good. Moving on, I would like to improve the documentation to make sure people understand the behavior of the tool especially with the glob feature and add some notes on using the tool in a Windows environment.

Well the answer to the question in the title is simply: YES, you can clone a Git repository to an in-memory file system. Let us see how.

In this post I will present two ways we can accomplish this; using a temporary filesystem using facilities provided by the OS (Linux in this case) and using a Git library which has capability to clone and work with repositories in-memory (using Go). Maybe we can even combine the approaches?

Clone to a RAM storage (on Linux)

The first approach involves using the OS and relevant programs to create a temporary filesystem that runs in-memory and does not persist across reboot. Linux supports creating file system which keeps files in virtual memory. In order to accomplish our particular task, we create a tmpfs filesystem, mount it to a directory then navigate to that directory and perform the clone operation there.The basic idea would work as follows:

$ sudo mkdir /mnt/tempgit$ sudo mount -t tmpfs -o size=50m tmpfs /mnt/tempgit$ cd /mnt/tempgit$ git clone https://github.com/zikani03/multiclone.git$ lsmulticlone

Just take note that "tmpfs may use SWAP space. If your system runs out of physical RAM, files in your tmpfs partitions may be written to disk based SWAP partitions and will have to be read from disk when the file is next accessed" (source)

Clone using go-git library

This approach requires a fairly recent installation of Go in-order to use the go-git library. The go-git library is

... a highly extensible git implementation library written in pure Go.

The library has a Clone function which we can use with a storage type. Fortunately for us, one of the storage types that comes out of the box in the library is the memory storage. We can use that to perform an in-memory clone easily:

package mainimport (    git "github.com/go-git/go-git/v5"    "github.com/go-git/go-git/v5/storage/memory")func main() {    _, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{        URL: "https://github.com/zikani03/multiclone.git",    })    if err != nil {        panic(err)    }    // ...}

Obviously, this approach will be easier if you are comfortable working with Go.

Why would you wanna do this?

🚅. Using an in-memory storage should improve performance of git operations as memory is generally faster than disk. As mentioned earlier, I am working on something that might need this behavior. If you are curious about the tool, I spoke a bit about it in response to another Tweet:

Well, that's all folks.

Prerequisites:

• zefaker (version 0.6 at the time of writing)
• JDK 15+
• Go 1.16+
• esbulk
• Elasticsearch 7+ / OpenSearch 1.0.0
• curl (optional, but really you should have this already)

Creating the zefaker file

Firstly, we need to create a Groovy script to use with the zefaker to specify the form of our random data.You can copy the code snippet below and place it in a file named data.groovy which we will pass to zefaker to generate our data.

// in data.groovyimport com.google.gson.JsonObjectfirstName = column(index= 0, name= "firstName")lastName  = column(index= 1, name= "lastName")age       = column(index= 2, name= "age")accountStatus = column(index=3, name="accountStatus")accountMeta   = column(index=4, name="accountMeta")generateFrom([    (firstName): { faker -> faker.name().firstName() },    (lastName): { faker -> faker.name().lastName() },    (age): { faker -> faker.number().numberBetween(18, 70) },    (accountStatus): { faker -> faker.options().option("Open", "Closed") },    // You can nest objects like this    (accountMeta): { faker ->         def meta = new JsonObject()        meta.addProperty("totalTokens", faker.number().numberBetween(5000, 10000))        meta.addProperty("activityStatus", faker.options().option("Active", "Dormant"))        return meta    }])

Generating the data

zefaker requires Java to be installed to run. I'm assuming you have the java command in your PATH.With that we can run the following to generate 5 million rows of random data exported into a JSON Lines format (basically a plain text file where each line is a JSON Object).

$ java -jar zefaker-all.jar -f data.groovy -jsonl -output elasticdata.jsonl -rows 5000000

You can also try the zefaker web instance I have running here, this will save you from having to install Java or zefaker on your machine. Make sure you select JSON Lines as the export option

Getting esbulk utility

We will use esbulk, a nifty small command-line program written in Go, to perform the indexing. We will have to build it first.

$ git clone https://github.com/miku/esbulk$ cd esbulk$ go build

This will create an executable named esbulk (esbulk.exe on Windows). You can add it on your PATH

Indexing the data in Elasticsearch

Again, it is assumed that you have installed Elasticsearch or OpenSearch and have it running. We can use the following command to index our data:

$ esbulk -index "people-2021.07.07" -optype create -server http://localhost:9200 < elasticdata.jsonl

After esbulk completes (silently) you can check that the operation was successful by visiting http://localhost:9200/people-2021.07.07/_search in your browser or using curl like so:

$ curl -G http://localhost:9200/people-2021.07.07/_search

And, that's all folks. Hope you found this useful.

Show some love and star zefaker :)