Thomas Countz

Job Scheduling with at(1)

2026-02-22T00:00:00+00:00

I recently hacked together an ESP-01 and a cheap solar garden lantern¹ so that I can have warm flickering candlelight whenever the mood strikes. For example, by using at(1), I can make teatime² just a little more magical:

$ echo "curl 'http://ember.local/on'" | at teatime
job 42 at Sun Feb 22 16:00:00 2026

at(1) is a Unix utility for scheduling one-off commands. In the example above, a request to turn on the lantern will be sent at 16:00 local time.

You can verify it’s queued with atq, and view the job details with at -c [job]:

$ atq
42	Sun Feb 22 16:00:00 2026

$ at -c 42
#!/bin/sh
# atrun uid=502 gid=20
# ...
curl 'http://ember.local/on'

Whereas cron(8) is used for scheduling recurring jobs, at is used for scheduling one-off tasks. For example, here are some ways I’ve found it useful:

Reminding myself to take a break

$ at now + 1 minute <<'EOF'
osascript -e "display dialog \"Stop what you're doing!\" with title \"Take a break\""
EOF

Posting fresh data before a team sync

$ echo "curl https://api.example.com/stats | jq . | ~/pipe-to-slack.sh" | at 9:15AM tomorrow

Stopping that debug container I’m probably going to forget about

$ echo "docker stop my-debug" | at 1700 friday

Although I haven’t tried it myself, you can even implement “recurring” jobs by recursively rescheduling the next run at the end of the current job. That said, if such a job fails, the chain breaks silently; there is no built-in retry or alerting like cron provides.

$ cat ~/daily.sh
#!/bin/bash
# Do some work here...
at -f ~/daily.sh 5pm tomorrow

$ at -f ~/daily.sh 5pm
job 43 at Mon Feb 23 17:00:00 2026

This example uses the -f flag to tell at to read the job from a file.

After a job is executed, its output will be captured and sent to you via local sendmail(8)³. On macOS, mail lands in /var/mail/$USER by default, and you can read it with mail(1).

Keep in mind that at snapshots your exported environment (working directory, env vars, umask) from wherever you schedule a job, but it does not capture your shell profile (e.g. anything from .bashrc, .zshrc, etc.). Therefore, it’s common practice to use absolute paths and avoid shell-specific features.

Quick Start

To get started using at on macOS, you’ll need to manually enable the atrun(8) daemon using launchctl(1):

$ sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

Command reference:

at <time>: schedules a job
atq: lists pending jobs
at -c <id>: shows what a job will run
atrm <id>: cancels a job
at -f script.sh 3pm: reads from a file instead of stdin
at -m: sends mail even if there’s no output

Like this: Lampioncino_solare.jpg, Antonia Mette, CC BY-SA 4.0, via Wikimedia Commons ↩
“…the following keywords may be specified: midnight, noon, or teatime (4pm)…” at(1) - POSIX specification ↩
By default, at only sends mail if a job produces output. If you want to receive mail even when there’s no output, you can use the -m flag. ↩

Good Citizen Syndrome

2026-01-26T00:00:00+00:00

Good Citizen Syndrome is what happens when engineering leaders—often unwittingly—confuse good reliable engineers with good reliability engineering.

Skilled engineers who care about reliability constantly step in to fill gaps in process and standards. They’re Good Citizens. While this can be a sign of a mature organization, latent conditions persist when leadership mistakes this vigilance for sustainable systemic health.

In his book about organizational culture, Karl Weick calls reliability a “dynamic non-event;” a phrase which underscores the false sense of safety good reliability can cause.

Reliability is dynamic in the sense that it is an ongoing condition in which problems are momentarily under control due to compensating changes in components… Operators see nothing and seeing nothing, presume that nothing is happening.¹

When reliability engineering² is done well, nothing bad happens. When nothing bad happens, it looks like nothing is being done at all. But it takes a lot of work to make nothing happen!

Weick, K. E. (1987). Organizational culture as a source of high reliability. California Management Review, 29(2), 112-127. ↩
Replace “reliability engineering” with any practice you care about, e.g. security, quality assurance, compliance, maintenance, etc. ↩

Event Streaming vs Background Jobs

2025-07-21T00:00:00+00:00

I was recently asked the question:

“Can we just use ActiveJob instead of Kafka?”

I think there may have been a misunderstanding of the differences between event streaming platforms (like Apache Kafka) and background job systems (like ActiveJob).

At their core, they both solve problems related to scaling and asynchronous processing. But, we can think of one as managing a log of events, and the other as managing a to-do list.

Log of Events

Streaming platforms like Kafka are designed for decoupling components which all need to be informed about specific events and when they happened, using a publish-subscribe model.

To do this, message brokers capture events from various sources called producers (or publishers), and broadcast them as a continuous stream (or log).

These events can then be read by one or more consumers (or subscribers), each using the data for its own purpose.

This is useful for scenarios where different consumers need to react differently to the same event. For example, one consumer might record the event data in a data lake, another might update a search index, and yet another might trigger an alert, all in response to the same event.

Crucially, the log preserves the order of events. This ensures that if a record is updated and then deleted, consumers process those changes in that exact sequence.

Each consumer can read the log independently of each other. The log of events is often durable, meaning they’re stored for a given period of time, allowing some consumers to react in near real time, while others choose to read at their leisure.

To-Do List

Background job systems, like Resque or SolidQueue, are built for offloading work from an application’s main process, using a queue-based model.

A job acts like an instruction for work to be done, most often as soon as possible. It’s placed on a queue by a job orchestrator (or scheduler).

Workers wait on the queue for a job to do. When the worker finds one, it consumes and processes it such that no other worker picks it up.

This is useful for tasks that are too resource-intensive to run in the main application process, but that you’d otherwise like to happen sequentially.

For example, when a user uploads a new avatar, you may want to resize the image and store it in a CDN without blocking the application server from responding to new requests.

Workers are interchangeable, which allows for load balancing between them and horizontally scaling additional capacity when needed. Background jobs can also be durable, allowing them to be retried if a worker crashes or a job fails.

The Confusion

The overlap between these two systems is that they both deal with asynchronous processing and scaling.

However, while Kafka can mimic a job queue using consumer groups, relying on it solely for background jobs means you incur the operational complexity of a streaming platform without leveraging its primary advantages.

In particular, the benefit of allowing multiple consumers to react differently to the same event is lost when you design for only one consumer to process the event like a background job worker would.

Additionally, platforms like Kafka often come with the cost of increased complexity. Although you can use Kafka as a background job system, you likely do not have a use case that justifies the overhead.

The Conclusion

Event Streams are about history. They broadcast facts (events) in a specific order to multiple services that may use that information in entirely different ways.

Background Jobs are about tasks. They offload specific units of work (jobs) to be executed once by any available worker.

If you need to tell the whole system that “something happened,” use a stream. If you need a specific worker to “do something now,” use a job queue.

Chrome's SSL Bypass Cheatcode

2025-07-17T00:00:00+00:00

This is Unsafe

If you type thisisunsafe on a Chrome SSL error page, Chrome will bypass the error and load the page for you.

Try it yourself here: https://expired.badssl.com/

There’s no textbox to type into, just type thisisunsafe blindly with the page in focus.

TIP: To revert the bypass, click the “Not Secure” button in the URL bar and then click “Turn on warnings.”

History of the Bypass Code

My first reaction upon discovering thisisunsafe was one of delight! I was flooded with memories of trying ↑ ↑ ↓ ↓ ← → ← → B A on every Playstation game I had as a kid (see Konami Code).

By why does Chrome have a cheatcode?

Like Tomb Raider searching for ancient artifacts, my sense of wonder led me to dig through the history of Chromium to try and find out how this code came to be. I was curious to know if it was a joke, a mistake, or something else entirely.

Here’s what I found.

Danger (2014)

https://codereview.chromium.org/480393002/patch/60001/70017

/*
 * This allows errors to be skippped [sic] by typing "danger" into the page.
 * @param {string} e The key that was just pressed.
 */
function handleKeypress(e) {
  var BYPASS_SEQUENCE = 'danger';
  if (BYPASS_SEQUENCE.charCodeAt(keyPressState) == e.keyCode) {
    keyPressState++;
    if (keyPressState == BYPASS_SEQUENCE.length) {
      sendCommand(CMD_PROCEED);
      keyPressState = 0;
    }
  } else {
    keyPressState = 0;
  }
}

The bypass code was first introduced in 2014. Originally set to “danger,” it was newly created part of a larger piece of work related to DRY-ing up duplication in chrome/browser/resources/safe_browsing/ and chrome/browser/resources/ssl/.

See: Aug 11, 2014 18:03UTC - Chromium Issue #41125304:

Unfortunately, the reason it was added doesn’t appear to have been documented. My guess is that developers needed a convenient way to bypass SSL errors during the rise of HTTPS adoption and enforcement.

Bad Idea (2015)

https://codereview.chromium.org/1416273004/patch/1/10001.

--- a/components/security_interstitials/core/browser/resources/interstitial_v2.js
+++ b/components/security_interstitials/core/browser/resources/interstitial_v2.js
@@ -40,7 +40,7 @@ function sendCommand(cmd) {
  * @param {string} e The key that was just pressed.
  */
 function handleKeypress(e) {
-  var BYPASS_SEQUENCE = 'danger';
+  var BYPASS_SEQUENCE = 'badidea';
   if (BYPASS_SEQUENCE.charCodeAt(keyPressState) == e.keyCode) {
     keyPressState++;
     if (keyPressState == BYPASS_SEQUENCE.length) {

A year later, in 2015, the BYPASS_SEQUENCE was changed to badidea. Like before, there’s little evidence left to understand the intention behind the changes.

However, as we’ll see, later changes were made due to concerns around the bypass code’s popularity and misuse, so it seems likely that the change to badidea was made for similar reasons.

Interestingly, in 2014, Google published a paper entitled Experimenting At Scale With Google Chrome’s SSL Warning, where authors experimented with ways to reduce the number of users who bypassed SSL warnings via the UI.

Our goal in this work is to decrease the number of users who click through the Google Chrome SSL warning… We investigate several factors that could be responsible: the use of imagery, extra steps before the user can proceed, and style choices.

Felt, Adrienne Porter, et al. “Experimenting At Scale With Google Chrome’s SSL Warning.” (2014).

This is not Safe (2018)

--- a/components/security_interstitials/core/browser/resources/interstitial_large.js
+++ b/components/security_interstitials/core/browser/resources/interstitial_large.js
@@ -13,7 +13,7 @@
  * @param {string} e The key that was just pressed.
  */
 function handleKeypress(e) {
-  var BYPASS_SEQUENCE = 'badidea';
+  var BYPASS_SEQUENCE = 'thisisnotsafe';
   if (BYPASS_SEQUENCE.charCodeAt(keyPressState) == e.keyCode) {
     keyPressState++;
     if (keyPressState == BYPASS_SEQUENCE.length) {

On January 03, 2018, the bypass code was updated again, this time to thisisnotsafe.

Unlike before, the code was changed explicitly due to concern around the growing popularity of being able to bypass SSL warnings in Chrome using the bypass code.

The security interstitial bypass keyword hasn’t changed in two years and awareness of the bypass has been increased in blogs and social media. Rotate the keyword to help prevent misuse.

Jan 03, 2018 03:03UTC - Chromium Issue #843085

Based on the source code, the bypass code was only intended for testing and not for general use. The click-through UI or Chrome flags were abled to be monitored for patterns, like we saw in the 2014 paper. Use of the bypass code, however, doesn’t seem to have been tracked.

dGhpc2lzdW5zYWZl (2018 - Present)

--- a/components/security_interstitials/core/browser/resources/interstitial_large.js
+++ b/components/security_interstitials/core/browser/resources/interstitial_large.js
@@ -13,7 +13,10 @@
  * @param {string} e The key that was just pressed.
  */
 function handleKeypress(e) {
-  var BYPASS_SEQUENCE = 'thisisnotsafe';
+  // HTTPS errors are serious and should not be ignored. For testing purposes,
+  // other approaches are both safer and have fewer side-effects.
+  // See https://goo.gl/ZcZixP for more details.
+  var BYPASS_SEQUENCE = window.atob('dGhpc2lzdW5zYWZl');
   if (BYPASS_SEQUENCE.charCodeAt(keyPressState) == e.keyCode) {
     keyPressState++;
     if (keyPressState == BYPASS_SEQUENCE.length) {

Just a few days later, on January 10, 2018, the bypass code was changed once again:

thisisnotsafe was changed to dGhpc2lzdW5zYWZl, in what I believe was an attempt at obfuscation.

$ echo dGhpc2lzdW5zYWZl | base64 -d
thisisunsafe

This made no difference in terms of functionality; the code was simply base64 encoded, and the window.atob function was used to decode it back to its original form.

Along with this code change, the Chromium developers added a comment linking to a public document titled: Deprecating Powerful Features on Insecure Origins.

Though the document made no mention of the bypass code itself, it included instructions for how to bypass SSL errors during development and testing:

You can use chrome://flags/#unsafely-treat-insecure-origin-as-secure to run Chrome, or use the --unsafely-treat-insecure-origin-as-secure="http://example.com" flag (replacing "example.com" with the origin you actually want to test), which will treat that origin as secure for this session.

I presume this is what was meant by the “…other approaches are both safer and have fewer side-effects,” comment in the code snippet above.

Is this Unsafe?

Despite all the digging, I am still unaware of the intention behind the bypass code’s original creation.

/**
 * This allows errors to be skippped [sic] by typing a secret phrase into the page.
 * @param {string} e The key that was just pressed.
 */
function handleKeypress(e) {
  // HTTPS errors are serious and should not be ignored. For testing purposes,
  // other approaches are both safer and have fewer side-effects.
  // See https://goo.gl/ZcZixP for more details.
  const BYPASS_SEQUENCE = window.atob('dGhpc2lzdW5zYWZl');
  if (BYPASS_SEQUENCE.charCodeAt(keyPressState) === e.keyCode) {
    keyPressState++;
    if (keyPressState === BYPASS_SEQUENCE.length) {
      sendCommand(SecurityInterstitialCommandId.CMD_PROCEED);
      keyPressState = 0;
    }
  } else {
    keyPressState = 0;
  }
}

As of July 2025, the bypass (along with the skippped typo) has remained unchanged. You can see it in the latest version: Chromium (140.0.7301.1).

It still shows up in blogs and social media posts today (though none of them do a deep dive like this one!), which is how I stumbled upon it.

What may have started as a convenience for developers, later became a point of concern due to its potential misuse. The change to base64 encoding was likely an attempt to obscure the code from casual users or code scanners, but it is by no means a secret.

The rise of HTTPS enforcement has been a net-positive for the web, but it’s hard to articulate the risks of broken SSL to everyday users and why they shouldn’t simply ignore it.

Is it the job of a browser to keep users safe? Even if one could argue that web access shouldn’t be gated by corporations on the basis of security, the enforcement of HTTPS has undoubtedly incentivized web developers to adopt better security practices.

What do you think?

Debouncing API Calls

2025-07-02T00:00:00+00:00

Debounce Delay: 500ms

Hardware

The term “debouncing” originates from electronics (at least that’s where I first came across it). When you press a physical button or flip a switch, its metal contacts don’t connect cleanly just once; they “bounce,” opening and closing rapidly, creating multiple pulses of current as the circuit connects and disconnects.

Although this happens on the order of milliseconds, filtering the rapid switching is necessary to prevent other components from misinterpreting the bounces as intentional signals. Debouncing in hardware ensures that only a single, clean signal is sent when the button is pressed.

Software

In software, we hit this same problem.

Take search bars, for example. They often fire off an API call on every keystroke to give search results to users with broken enter keys.

With a typing speed of around 200 characters-per-minute (3.33 characters-per-second), a user searching for “parachute pants” could trigger 16 API calls in about 5 seconds—and that’s assuming they don’t make any typos.

We can’t possibly know which of those 16 API calls actually improve the user experience, but if I had to guess, search suggestions are most effective when presented after a natural pause in typing—not during every keystroke.

Debouncing with Delays

This debounce function uses setTimeout to delay the execution of a callback.

function debounce(callback, delay) {
  let timerId;
  return (...args) => {
    clearTimeout(timerId);
    timerId = setTimeout(() => {
      callback(...args);
    }, delay);
  };
}

If the returned function gets called again before the timer finishes, it clears the previous timer and starts a new one. This resets the wait time and prevents callbacks from executing until there has been a pause in calls for the specified delay.

This is exactly what the demo above does. It waits for you to stop typing for a set number of milliseconds before updating the output field with your input.

Delays and Cancellations

For input-triggered API calls (like the search bar example) we can take this further by combining debouncing with request cancellation.

When the user stops typing in the search bar, the app waits until after a delay (similar to our earlier debounce implementation) before making an API call to fetch search suggestions.

If, however, the user begins typing again before the fetch resolves, we need a way to throw away the request. This is because, by the time we would receive a response, the user has already moved on and started searching for something else.

So, before making any new API requests, we first cancel the previous fetch, using the AbortController API.

function debouncedFetch(delay) {
  let timerId;
  let controller;

  return (...args) => {
    clearTimeout(timerId);
    if (controller) {
      controller.abort();
    }

    controller = new AbortController();
    const signal = controller.signal;

    timerId = setTimeout(async () => {
      const response = await fetch(`https://example.com`, {
        signal: controller.signal
      });
      // Handle the response here
    }, delay);
  };
}

This debouncedFetch function works in three steps.

First, it clears any pending timer and cancels any in-flight request. If there’s no previous request (like on the first call), the latter step is skipped.

Then, it creates a new AbortController and extracts its signal. This signal acts like a remote control that can cancel the upcoming fetch request at any time. This works because the fetch API supports aborting requests using an AbortSignal.

Finally, it starts a new timer. When the timer expires, it calls fetch. And when fetch resolves, we handle the response as needed.

So, as before, if the user types again before the timer expires, the timer is restarted before a new fetch request is made.

In addition to this, if the user types again while the previous fetch is still in progress, that request is cancelled, the new input is debounced, and then finally, a new fetch request is made with the latest input.

Where Does Apple Hide Your Voice Memo Transcripts?

2025-06-08T00:00:00+00:00

Much like having thoughts in the shower and working on trains, I often find my best thinking happens while walking my dog in the morning. To not lose track of all this good thinking, I’ve started recording voice notes on my iPhone using Apple’s Voice Memos app.

After recording, the app automatically generates transcriptions, allowing me to easily search, copy, and edit my memos. Unfortunately—Apple being Apple—you can’t access these transcripts from anywhere outside the Voice Memos app.

Just want to see the code? Check out these Gists:

Ruby Bash

Using Voice Memos ticks a lot of boxes for me:

I can be a better dog dad: I used to bring a small notebook on our morning walks, but I felt rude asking my dog to wait while I scribbled something every few meters.
I get to think out loud: I’ve found that verbalizing my ideas helps me organize, prioritize, and clarify them.
I actually use my notes: Writing emails, drafting blog posts, making todo lists; automatic transcriptions are queryable and ready for editing.

This is what’s called a “win-win-win”—and I love it when it happens—but that last “win” doesn’t come as easily as the others.

Step 0: The Problem

Unfortunately, the only way to get your transcripts out of Voice Memos and into somewhere useful, is to clumsily copy-and-paste them.

This is an example of the worst kind of problem.

Copying-and-pasting isn’t a big deal, I know. But it is just enough of a deal to stop me from using the transcripts like I want to—which have become more valuable to me than the audio itself.

Plus, if there’s anything we programmers hate most, it’s copying-and-pasting… isn’t that what we have computers for?

Step 1: Find the Transcripts

Thinking of how I might solve this, I was reminded of a time when I tried exporting text from Apple Notes. When synced via iCloud, I learned that Notes uses SQLite and that it was accessible on Mac.

~/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite

Thankfully, the authors of tools like apple-notes-to-sqlite had already deciphered and documented the complicated schemas and built open source tools to solve my problem.

I dove in hoping the same would be true for Voice Memos.

However, in the far reaches of Apple StackExchange, I came across this:

Q. Where are the transcripts that are generated from recordings in the Voice Memos app stored in the file system?

A. There is no documentation from Apple that I know of, but the MPEG-4 standard provides for encoding a text stream in the container itself; i.e., the *.m4a file. That must be what Voice Memos is doing. I don’t see any separate file resulting from transciption [sic]. See: ISO/IEC 14496-17:2006(en) Information technology — Coding of audio-visual objects — Part 17: Streaming text format (iso.org)

— Location of Voice Memos transcripts in the file system?, Apple StackExchange

The good news was that, when synced via iCloud, the Voice Memos audio could be accessed as .m4a files directly.

~/Library/Group Containers/group.com.apple.VoiceMemos.shared/Recordings/

The bad news was that I didn’t know anything about audio file formats or what “…encoding a text stream in the container itself…” meant.

Step 2: Explore the Files

To postpone having to open up any part of ISO/IEC 14496¹, I thought I’d just poke around the .m4a files a bit.

If you try opening one of those files in a text editor, you’ll see that it is not meant to be read by humans. But, useful plaintext strings can often be found buried within various binary formats.

Finding these hidden strings is something reverse engineers do when they otherwise don’t know how to decompile or deconstruct a binary file. (.m4a is a well established format, but we haven’t read the documentation yet, remember?)

If Apple is encoding transcripts directly within the .m4a file, as the StackExchange answer suggested, we might be able to see them using the command line tool, strings.

$ strings -n 24 -o '*.m4a'

7724    com.apple.VoiceMemos (iPhone Version 18.5 (Build 22F76))
7438979 tsrp{
  "attributedString":{
    "attributeTable":[{"timeRange":[0,2.52]},{"timeRange":[2.52,2.7]},{"timeRange":[2.7,2.88]},{"timeRange":[2.88,3.12]},{"timeRange":  [3.12,3.72]},{"timeRange":[3.72,4.74]},{"timeRange":[4.74,4.92]},{"timeRange":[4.92,5.22]},{"timeRange":[5.22,5.34]},{"timeRange":[5.34,5.52]},{"timeRange":[5.52,5.7]},{"timeRange":[5.7,6.6]},{"timeRange":[6.6,6.78]},{"timeRange":  [6.78,6.96]},{"timeRange":[6.96,7.2]},...],
    "runs":["OK,",0," I",1," went",2," back",3," and",4," re",5," read",6," at",7," least",8," the",9," beginning",10," of",11,  " the",12," Google",13," paper",14,...]
  },
  "locale":{"identifier":"en_US","current":0}
}
7607856 date2025-06-08T08:46:16Z
7607958 Self-Replicators Metrics & Analysis Architecture

# -n <number>: The minimum string length to print
# -o: Preceded each string by its offset (in decimal)

Sure enough, strings reveals that, starting at byte 7438979, the .m4a file contains a JSON-esque string which has some of my Voice Memos transcript in it.

If it really was JSON (spoiler: it was, except for the “tsrp” part before the opening brace), then we should be able to extract and parse it.

Step 3: JSON Extraction

Sticking to the commandline, let’s use a few more tools for wrangling this text.

strings, as before, will output printable strings,
rg, will filter for that tsrp prefix,
sed, will remove the prefix and leave just the JSON, and finally,
jq, will let us explore and manipulate the payload.

Author’s note: this was a very exploratory and iterative process, especially the jq code. I’ve written a tool called ijq to help with this. You can read about it here: Interactive jq

The first thing I’m interested in is the structure of the JSON.

  # Extract printable strings from the audio file
$ strings '*.m4a' \
  # Only keep line that starts with the "tsrp" prefix
| rg 'tsrp' \
  # Remove the "tsrp" prefix keep only the JSON
| sed 's/tsrp//g' \
    # Extract the paths of every key in the JSON
    # If the path is an array index, replace it with brackets
    # Join the paths with a dot
    # Remove duplicates
| jq '[paths | map(if type == "number" then "[]" else . end) | join(".")] | unique'

[
  "attributedString",
  "attributedString.attributeTable",
  "attributedString.attributeTable.[]",
  "attributedString.attributeTable.[].timeRange",
  "attributedString.attributeTable.[].timeRange.[]",
  "attributedString.runs",
  "attributedString.runs.[]",
  "locale",
  "locale.current",
  "locale.identifier"
]

Take a closer look at how the jq code works

$ cat demo.json
{ "foo": [ { "bar": [ 1, 2, 3 ] } ] }%

# Extract the paths of every key in the JSON
$ cat demo.json | jq -c '[paths]'
[["foo"],["foo",0],["foo",0,"bar"],["foo",0,"bar",0],["foo",0,"bar",1],["foo",0,"bar",2]]

# If the path is an array index, replace it with
$ cat demo.json | jq -c '[paths | map(if type == "number" then "[]" else . end)]'
[["foo"],["foo","[]"],["foo","[]","bar"],["foo","[]","bar","[]"],["foo","[]","bar","[]"],["foo","[]","bar","[]"]]

# Join the paths with a dot
$ cat demo.json | jq -c '[paths | map(if type == "number" then "[]" else . end) | join(".")]'
["foo","foo.[]","foo.[].bar","foo.[].bar.[]","foo.[].bar.[]","foo.[].bar.[]"]

# Remove duplicates
$ cat demo.json | jq -c '[paths | map(if type == "number" then "[]" else . end) | join(".")] | unique'
["foo","foo.[]","foo.[].bar","foo.[].bar.[]"]

The transcript data is stored in an attributedString object, which has an attributeTable (an array of timeRange objects) and a runs array. The locale object contains the language identifier and current locale.

The actual text of the transcript is inside the runs array. Each word is followed by an index.

$ strings '*.m4a' \
  | rg "tsrp"  \
  | sed 's/tsrp//g'
  | jq -c '[limit(30; .attributedString.runs[])]'

 ["OK,",0," I",1," went",2," back",3," and",4," we",5," read",6," at",7," least",8," the",9," beginning",10," of",11," the",12," Google",13," paper",14]

The index maps to a timeRange in the attributeTable array.

Each timeRange is an array itself, containing two numbers: the start and end time offset of each transcribed word, in seconds.

$ strings '*.m4a' \
  | rg "tsrp"  \
  | sed 's/tsrp//g'
  | jq  -c '[limit(15; .attributedString.attributeTable[].timeRange)]'

 [[0,2.52],[2.52,2.7],[2.7,2.88],[2.88,3.12],[3.12,3.72],[3.72,4.74],[4.74,4.92],[4.92,5.22],[5.22,5.34],[5.34,5.52],[5.52,5.7],[5.7,6.6],[6.6,6.78],[6.78,6.96],[6.96,7.2]]

Let’s just take a moment to pause and appreciate that, despite not knowing anything about audio formats or how text streaming works in ISO/IEC 14496-17:2006(en), we were able to find the raw transcript embedded in the .m4a file!

From here, we could continue wrangling and reconstruct the text.

$ ... | jq -c '[limit(30; .attributedString.runs[])] | map(if type == "string" then . else empty end) | join("")'

"OK, I went back and we read at least the beginning of the Google paper"

But, since the dog has already gone for a walk, I guess there’s no better time than now to crack open ISO/IEC 14496, and really try to understand what’s going on.

Containers

.m4a files are a type of MPEG-4 (.mp4), specifically an audio-only variant. The MPEG-4 Part 14 (.mp4) standard is an implementation of the ISO Base Media File Format (Part 12 of ISO/IEC 14496).

These files are called “container formats” because they bundle together multiple different types of data.

The “A” in .m4a stands for “audio”, but these file types can also hold things like images, video, artist metadata, chapter markers, and yes, even transcriptions.

Atoms

The “container” is organized in a hierarchical structure of “boxes” (which Apple calls “atoms”). Each atom contains a particular piece of data related to the media being stored; like metadata, configuration, the media itself, or other nested atoms.

We can use the mp4dump command-line tool to look at the atom structure of our .m4a file.

$ mp4dump '.m4a'

View full mp4dump output

[ftyp] size=8+20
  major_brand = `.m4a`
  minor_version = 0
  compatible_brand = `.m4a`
  compatible_brand = isom
  compatible_brand = mp42
[mdat] size=16+7279219
[moov] size=8+328928
  [mvhd] size=12+96
    timescale = 16000
    duration = 38190080
    duration(ms) = 2386880
  [trak] size=8+328457
    [tkhd] size=12+80, flags=1
      enabled = 1
      id = 1
      duration = 38190080
      width = 0.000000
      height = 0.000000
    [mdia] size=8+159480
      [mdhd] size=12+20
        timescale = 16000
        duration = 38190080
        duration(ms) = 2386880
        language = und
      [hdlr] size=12+37
        handler_type = soun
        handler_name = Core Media Audio
      [minf] size=8+159391
        [smhd] size=12+4
          balance = 0
        [dinf] size=8+28
          [dref] size=12+16
            [url ] size=12+0, flags=1
              location = [local to file]
        [stbl] size=8+159331
          [stsd] size=12+91
            entry_count = 1
            [mp4a] size=8+79
              data_reference_index = 1
              channel_count = 2
              sample_size = 16
              sample_rate = 16000
              [esds] size=12+39
                [ESDescriptor] size=5+34
                  es_id = 0
                  stream_priority = 0
                  [DecoderConfig] size=5+20
                    stream_type = 5
                    object_type = 64
                    up_stream = 0
                    buffer_size = 6144
                    max_bitrate = 24000
                    avg_bitrate = 24000
                    DecoderSpecificInfo = 14 08
                  [Descriptor:06] size=5+1
          [stts] size=12+12
            entry_count = 1
          [stsc] size=12+28
            entry_count = 2
          [stsz] size=12+149188
            sample_size = 0
            sample_count = 37295
          [stco] size=12+9952
            entry_count = 2487
    [udta] size=8+168869
      [tsrp] size=8+168861
  [udta] size=8+347
    [date] size=8+20
    [meta] size=12+307
      [hdlr] size=12+22
        handler_type = mdir
        handler_name =
      [ilst] size=8+265
        [.nam] size=8+64
          [data] size=8+56
            type = 1
            lang = 0
            value = Self-Replicators Metrics & Analysis Architecture
        [----] size=8+107
          [mean] size=8+20
            value = com.apple.iTunes
          [name] size=8+19
            value = voice-memo-uuid
          [data] size=8+44
            type = 1
            lang = 0
            value = DECAFCAFE-ABCD-ABCD-ABCD-AAAAAAAAAAAA
        [.too] size=8+70
          [data] size=8+62
            type = 1
            lang = 0
            value = com.apple.VoiceMemos (iPad Version 15.5 (Build 24F74))

Each atom begins with an 8-byte header (typically). The first 4 bytes specify the atom size (a 32-bit integer indicating the total size of the atom, including the header). The next 4 bytes specify the type (e.g., ftyp, mdat, moov), a four-character code identifying how to interpret the atom’s payload.²

Note: mp4dump outputs the size as "#{header_size}+#{payload_size}".

[ftyp] size=8+20
[moov] size=8+328928
  ...
  [trak] size=8+328457
    [tkhd] size=12+80, flags=1
    [mdia] size=8+159480
      [mdhd] size=12+20
    ...
    [udta] size=8+168869
      [tsrp] size=8+168861
...

The mp4dump output shows us the headers of each atom. Here, we can see one is of type "tsrp", which matches the string we found earlier with strings.

tsrp is a “leaf” atom since it contains no child atoms in the hierarchy. Leaf atoms are the ones that usually contain media data, and this particular tsrp atom takes up 168861 bytes.

Apple defines what each of these atoms are for and how they’re structured in their Quicktime File Format documentation.³ The documentation provides a list of the most common atoms, their types, and their purposes.

Atom Type	Description
`ftyp`	An atom that identifies the file type specifications with which the file is compatible.
`moov`	An atom that specifies the information that defines a movie.
`trak`	An atom that defines a single track of a movie.
`tkhd`	An atom that specifies the characteristics of a single track within a movie. (Required by `trak`)
`mdia`	An atom that describes and defines a track’s media type and sample data. (Required by `trak`)
`mdhd`	An atom that specifies the characteristics of a media, including time scale and duration. (Required by `mdia`)
`udta`	An atom where you define and store data associated with a QuickTime object, e.g. copyright.
`tsrp`	NULL??

As you might have noticed, I didn’t include a definition for the tsrp atom in the table above.

This is because Apple doesn’t document it, and being a custom atom, it’s not part of the ISO/IEC 14496 standard itself.

All we know is that tsrp is a custom atom type used to store transcriptions of audio recordings in JSON.

I’m very curious about this:

Why doesn’t Apple document their custom tsrp atom, similar to how they document other custom atoms?

Why have a custom atom at all? Surely there might be a suitable type already in the standards?⁴

If not, why use a JSON string, instead of nesting text atoms within the tsrp atom?⁵

And, what are the trade-offs to storing the transcript in SQLite, similar to Notes?

One benefit of storing the transcript directly in the audio file is its portability. Saving the file means you also save the transcript, even outside of Apple’s ecosystem.

Then again, Apple has this to say about undocumented atom types:

If your application encounters an atom of an unknown type, it should not attempt to interpret the atom’s data. Use the atom’s size field to skip this atom and all of its contents. This allows a degree of forward compatibility with extensions to the QuickTime file format.

—QuickTime File Format/Storing and sharing media with QuickTime files/Atoms

Accessing the Transcript Atom

Ignoring Apple’s advice, I’ve written a Ruby script to attempt to interpret this atom’s data.

Using what we’ve discovered, the script parses atom headers, recursively traverses the hierarchy to find the tsrp atom, and then extracts and reconstructs the JSON payload within.

Ruby	https://gist.github.com/thomascountz/b84b68f

It was a great learning experience to read through the documentation and write the Ruby script. But, to make the job even easier, we can use purpose-built tools and libraries.

mp4extract (which comes bundled with the same tools as mp4dump), takes an atom path, like “moov/trak/udta/tsrp”, and outputs its payload (using the --payload-only option avoids also outputting the header).

$ mp4extract --payload-only moov/trak/udta/tsrp '*.m4a' tsrp.bin

Once we have the payload, we can use jq to not only parse the JSON, but also extract the text from the runs array and concatenate it into a single string.

cat tsrp.bin | jq '.attributedString.runs | map(if type == "string" then . else empty end) | join("")'

This now means that we only need to copy-and-paste a few lines of code to get the transcripts out of Voice Memos and into somewhere useful!

Better dog dad? Win.
Thinking out loud? Win.
~~No copy-and-paste?~~ Improved workflow? Win.

Postscript

First, I hope you enjoyed this journey into reverse engineering Apple’s Voice Memos format. I had fun and I hope you did too.

If you have any questions, comments, or actually know something about this stuff, please feel free to reach out!

Oh, and if we actually want to avoid having to copy-and-paste, we can, of course, write a bash script :)

Bash Script

https://gist.github.com/thomascountz/287d7dd

Footnotes

The ISO/IEC 14496 standard is a huge standard with multiple parts. Part 12 defines the ISO Base Media File Format, which is the core of many media file formats, including .mp4 and .m4a. Part 17, which the StackExchange answer mentions, defines the Streaming Text Format, which is used for text streams in media files. ↩
The size field may also contain a special value. A 0 is used to indicate that the atom is the last one in the file. A 1 is used to indicate that the atom is larger than 2^32 bytes, and it is followed by an additional 64-bit unsigned integer that contains the actual size of the atom. ↩
Which is convenient, since the real source-of-truth (ISO/IEC 14496) is not available for free. ↩
Perhaps a standard timed text track (e.g., tx3g) within the minf atom could have been used? ↩
Or use a QT atom or atom container. See: https://developer.apple.com/documentation/quicktime-file-format/qt_atoms_and_atom_containers ↩

Forth Feedback Loop

2025-06-03T00:00:00+00:00

Inspired by Andreas Wagner’s ecr word, I created the word go:

\ go.fs

: go#empty  s" ---marker--- marker ---marker---" evaluate ;
: go#open   s" vim my_project.fs" system ;
: go#run    s" my_project.fs" included ;

: go go#empty go#open go#run ;

marker ---marker---

marker ---marker--- defines a special word: ---marker---

When executed, it makes Forth forget all definitions made after it (including itself).

The go word orchestrates an edit-reload cycle:

go#empty is the clever bit.
- It evaluate-s the string "---marker--- marker ---marker---".
- This first runs the existing ---marker--- (clearing definitions from the previous load of my_project.fs).
- Then it defines a new ---marker--- at the now-empty top of the dictionary, priming it for the next cycle.
go#open uses gforth’s system to open my_project.fs in vim.
go#run uses included to load the freshly edited my_project.fs.

Color Strings Refinement

2025-05-27T00:00:00+00:00

module Colors
  refine String do
    def red = "\e[31m#{self}\e[0m"
    def green = "\e[32m#{self}\e[0m"
    def yellow = "\e[33m#{self}\e[0m"
    def blue = "\e[34m#{self}\e[0m"
    def magenta = "\e[35m#{self}\e[0m"
    def cyan = "\e[36m#{self}\e[0m"
    def white = "\e[37m#{self}\e[0m"
  end
end

# Usage
using Colors

puts "Hello, World!".green

Good Morning, Team!

2025-03-26T00:00:00+00:00

Every so often, I like to write a Ruby script that says good morning to my team in a fun way. Here is the one I made this morning which mimics the famous bouncing DVD logo.

require "io/console"

greeting = [71, 111, 111, 100, 32, 77, 111, 114, 110, 105, 110, 103, 44, 32, 84, 101, 97, 109, 33].map(&:chr).join
colors = ["\e[31m", "\e[33m", "\e[32m", "\e[36m", "\e[34m", "\e[35m"]
rows, columns = begin
  IO.console.winsize
rescue
  [24, 80]
end
g_width = greeting.size
g_height = 1
x = rand(columns - g_width)
y = rand(rows - g_height)
dx = 1
dy = 1
frame = 0
loop do
  print "\e[H\e[2J"
  puts "\n" * y
  color = colors[frame % colors.size]
  print(" " * x + color + greeting + "\e[0m")
  sleep 0.05
  if (x + dx < 0) || (x + g_width + dx > columns)
    dx = -dx
    frame += 1
  end
  if (y + dy < 0) || (y + g_height + dy > rows)
    dy = -dy
    frame += 1
  end
  x += dx
  y += dy
  sleep 0.25
end

Run Commands on File Change

2025-03-17T00:00:00+00:00

I use entr to run commands automatically when files change. Here’s how I use it to run tests.

Use:

ls lib/* spec/* | entr -c bin/test

Notes:

Install: brew install entr
-c clears the screen before running the command
find lib spec -name '*.rb' works similarly to ls, but I can’t always remember the syntax
git ls-files is another option for watching tracked files in a repo
watchexec is an alternative to entr, known for being more feature-rich

Ruby Bug #21117: Numbered Block Params

2025-02-20T00:00:00+00:00

tl;dr

Numbered block params (_1, _2, etc.) are reserved and read-only, while it (the new block param in Ruby 3.4) has different mutation rules for compatibility.
Using combined assignment operators (e.g. +=, &&=), numbered block params could be overwritten without error due to a discrepancy between parse.y and Prism (Ruby’s default parser since 3.3).
A patch has been applied to Prism in Ruby 3.4.2 to explicitly check for writes to numbered params via combined assignment operators.

The Bug

In Ruby, block-local numbered parameters (_1, _2, etc.) provide a concise way to reference block arguments without explicit naming. They have received renewed attention recently with the introduction of the it parameter in Ruby 3.4.

Though they serve a similar purpose, numbered parameters are designed to be read-only, while it is intentionally mutable for backward compatibility reasons.

In Bug #21117, radarek (Radosław Bułat) identified inconsistencies in how Ruby handles assignments to _1 and it. Most concerning was that _1 could be modified in specific cases, violating its intended immutability.

Direct assignment correctly raises SyntaxError:

irb(main):001> [1, 2, 3].each { _1 = _1 + 1; p _1 }
<internal:kernel>:168:in 'Kernel#loop': (irb):1: syntax error found (SyntaxError)
> 1  | [1, 2, 3].each { _1 = _1 + 1; p _1 }
     |                  ^~ Can't assign to numbered parameter _1

However, combined assignment operators work without error:

irb(main):002> [1, 2, 3].each { _1 += 1; p _1 }
2
3
4
=> [1, 2, 3]

The root cause was an implementation difference between Prism (Ruby’s default parser since 3.3) and parse.y. Prism failed to enforce numbered parameter immutability for combined assignment operations, while the legacy parse.y parser correctly rejected these operations:

# $ ruby --parser 'parse.y' -e 'binding.irb'

irb(main):001> [1, 2, 3].each { _1 += 1; p _1 }
<internal:kernel>:168:in 'Kernel#loop': _1 is reserved for numbered parameter (SyntaxError)

The Fix

Commit d3fc56d ports Kevin Newton’s (kddnewton) fix from Prism. The update adds explicit checks when parsing combined assignment tokens (PM_TOKEN_#{op}_EQUAL) to reject any attempts to modify numbered parameters.

Simplified code snippet from the fix:

// In prism.c:
switch (token.type) {
  // Cases for all combined assignment operators
  case PM_TOKEN_PIPE_PIPE_EQUAL:
  // ...other operator cases...
  case PM_TOKEN_PLUS_EQUAL:
  case PM_TOKEN_SLASH_EQUAL:
  case PM_TOKEN_STAR_EQUAL:
  case PM_TOKEN_STAR_STAR_EQUAL:
    switch (PM_NODE_TYPE(node)) {
        case PM_LOCAL_VARIABLE_READ_NODE: {
            // Check if we're trying to modify a numbered parameter
            if (pm_token_is_numbered_parameter(node->location.start, node->location.end)) {
                PM_PARSER_ERR_FORMAT(parser, node->location.start, node->location.end,
                                     PM_ERR_PARAMETER_NUMBERED_RESERVED, node->location.start);
                parse_target_implicit_parameter(parser, node);
            }
        }
        // ...other cases...
    }
    // ...
}

What About `it`?

The it keyword intentionally has different behavior than numbered parameters for backward compatibility. While direct assignment (e.g. tap { it = 2; p it }) is currently allowed to avoid breaking legacy code, Ruby core developers encourage treating it as read-only for consistency with numbered parameters.

Routine Complications

2025-02-15T00:00:00+00:00

In A Life-Saving Checklist, author Atul Gawande describes a patient suffering from an infection introduced by medical equipment. These “line infections” are so common, they’re said to be a routine complication.

In medical practice, a routine complication refers to an adverse event or difficulty that is known to occur with some regularity in association with a particular procedure or treatment. These complications are expected in the sense that they are well-documented and occur with a known frequency, allowing healthcare providers to anticipate and manage them effectively.

In other domains, such as software development, it is useful to identify routine complications inherent to various operations, and understand both their potential for happening and their potential negative impacts.

As an example, during incident response, if standard operating procedure (SOP) includes restarting a web server, routine complications to be understood include: accidentally restarting a healthy server (misdiagnosis), executing a restart incorrectly (human error), cutting off further observations of the issue before it’s fully understood (state evaporation), and encountering unexpected dependencies that cause cascading failures (hidden coupling).

Just because restarting a web server carries with it these potential issues, doesn’t mean we should shy away from the tried-and-true “turn it off and on again” procedure (every operation carries risk after all). Instead, we should manage routine complications by understanding them honestly, preparing for them openly, and being blameless when they happen.

Hill Chart Web App

2025-02-10T00:00:00+00:00

I built an interactive tool for visualizing task progress.

A hill chart is a visual tool for tracking tasks and projects, using the metaphor of going up and down hill to communicate the different phases of work.

Every piece of work has two phases: an uphill phase where you figure out your approach, and a downhill phase focused on execution.¹

I appreciate hill charts because they embrace the fuzziness of task-based work. Unlike rigid progress metrics like “done/not done,” percentage completion, or T-shirt sizes, hill charts offer a more nuanced way to describe progress. If you export the chart each time you update it, you can also detect trends over time.

This tool runs in the browser via Github Pages and is built with plain JavaScript, d3.js, and is styled with TailwindCSS. You can explore the code by right-clicking → View Source or checking out the GitHub repo.²

Capture ActiveRecord Queries

2025-02-09T00:00:00+00:00

When you’re interested in testing the actual SQL queries generated by ActiveRecord, rather than just what’s returned from the database, you can capture them using ActiveSupport::Notifications.

module QueryCapturing
  IGNORED_QUERIES = ["SCHEMA", "TRANSACTION"]

  def capture_queries
    captured_queries = []
    subscriber = ActiveSupport::Notifications.subscribe("sql.active_record") do |event|
      captured_queries << event.payload[:sql] unless IGNORED_QUERIES.include?(event.name)
    end
    yield captured_queries
  ensure ActiveSupport::Notifications.unsubscribe(subscriber)
  end
end

class ServiceMetadata::QueryCommentsTest < Minitest::Test
  include QueryCapturing

  def test_annotates_queries_with_metadata_comments
    capture_queries do |queries|
      ServiceMetadata::QueryComments.comment { Comment.all.load }

      assert queries.all? { |q| q.match?(/\/\*.*?origin_service: my_service.*?\*\//m) }
    end
  end
end

Ruby CLI Progress Bar

2025-02-08T00:00:00+00:00

def print_progress(title, total, current_progress, bar_width: 50)
  progress_pct = (current_progress.to_f / total) * bar_width
  printf("\r#{title}: [%-#{bar_width}s ] -- %s", "▤" * progress_pct.round, "#{current_progress}/#{total} ")
end

# Usage
1.upto(10) do |i|
  print_progress("Here we go!", 10, i)
  sleep 0.2
end;print("\n")

# Output Example:
# Here we go!: [▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤▤                ] -- 7/10

Checklist App Prototype

2025-02-01T00:00:00+00:00

I’ve been playing around with the idea of building a checklist app that’s geared towards completing procedures.

I remember watching a livestream of a rocket launch and seeing the closeout crew go through a checklist using a tablet strapped to their thigh. I thought, “I’m no NASA engineer, but that looks useful!”

Interactive jq

2025-01-31T00:00:00+00:00

I built an interactive jq TUI using fzf!

I was searching for an interactive jq editor when I came across this repo, which had an intriguing suggestion:

echo '' | fzf --print-query --preview "cat *.json | jq {q}" – An fzf hack that turns it into an interactive jq explorer.

This sent me down a rabbit hole, and I discovered just how incredibly configurable fzf is, e.g.:

You can bind custom keys to execute non-default behaviors:

--bind=ctrl-y:execute-silent(jq {q} $tempfile | pbcopy)

You can start fzf with an initial query:
```
--query="."
```
You can configure fzf with different layouts:
```
--preview-window=top:90%:wrap
```

You can add a multi-line header to provide instructions:

--header=$'ctrl+y : copy JSON\nctrl+f : copy filter\nenter : output\nesc : exit'

I wonder how many different TUIs I can create with just fzf?

Checkout the code for ijq here.

ActiveRecord Models to a Mermaid ERD

2025-01-28T00:00:00+00:00

Turn your ApplicationRecord models into a Mermaid ERD

Rails.application.eager_load!

# Instead of all ApplicationRecord descendants, you can
# make an Array of only the models you care about
data = ApplicationRecord.descendants.each_with_object({}) do |model, data|
  model_name = model.name.upcase
  data[model_name] = {
    columns: model.columns.map { |column| [column.name, column.sql_type] },
    associations: model.reflect_on_all_associations.each_with_object({}) do |reflection, assoc_data|
      assoc_data[reflection.name] = {
        klass: reflection.class_name.upcase,
        relationship: reflection.macro
      }
    end
  }
end

# Intermediate step to save the data as JSON
File.open("data.json", "w") { |file| file.write(data.to_json) }

mermaid_erd = ["erDiagram"]

data.each do |table, details|
  columns = details[:columns].map { |name, type| "#{name} #{type}" }.join("\n  ")
  mermaid_erd << "#{table} {\n  #{columns}\n}"
end

data.each do |table, details|
  details[:associations]&.each do |association_name, association_details|
    other_table = association_details[:klass]
    case association_details[:relationship]
    when :belongs_to
      mermaid_erd << "#{table} ||--o{ #{other_table} : \"#{association_name}\""
    when :has_one
      mermaid_erd << "#{table} ||--|| #{other_table} : \"#{association_name}\""
    when :has_many
      mermaid_erd << "#{table} ||--o{ #{other_table} : \"#{association_name}\""
    end
  end
end

File.open("out.mermaid", "w") { |file| file.puts mermaid_erd.join("\n") }

Adding Journals

2025-01-20T00:00:00+00:00

Update 2026-01-04

After trying this out for a while, I’ve found that I added a new, potentially worse, friction: having to ask myself “…is this worth a blog post or should it just be a ‘journal’ entry?”

What I think I tried to capture was this concept of a “Twitter-like” impermanence to lower the barrier to sharing. But in practice, I’ve found myself hesitating to write anything at all… Now, “journals” are just regular blog posts with a “journal” tag. This keeps things simple and makes my blog feel calmer.

The thing I love most about social media feeds is the immediacy of capturing and sharing thoughts and ideas. Writing on my blog, being built with Jekyll, creates just enough friction for me to overthink what I’m sharing.

Inspired by Julia Evans’s TIL blog implementation, I’m hoping a dedicated page and a rake task removes enough friction for me to share more often.

Low-Poly Image Generation using Evolutionary Algorithms in Ruby

2023-07-30T00:00:00+00:00

Inspired by biological systems, evolutionary algorithms model the patterns of multi-generational evolution in order to unearth unique ideas. They work by generating a vast number of potential solutions to a particular problem and then pitting them against each other in a process akin to natural selection: only the fittest survive. In this way, evolutionary algorithms are able to navigate large ambiguous search spaces in order to find solutions to problems that may be difficult or inefficient to solve using other methods.

These algorithms are used for a wide variety of tasks: from optimizing neural network parameters, evolving mechanical structures, simulating protein folding, and even generating art!

Today, we will use one such algorithm to create a low-poly version of the Ruby logo, using Ruby¹.

Generation: 0000

Max Fitness: 0078.1640

Time (seconds): 0001.03

In this post, we’re going to be using the petri_dish_lab gem for the task of image reconstruction. Image reconstruction is a great example to use because we’ll be able to visualize the evolutionary process working over time.

We’ll begin by going over what an evolutionary algorithm is and how it works. Then, we’ll take a look at the Petri Dish framework and how we can use it to implement an evolutionary algorithm. Next, we’ll define the objective of our image reconstruction problem and how we can represent it in code. Then, we’ll define the genetic operators that we’ll use to evolve the population. Finally, we’ll put it all together and see the results!

🧑‍💻 Code Here
The complete code for this blog post can be found as an example in the examples/low_poly_image_reconstruction directory of the Petri Dish repository.

🔷 What is a Low-Poly Image?
A Low-poly image is a type of 2-D image that’s created by a series of variously sized polygons. Usually, each polygon shares at least one edge with another, and they do not overlap. By using different colors and varying the total polygon count, the result is a low-resolution version of an image.

Evolutionary Algorithm Overview
- The Petri Dish Framework
Understanding the Objective
Member Representation
Population Initialization
Fitness Function
Genetic Operators
Putting it Together
- Final Configuration
Results
- Subjective Analysis
- Data Analysis and Interpretation
  - Log Data
  - Image Data
Conclusion
Footnotes

Evolutionary Algorithm Overview

Evolutionary algorithms are optimization algorithms that “search” through an expansive space of potential solutions. The goal of a search algorithm is to find a solution that satisfies some criteria, or objective. Evolutionary algorithms are particularly good in cases where we want a solution that is “good enough,” rather than an absolute optimum. In our case, we want to find a set of polygon vertices and grayscale values that, when combined, approximate a given image. For this usecase, there’s no one “best” solution, so evolutionary algorithms are a good fit.

The image reconstruction process begins by creating a bunch of images with random polygons. We’re going to use triangles, but we could technically use any shape, and we’ll stick to grayscale colors in order to make the problem a bit simpler. This collection of images is called the population, and each image is called a member of the population.

Once we have a bunch of random guesses, we compare each member to the target image, pixel-by-pixel, to determine their likeness to the target. This measure of likeness is what we refer to as their fitness, and will be determined by how close each pixel is to the correct grayscale value.

Based on their fitness, parent members are then chosen and combined to create a new child member in a process called crossover. The way we select and crossover parent members is by using selection and crossover functions, and we cover these function (along with mutation functions, which mirror the biological process of genetic mutation by injecting random changes into the child member) later. Together, these functions are called the genetic operators. The exact implementation of these operators is perhaps the most important part of developing an evolutionary algorithm, and we’ll spend a lot of time on them later. If you’re familiar with machine learning, you can think of the genetic operators as hyperparameters.

And, just like in biological systems, the process of parent selection, child creation, and mutation repeats and repeats until we have enough new members to fill a new population. If any of the new members meet our objective, we’re done! Otherwise, we’ll repeat this entire process, known as a generation, until we find a member that meets the objective, or we otherwise get close enough.

An evolutionary algorithm configured in this way is called a genetic algorithm. There are other types of evolutionary algorithms, but they all share a similar recursive structure, shown here in Ruby-esque code:

def run(population = [])
  # If population is empty or not provided, create a new population
  population = Array.new(POPULATION_SIZE) { create_random_member } if population.empty?

  # Try to find a member of the population that meets the objective
  best_member = population.find { |member| objective_met?(member) }

  # If a member meeting the objective is found, return it
  return best_member if best_member

  # Otherwise, create a new generation and recursively search again
  new_population = Array.new(POPULATION_SIZE) do
    parents = select_parents(population)
    child = crossover(parents)
    mutate(child)
  end

  run(new_population)
end

🔄 Tail Call Recursion
Notice that the run method above is recursive. Recursion isn’t always the most efficient way to implement an algorithm because each recursive call in Ruby creates a new stack frame, which is a chunk of memory that stores the state of the method. Depending on how many generations we need our algorithm to run, this could cause an SystemStackError (stack level too deep) error. However, when a recursive call is the last thing that happens in a method (a method returns a call to itself), we can avoid allocating new stack frames by using a technique called tail call optimization. In Ruby, this is not enabled by default, but we can enable it using the following compiler options:
RubyVM::InstructionSequence.compile_option = {
  tailcall_optimization: true,
  trace_instruction: false
}
This prevents the stack from growing indefinitely by reusing the same stack frame for each recursive call, rather than creating a new one.²

The Petri Dish Framework

The Petri Dish framework is a Ruby gem that implements the evolutionary algorithm structure for us in the PetriDish::World.run method. We need to supply the members of the population, how to evaluate their fitness, the genetic operators, and when to stop. These specifics of an evolutionary algorithm are highly dependent on the problem we’re trying to solve, but the underlying structure is always the same.

Here’s what a stripped down version of what the PetriDish::World.run method looks like. See if you can spot the similarities to the pseudocode above:

module PetriDish
  class World
    class << self
      attr_accessor :metadata
      attr_reader :configuration, :end_condition_reached

      def run(
        members:,
        configuration: Configuration.new,
        metadata: Metadata.new
      )
        end_condition_reached = false
        max_generation_reached = false

        max_generation_reached = metadata.generation_count >= configuration.max_generations

        new_members = (configuration.population_size).times.map do
          parents = configuration.parents_selection_function.call(members)
          child_member = configuration.crossover_function.call(parents)
          configuration.mutation_function.call(child_member).tap do |mutated_child|
            end_condition_reached = configuration.end_condition_function.call(mutated_child)
          end
        end

        metadata.increment_generation

        unless end_condition_reached || max_generation_reached
          run(members: new_members, configuration: configuration, metadata: metadata)
        end
      end
    end
  end
end

The PetriDish::World.run method accepts a members Array, which contains the evolving population; a configuration object, which holds the user-defined genetic operators and other configuration options; and an internally used metadata object, which holds information about the current state of the algorithm, like the current generation number.

The Petri Dish framework exposes the PetriDish::Configuration#configure method which takes a block of configuration options.

The core configuration options that we’re interested in for this post are:

Parameter	Description
`fitness_function`	A function used to calculate the fitness of an individual
`parents_selection_function`	A function used to select parents for crossover
`crossover_function`	A function used to perform crossover between two parents
`mutation_function`	A function used to mutate the genes of an individual
`mutation_rate`	The chance that a gene will change during mutation
`max_generations`	The maximum number of generations to run the evolution for
`end_condition_function`	A function that determines whether the evolution process should stop premature of `max_generations`

We’ll take a look at each of these functions in detail later, but for now, let’s take a step back and define the objective of our problem.

Understanding the Objective

As mentioned earlier, evolutionary algorithms work best when we define a criteria to aim towards, versus a discrete target. In our case, we are trying to replicate a given grayscale image using triangles. If we break this down in terms of drawing polygons, we want to find a set of triangles that, when drawn together, approximate the given image. That is our objective.

We’ll define the triangles by their three vertices. Each vertex can be encoded as an (x,y) coordinate and a grayscale value. These are our decision variables.

A series of these three-vertices-and-a-grayscale-value groupings is all the information we need create a low-poly image. We’ll call these groupings “points” and an Array of them will represent each members’ genes. Each member of the population will have a distinct set of these points and the algorithm will optimize for a member whose points, or genes, are more fit.

☝️ Why Points instead of Triangles?
As we’ll see later, we’ll be using what’s called Delaunay triangulation to create triangles from a member’s points/genes, rather than define triangles directly. Delaunay Triangulation is a technique that takes a set of points and uses them as vertices to creates a set of non-overlapping triangles. This is a creative decision as much as it is a technical one. I began by evolving triangles directly, but found that I didn’t like the results as much as when I evolved points.

Lastly, we need to define our constraints, or the boundaries of the search space. In our case (mostly due to the arbitrary limit of my laptop’s processing power), we will limit the height and width of the image to 100x100 pixels and the color space to 8-bit grayscale. Independently, we will also limit each members’ genes to be 100 points.

🗺️ Quantifying the Search Space
With each point each having an (x,y) coordinate range of (0..100, 0..100) and a grayscale value range of 0..255, each point can be in one of 101 * 101 * 256 = 2,626,816 possible states (roughly speaking). And, since each member has 100 of these as its genes, this leads to a search space of (101 * 101 * 256)**100 dimensions, which is an enormously large, but in practice, quite small for an evolutionary algorithm.

Now that we’ve taken a look at the objective, decision variables, and constraints at a high level, let’s take a look at how we can represent them in code.

To recap:

Objective: Find a set of non-overlapping triangles that approximate the given image
Decision Variables: Each vertex can be encoded as an (x,y) coordinate and a grayscale value
Constraints: 100x100 pixel image, 8-bit grayscale, 100 points

Member Representation

Evolutionary algorithms are generic in the sense that they can be applied to a wide variety of problems. Therefore, just as we would for a neural network, we must engineer, or encode, our problem into a format that can be understood by the algorithm’s architecture.

We need to encode:

The input target image,
The genes of each member of the population, and
The reconstructed output image.

Input Target Image

For both the input and output images, we’ll use the Magick gem which provides a Ruby interface to the ImageMagick image processing library.

Starting with an input image path, we can read the image into memory, center crop it to a 100x100 pixel square, and then convert it to grayscale. We’ll save the image to a file for later comparison.

require 'rmagick'

def import_target_image(input_path, output_path)
  image = Magick::Image.read(input_path).first

  # Calculate crop size and coordinates for center crop
  crop_size = [image.columns, image.rows].min
  crop_x = (image.columns - crop_size) / 2
  crop_y = (image.rows - crop_size) / 2

  image
    .crop(crop_x, crop_y, crop_size, crop_size)
    .resize(IMAGE_HEIGHT_PX, IMAGE_WIDTH_PX)
    .quantize(256, Magick::GRAYColorspace)
    .write(output_path)

  image
end

Let’s import the Ruby logo and see what it looks like:

IMAGE_WIDTH_PX = 100
IMAGE_HEIGHT_PX = 100

target_image = import_target_image("ruby_logo.png", "ruby_logo_grayscale.png")

100x100 pixel grayscale version of the Ruby logo: our target image.

Using the Magick::Image class to represent our target image is arbitrary, but the rmagick library provides a convenient interface for reading, writing, and importantly, comparing images pixel-by-pixel. (We’ll use this later to calculate the fitness of each member).

Member Genes

When defining of our objective, we identified that the genes of each member can be represented as an Array of 100 points, each with an (x,y) coordinate and a grayscale value that defines the vertices of triangles. We can encode this using a Point Struct with x, y, and grayscale attributes.

Point = Struct.new(:x, :y, :grayscale)

📐 Why a Struct?
A Struct is a simple way to define a class with attributes. It’s a good choice here because we do not need to define any behavior on Point-s; it is a data-only object. Based on my benchmarks³, a hash would be more performant, though it’s less flexible and less explicit. A class or Ruby 3.2 Data object offer no additional benefits.

🌈 Why does a Point have color?
Notice here that a the grayscale attribute is defined on the Point Struct and not on some representation of a triangle. We’ll need a grayscale value to fill each triangle, but a triangle will be made up of three Point-s. So, which of the three Point-s’ grayscale attribute do we use? We will average them. This is because, as we will see, we aren’t sure which three Point-s will be used together as the vertices of any particular triangle. Therefore, for any given triangle, we’ll average the grayscale values of its three Point-s together to determine the fill color. This is only one approach to this problem.

Now that we have a Point, we can create an Array of them to form a member’s genes.

The Petri Dish framework provides a PetriDish::Member class, which acts as an interface to the library. The PetriDish::Member class holds onto the genes Array, which is then directly exposed via the PetriDish::Member#genes method.

Here is the entire definition of the PetriDish::Member class:

module PetriDish
  class Member
    attr_reader :genes

    def initialize(genes:, fitness_function:)
      @fitness_function = fitness_function
      @genes = genes
    end

    def fitness
      @fitness ||= @fitness_function.call(self)
    end
  end
end

The initializer also accepts a fitness_function which is used to evaluate the fitness of a member. (We’ll define the fitness function later, but for now, it only needs to respond to #call and take a PetriDish::Member as its only argument).

To create a PetriDish::Member with an Array of 100 random Point-s as its genes, we do the following:

GRAYSCALE_RANGE = 0..255

def random_member
  genes = Array.new(100) do
    Point.new(rand(0..IMAGE_WIDTH_PX), rand(0..IMAGE_HEIGHT_PX), rand(GRAYSCALE_RANGE))
  end

  PetriDish::Member.new(genes: genes, fitness_function: ->(_member) { })
end

puts random_member.genes.sample(3)
#<struct Point x=57, y=21, grayscale=235>
#<struct Point x=16, y=49, grayscale=64>
#<struct Point x=68, y=6, grayscale=210>

Reconstructed Output Image

To turn a PetriDish::Member into a low-poly image, we’ll create triangles using its #genes in a process called Delaunay triangulation. This technique takes a set of points and creates a set of triangles such that no point is inside the circumcircle of any triangle, i.e. no triangle overlaps another. The delaunator gem is a Ruby library that performs Delaunay triangulation, and we’ll use it here instead of implementing it ourselves.

Let’s define a function called member_to_image that takes a PetriDish::Member and returns an Magick::Image object based on the member’s genes (while also grossly violating the single-responsibility principle).

We begin by initializing a new Magick::Image object, which is the same object type as our input image.
We then perform the Delaunay triangulation using the Delaunator.triangulate method and passing in the x and y values from the Point-s returned from the PetriDish::Member#genes method.
To determine the grayscale fill value, we use the grayscale value from PetriDish::Member#genes, and average them, as we discussed earlier.
Finally, we draw each triangle onto the image using the Magick::Draw#polygon method.

require 'petri_dish'
require 'delaunator'

def member_to_image(member)
  # Create a new image with a white background
  image = Magick::Image.new(IMAGE_WIDTH_PX, IMAGE_HEIGHT_PX) { |options| options.background_color = "white" }

  # Create a new draw object to draw onto the image
  draw = Magick::Draw.new

  # Perform Delaunay triangulation on the points
  #
  # Delaunator.triangulate accepts a nested array of [[x1, y1], [xN, yN]]
  # coordinates and returns an array of triangle vertex indices where each
  # group of three numbers forms a triangle
  triangles = Delaunator.triangulate(member.genes.map { |point| [point.x, point.y] })

  triangles.each_slice(3) do |i, j, k|
    # Get the vertices of the triangle
    triangle_points = member.genes.values_at(i, j, k)

    # Use the average color from all three points as the fill color
    color = triangle_points.map(&:grayscale).sum / 3

    # The `Magick::Draw#fill` method accepts a string representing a color in the form "rgb(r, g, b)"
    draw.fill("rgb(#{color}, #{color}, #{color})")

    # Magick::Image#draw takes an array of vertices in the form [x1, y1,..., xN, yN]
    vertices = triangle_points.map { |point| [point.x, point.y] }
    draw.polygon(*vertices.flatten)
  end

  draw.draw(image)
  image
end

Let’s see what the result looks like if we run it a few times:

5.times do |i|
  member_to_image(random_member).write("member#{i}.png")
end

Five random members of the population.

Here, we can see the Delaunay triangulation in action. Each member has a different set of points, and therefore a different set of triangles. We can also see that the grayscale values are being averaged together to determine the fill color of each triangle.

When we use an Magick::Image object to represent a member of the population, we do so to easily compare it to the Magick::Image target object and visualize the results from the algorithm.

When we use the Petridish::Member object to represent the same member, we do so to enable the algorithm to easily calculate and evolve new members.

This type of data engineering, akin to feature engineering in machine learning, is a critical step in the process of developing an evolutionary algorithm. The way we encode the problem and represent potential solutions has a large impact on the performance of the algorithm and our ability to interpret its output.

⬜️ Why the White Background?
When we create a new Magick::Image object to represent a member in the member_to_image method, we opt for white as the background color. This choice, although arbitrary, influences the performance of the algorithm. For instance, with the grayscale Ruby logo, whose background is largely white, starting with a white background might expedite the convergence on a solution. However, hardcoding this value may bias the algorithm towards reconstructing targets with white backgrounds. The best solution would be to make this choice configurable in order to make this parameter tunable. This is a good example of how the smallest details in the way we model a problem can impact the performance of the algorithm.

Population Initialization

Often it is the case that completely random population initialization is a good place to start, but we can sometimes quickly see faster results if seed the population with some prior knowledge; similar to choosing a good initial set of weights for a neural network.

In the case of image reconstruction, I found that random initialization lead to a lot of waste in the early generations on distributing the points across the image dimensions. To avoid this, we can instead evenly distribute the points across the x and y dimensions of the image and then randomly assign a grayscale value to each point.

def init_member
  PetriDish::Member.new(
    genes: (0..IMAGE_WIDTH_PX).step(10).map do |x|
              (0..IMAGE_HEIGHT_PX).step(10).map do |y|
                Point.new(x, y, GRAYSCALE_VALUES.sample)
              end
            end.flatten,
    fitness_function: ->(_member) { }
  )
end

Let’s see what the result looks like if we run this method a few times:

5.times do |i|
  member_to_image(init_member).write("member#{i}.png")
end

Five non-random members of the population.

We can see that the points are evenly distributed across the image dimensions, and that the grayscale values are randomly assigned. Whether or not this is a better starting point than a completely random initialization is a question of experimentation. In the case of image reconstruction and other generative tasks, a random initialization may be better because it allows the algorithm to explore the search space more creatively, though it may take longer to converge on a solution.

Fitness Function

The fitness function is a function that takes a Member and returns a number that represents how well that member solves the problem. In our case, we want to know how well a member approximates the target image.

Modeling a fitness function to map to the search space is often the most difficult part of developing an evolutionary algorithm. It’s also the most important part because it’s what the algorithm uses to determine which members are better than others. Two key qualities of a fitness function are determinism and discrimination.

Deterministic means that given the same Member, the fitness function should always return the same fitness score. This is because the fitness of a member may be evaluated multiple times during the evolutionary process, and inconsistent results could lead to unpredictable behavior.

Discriminative means that the fitness function should be able to discriminate between different members of the population. That is, members with different genes should have different fitness scores. Although fitness functions do not have to be strictly discriminative, if many members have the same fitness score, the evolutionary algorithm may have a harder time deciding which members are better.

Lucky for us, the rmagick library provides an Image#difference method that fits the bill. Image#difference compares two images and returns three numbers that represent how different they are: mean_erorr_per_pixel, normalized_mean_error, and normalized_maximum_error. We’ll use the normalized_mean_error to calculate our fitness score.⁴

require 'petri_dish'
require 'rmagick'

  def calculate_fitness(target_image)
    ->(member) do
      member_image = member_to_image(member, IMAGE_WIDTH_PX, IMAGE_HEIGHT_PX)
      # Magick::Image#difference returns a tuple of:
      # [mean error per pixel, normalized mean error, normalized maximum error]
      (1.0 / target_image.difference(member_image)[1])**2 # Use normalized mean error
    end
  end

The fitness function is a lambda that takes a Member and returns a fitness score. The number is the inverse of the normalized mean error between the target image and the image generated from the member, squared. This means that the higher the fitness score, the better the member approximates the target image.

🎛️ Why normalized?
The normalized mean error is a number between 0 and 1 that represents the average difference between the grayscale values of each pixel in the target image and the member image. This puts the error of every member on the same scale. When comparing members that all have the same dimension and color scale, normalization may be moot, but sneakily, we can reuse this fitness function for other images that may have different dimensions and color scales and generally compare the performance of the algorithm across those different target images.

🙃 Why the inverse?
When the normalized mean error is inverted (i.e., 1 divided by the normalized mean error), we get a fitness score such that smaller errors (which are closer to 0 after normalization) yield larger fitness values, and larger errors yield smaller fitness values. This means the less “wrong” a member is, the higher its fitness will be.⁵

2️⃣ Why squared?
In effect, this transformation makes our algorithm very sensitive to the quality of the solutions. Solutions that are closer to the target (i.e., have a larger inverted normalized mean error) are assigned significantly higher fitness scores, and are therefore more likely to be selected for reproduction in the next generation (generally true, but depending on the selection function we define later). This can potentially speed up convergence, but as always, we should be cautious of premature convergence on sub-optimal solutions if there is not enough diversity in the population.

Lastly, we define #calculate_fitness as a lambda because the Petri Dish framework requires the fitness_function to respond to #call. The framework will call this lambda via the #fitness method on Member in order to memoize and return the fitness score. We use a closure here so that the target_image is available to the lambda when it’s called.

📫 Closures
Closures are a powerful feature of Ruby and other languages. They allow us to define a function that can be called later, but that also has access to the variables that were in scope when the function was defined. It’s like putting a note inside an envelope for the function to open later. In our case, we want to define a function that can be called later by the Petri Dish framework, but that also has access to the target_image that we defined earlier.

Now that we have a way to represent a member of the population and a way to evaluate the fitness of a member, we can start to evolve the population by defining the evolutionary operators.

Genetic Operators

The genetic operators are the functions that we use to evolve the population generation after generation—they are what allow the algorithm to navigate the vast search space of potential solutions.

The operators we’ll take a look at are grouped by selection, crossover, mutation, and replacement.

Parent Selection Function

Selection is the process of choosing which members of the population are the most fit and therefore should be used as parents to create the next generation of children. There are many different selection strategies, but for our task, we’ll use one called fitness proportionate selection.

Also called roulette wheel selection or stochastic acceptance, fitness proportionate selection works by assigning each member a weighted probability of being selected as a parent, and then randomly selects parents based on those probabilities.

The weighted probability assigned to each member is proportional to that member’s fitness score, which means that members with higher fitness scores are more likely to be selected.

For example, let’s say we have the following Member-s with the following fitness scores:

population = [
  Member.new(genes: [1, 2, 3], fitness_function: ->(_member) { 2 }),
  Member.new(genes: [4, 5, 6], fitness_function: ->(_member) { 3 }),
  Member.new(genes: [7, 8, 9], fitness_function: ->(_member) { 5 }),
]

To calculate the proportional fitness for each member, we divide the member’s fitness by the total fitness of the population. In our case, the total fitness is 2 + 3 + 5 = 10. Dividing each members’ fitness by this number gives us the proportion of the total fitness that that member contributes.

If we calculate a proportional fitness for each member from our example, we get the following results:

weights = population.map { |member| member.fitness / population.sum(&:fitness).to_f }
# => [0.2, 0.3, 0.5]

In our example, if we were to select multiple members using these weighted probabilities, we would expect to get the first member (population[0]) 20% of the time, the second member (population[1]) 30% of the time, and the third member (population[2]) 50% of the time.

We can then use these proportional fitnesses, to weigh, or bias the otherwise equally random probability of each member being selected.

Here’s what that looks like:

  def roulette_wheel_parent_selection_function
    ->(members) do
      population_fitness = members.sum(&:fitness)
      members.max_by(2) do |member|
        weighted_fitness = member.fitness / population_fitness.to_f
        rand**(1.0 / weighted_fitness)
      end
    end
  end

The proportional fitness is calculated by dividing the member’s fitness by the total population fitness, like before. Then, we raise a random number between 0 and 1 to the inverse of the proportional fitness in order to bias the selection towards members with higher fitness scores. The Enumerable#max_by method is used to select two members with the highest result from the block.

The maths are a bit annoying to me personally, but nevertheless, the result of all of this is like spinning a roulette wheel where the size of each slice is proportional to the member’s fitness. The higher the fitness, the larger the slice, and the more likely the member is to be selected.

Notice how the parent selection function makes no mention of Point-s or Magick::Image-s. This is because the selection function is generic and can be used for any problem. It only needs to know how to select members based on their fitness scores.

There are many other selection methods, like simply selecting the fittest members (elite selection) or by first choosing a random subset of the population and then choosing the fittest amongst those (tournament selection).

The benefit of fitness proportional selection is that it allows for a balance between exploration (trying new things) and exploitation (using what we know works). This is because members with lower fitness scores still have a chance of being selected, but members with higher fitness scores are more likely to be selected.

This is the selection method we’ll use for our task, but it may be worth experimenting with other selection methods to see if they work better for our problem.

👨‍👨‍👦 Multiple Parents
Although most implementations of evolutionary algorithms that I’ve seen select two parents, it is possible to select more than two. In fact, the Petri Dish framework allows for any number of parents to be inside the Array returned by the selection function lambda.

Crossover Function

In biology, crossover is when paired chromosomes from each parent swap segments of their DNA. This creates new combinations of genes, leading to genetic diversity in offspring. In evolutionary algorithms, crossover is the process of combining the genes of parent members to create a new child member.

After selecting parents, their genes, an Array of Point-s in our case, are combined to create a new Petridish::Member. There are many different ways to combine the genes of parents, but for our task, we’ll use a method called random midpoint crossover to crossover two parents.

Random midpoint crossover works by randomly selecting a midpoint in the genes of each parent and then combining the genes before and after that midpoint to create a new child. Specifically, we’ll randomly select a midpoint between 0 and the length of the genes Array, and then take the genes before that midpoint from the first parent and the genes after that midpoint from the second parent.

For example, say we have two parents with the following genes:

parent_1 = PetriDish::Member.new(
  genes: [
    Point.new(1, 2, 25),
    Point.new(3, 4, 50),
    Point.new(5, 6, 100),
  ],
  # fitness_function: ...
)
parent_2 = PetriDish::Member.new(
  genes: [
    Point.new(7, 9, 125),
    Point.new(9, 10, 150),
    Point.new(11, 12, 200),
  ],
  # fitness_function: ...
)

If we were to perform random midpoint crossover on these parents, and the midpoint = 1, we would get the following child:

child = PetriDish::Member.new(
  genes: [
    Point.new(1, 2, 25),
    Point.new(9, 10, 50),
    Point.new(11, 12, 200),
  ],
  # fitness_function: ...
)

Here’s our random_midpoint_crossover_function implementation for combining two parents:

  def random_midpoint_crossover_function(configuration)
    ->(parents) do
      midpoint = rand(parents[0].genes.length)
      PetriDish::Member.new(genes: parents[0].genes[0...midpoint] + parents[1].genes[midpoint..], fitness_function: configuration.fitness_function)
    end
  end

Note again, our function returns a lambda, as required by the Petri Dish framework. The lambda takes an Array of parents and returns a new PetriDish::Member with the combined genes of the parents, while also passing along the fitness_function from the configuration object captured by the closure. As before, the fitness function works generically and does not need to know about Point-s or Magick::Image-s.

Other crossover methods include uniform crossover, where each gene is randomly selected from either parent, and majority rule crossover, which works well for instances of selecting more than two parents. It works by having each gene of the offspring determined by a majority vote among the parents. If the parents have equal votes, one parent is chosen randomly to determine the gene.

Random midpoint crossover is a good choice for starting out for its simplicity, but again, it’s worth experimenting with other crossover methods to see if they work better for a particular problem. (In fact, its possible to use one genetic algorithm to optimize the genetic operators of another genetic algorithm, but that’s a topic for another day…).

👩‍🔬 Meta Experimentation
This type of meta experimentation is akin to hyperparameter tuning in machine learning. A core design philosophy of the Petri Dish framework is to make it easy to experiment with different configurations in this way. By passing in different functions for the genetic operators, we can easily experiment with different configurations and observe which ones work best.

That said, for our task, random midpoint crossover works well enough, so we’ll stick with it for now.

Mutation Function

Mutation is the process of randomly changing the genes of a child member after crossover. Combining well-fit parents can get us a long way towards a great solution, but mutation is done to introduce new genetic material into the population.

There are many different ways to mutate a member, but for our task, we’ll use a domain-specific implementation that I call nudge mutation.

🧬 Why mutate at all?
Mutation is not strictly necessary for an evolutionary algorithm to work. However, mutation increases diversity and diversity is what allows the algorithm to explore the search space more thoroughly. Without mutation, the algorithm may get stuck in a local maximum, i.e. be unable to find a better solution, though one may exist. Mutation allows the algorithm to escape this local maximum and continue searching for a better solution.

If we think back to our genes as an Array of Point-s, then nudge mutation is the process of randomly moving each point’s position and grayscale value by a small amount.

This is done by adding a random number between -10 and 10 to the x and y coordinates of each point clamp-ed between the image dimensions. And, adding a random number between -25 and +25 to the grayscale value clamp-ed between 0..255 (i.e. valid grayscale values).

If we implemented this as a lambda for the Petri Dish mutation_function configuration (RBS type Proc[Member, Member]), it might look something like this:

  def nudge_mutation_function(configuration)
    ->(member) do
      mutated_genes = member.genes.dup.map do |gene|
        Point.new(
          (gene.x + rand(-10..10)).clamp(0, IMAGE_WIDTH_PX),
          (gene.y + rand(-10..10)).clamp(0, IMAGE_HEIGHT_PX),
          (gene.grayscale + rand(-25..25)).clamp(GRAYSCALE.min, GRAYSCALE_RANGE.max)
        )
        end
      PetriDish::Member.new(genes: mutated_genes, fitness_function: configuration.fitness_function)
    end
  end

However, we don’t want to mutate every gene of a child; we want to preserve qualities from the parents and strike a balance between exploration and exploitation. Therefore, we use a mutation rate to determine the probability that any particular gene will be mutated.

For example, if the mutation rate is 0.1, then there is a 10% chance that a gene will be mutated.

If we add the concept of a mutation rate to our mutation function, we get this:

def nudge_mutation_function(configuration)
  ->(member) do
    mutated_genes = member.genes.dup.map do |gene|
+      if rand < configuration.mutation_rate
        Point.new(
          (gene.x + rand(-10..10)).clamp(0, IMAGE_WIDTH_PX),
          (gene.y + rand(-10..10)).clamp(0, IMAGE_HEIGHT_PX),
          (gene.grayscale + rand(-25..25)).clamp(GRAYSCALE.min, GRAYSCALE_RANGE.max)
        )
+      else
+        gene
+      end
    end
    PetriDish::Member.new(genes: mutated_genes, fitness_function: configuration.fitness_function)
  end
end

Here, we use the configuration.mutation_rate to determine whether or not to mutate each gene. If the random number is less than the mutation rate, we mutate the gene, otherwise, copy it over to the new PetriDish::Member as-is.

📈 Determining the mutation rate
The mutation rate is a hyperparameter that can be tuned to improve the performance of the algorithm. A higher mutation rate will increase diversity, but it may also cause the algorithm to take longer to converge. A lower mutation rate will decrease diversity, but it may also cause the algorithm to get stuck in a local maximum. The mutation rate is often a good hyperparameter to tune when trying to improve the performance of an evolutionary algorithm.⁶

Other common types of mutation strategies include swap mutation, where two genes’ positions are randomly swapped, and scramble mutation, where a random subset of genes are randomly shuffled. Like all other genetic operators, the best strategy for the problem at hand is highly dependent on the problem itself and is often worth experimenting with.

In our case, the mutation function implementation was developed to represent the idea of nudging around the points of a polygon until the resulting image looks like the target image.

🪲 Point jitter??
You may see that in the actual implementation of this method, I’ve added a point_jitter of rand(-0.0001..0.0001) to each x and y of Point. This is because the particular implementation of the Delaunay algorithm would fall into a divide-by-zero error if all of the points were collinear (on exactly on the same line). This is a good example of how reality is often messier than theory, and how we must adapt our models and implementation to the problem at hand.

Now that we have a way to select parents, combine their genes, and mutate the resulting child, we can start to evolve the population. The last step is to define how we’ll replace the old population with the new population.

Replacement

In the most simple case, we can replace the entire population with the new population, sometimes called generational replacement. This often works well, but it can sometimes cause too much diversity, which can cause the algorithm to take longer to converge as some of the most-fit members are lost. To prevent this, we can use a technique called elitism.

Elitism, which I personally like to call grandparenting, is the process of preserving the fittest members of the population from one generation to the next. This is done by taking the top n members of the population and adding them to the new population. The rest of the new population is filled with the children of the parents.

In the Petri Dish framework, we tune this parameter using the elitism_rate configuration value. This is the proportion of the population that is preserved through elitism. For example, if the elitism rate is 0.1, then the top 10% of the population will be preserved through to the next generation.

Other replacement strategies include steady-state replacement, where only a single member of the population is replaced with a child, and age-based replacement, where the oldest members of the population are replaced with children.

For our image reconstruction task, we’ll use generational replacement with grandparenting set to 0.05, or 5%.

End Condition

Lastly, we need to define when the algorithm should stop. The two most common end conditions are 1) when a member of the population meets the criteria, or 2) when a certain number of generations have passed. We’ll use the latter.

An end_condition_function can otherwise be used to determine if any of the members of the population meets a particular criteria. For example, if we were trying to find a member of the population that had a fitness score of 1.0, we could use an end_condition_function. Alternatively, we could run the algorithm for a given amount of wall time, or until the rate of improvement drops below a certain threshold.

In our case, we’ll use the max_generations configuration value to determine when the algorithm should stop. This is the maximum number of generations that the algorithm will run for, regardless of the fitness of the members of the population. I have found that 5000 generations is enough to get a good approximation of the target image, but not so many that it takes too long to run.

We now have all of the pieces we need to put together our evolutionary algorithm. Let’s see how it all works!

Putting it Together

Before we look at the results, let’s recap everything we’ve gone over.

A genetic algorithm is a type of evolutionary algorithm that uses genetic operators to evolve a population of members towards a solution to a problem.
We start the algorithm design by identifying our objective, defining our decision variables and search space, and defining our constraints.
We then define a way to represent a member of the population and a way to evaluate the fitness of a member.
1. In our case, we represent a member of the population as an Magick::Image object, and we evaluate the fitness of a member by comparing it to the target image using the Magick::Image#difference method.
2. We also represent a member of the population as a PetriDish::Member object with an Array of Point-s as genes, which is used by the Petri Dish framework to evolve the population.
The genetic operators are selection, crossover, mutation, and replacement.
1. Selection is the process of choosing which members of the population are the most fit and therefore should be used as parents to create the next generation of children.
2. Crossover is the process of combining the genes of parent members to create a new child member.
3. Mutation is the process of randomly changing the genes of a child member after crossover.
4. Replacement is the process of replacing the old population with the new population.
The end condition is the criteria that determines when the algorithm should stop.
1. In our case, we’ll use the max_generations configuration value to determine when the algorithm should stop. This is the maximum number of generations that the algorithm will run for, regardless of the fitness of the members of the population.

Final Configuration

The following is the configuration from the actual implementation and contains all of the pieces we’ve discussed so far, with the addition of a few callbacks that the framework provides.

def configuration
  PetriDish::Configuration.configure do |config|
    config.population_size = 50
    config.mutation_rate = 0.05
    config.elitism_rate = 0.05
    config.max_generations = 10_000
    config.fitness_function = calculate_fitness(target_image)
    config.parents_selection_function = roulette_wheel_parent_selection_function
    config.crossover_function = random_midpoint_crossover_function(config)
    config.mutation_function = nudge_mutation_function(config)
    # TODO: Don't pass in image dimensions, use constants instead
    config.highest_fitness_callback = ->(member) { save_image(member_to_image(member, IMAGE_WIDTH_PX, IMAGE_HEIGHT_PX)) }
    config.generation_start_callback = ->(current_generation) { generation_start_callback(current_generation) }
    config.end_condition_function = nil
  end
end

The highest_fitness_callback is invoked when a member with the highest fitness seen so far is found. When that happens, we save the image to a file so that we can see the progress of the algorithm.

The generation_start_callback is invoked at the start of each generation. We use this to keep track of the progress of the algorithm to do things like name the output images.

The last piece of the configuration we haven’t discussed yet is the population_size. This is the number of members in the population per generation, and like all other pieces of configuration, is a hyperparameter that should be tuned to improve the performance of the algorithm. A larger population size can increase diversity, but it may also cause the algorithm to take longer to converge. A smaller population size can decrease diversity, but can be aided with a higher mutation rate and maximum number of generations, as we’ve done here.

Results

Finally, it’s time to run the algorithm and see what happens! Let’s take a subjective look at the results before we get into the numbers.

Generation: 0000

Max Fitness: 0078.1640

Time (seconds): 0001.03

Subjective Analysis

Target image and result

After running the algorithm for over 6000 generations over the course of an hour, I’m really pleased with the results!

I’m surprised by the details the algorithm was able to replicate. For example, the curvature of the top of the ruby is really well defined, given that we’re working with only straight lines. Also, the highlights along the gem’s facets were captured well. If I squint hard enough, I have a hard time distinguishing the target image from the result!

With that said, there are some points stuck in the upper left corner that are shading part of the background the wrong color, but I think it’s those points are actually contributing to the gem’s curvature that I mentioned earlier.

It looks like algorithm had a lot of early success: by generation 120, we can already begin to see vast improvements! However, as the algorithm continued to run, the gains each generation appeared to diminish, which is to be expected from an evolutionary algorithm and an exponential fitness function. After about generation 2000, the algorithm was making smaller and smaller tweaks to the image.

Output of the highest_fitness_callback calls show a progressively more refined low-poly representation

The biggest improvements to the image after the early generations appear to be in the grayscale values, rather than the positions of the points. This makes sense because the position of the points only really matter in that the resulting triangle contributes to the grayscale value of the pixels it covers.

By looking only at the images, it’s hard for me to tell how much the algorithm improved after generation 2000 and whether or not we could have stopped it there.

I’m curious to see if the numbers tell a different story.⁷

Data Analysis and Interpretation

Now that we’ve taken a subjective look at the results, let’s take an objective look at the algorithm’s performance. We’re going to take these numbers with a grain of salt: since we’ve only ran the algorithm once and with one set of configurations, there is nothing to characterize the performance against.⁸ Additionally, we’re going to look at the numbers from the perspective of the algorithm’s performance only, rather than the performance of the Ruby code itself⁹ ¹⁰

Log Data

I, [2023-08-04T19:48:28.744335 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":0,"highest_fitness":0,"elapsed_time":0.0,"last_fitness_increase":0}
I, [2023-08-04T19:48:29.393945 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":0,"highest_fitness":45.43454105775939,"elapsed_time":0.65,"last_fitness_increase":0}
I, [2023-08-04T19:48:29.422413 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":0,"highest_fitness":73.14234456163621,"elapsed_time":0.68,"last_fitness_increase":0}
I, [2023-08-04T19:48:29.702998 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":0,"highest_fitness":75.73263190998411,"elapsed_time":0.96,"last_fitness_increase":0}
I, [2023-08-04T19:48:29.775709 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":0,"highest_fitness":78.16397715764293,"elapsed_time":1.03,"last_fitness_increase":0}
I, [2023-08-04T19:48:29.990462 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":1,"highest_fitness":78.16397715764293,"elapsed_time":1.25,"last_fitness_increase":0}
I, [2023-08-04T19:48:30.146199 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":1,"highest_fitness":85.22354925417734,"elapsed_time":1.4,"last_fitness_increase":1}
I, [2023-08-04T19:48:30.543062 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":2,"highest_fitness":85.22354925417734,"elapsed_time":1.8,"last_fitness_increase":1}
I, [2023-08-04T19:48:30.711047 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":2,"highest_fitness":95.88177285281498,"elapsed_time":1.97,"last_fitness_increase":2}
I, [2023-08-04T19:48:31.090201 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":3,"highest_fitness":95.88177285281498,"elapsed_time":2.35,"last_fitness_increase":2}
I, [2023-08-04T19:48:31.405858 #83090]  INFO -- : {"id":"869b3cc4-b37b-47ff-8beb-602aaa52c48f","generation_count":3,"highest_fitness":100.92915163430119,"elapsed_time":2.66,"last_fitness_increase":3}

Using the logs as a data source, let’s start by looking at a plot of the highest fitness score of any population over time. The fitness was determined by the inverse of the normalized mean error between the target image and the image generated from the member, squared. The higher the fitness, the better the member approximates the target image.

Fitness of the fittest member of the population over time.

We can see that the fitness of the fittest member of each population trends heavily positive over time; somewhat linearly but with a steeper slope at the beginning. This is to be expected because the algorithm starts with a lot to gain from its initial, somewhat random, initial state, and then spends time improving on better and better results.

We can also see that the trend is not monotonically increasing, i.e. it does not always increase from one time step (generation) to the next. This is because the algorithm is not guaranteed to find a better solution in each generation.

We can visualize this by plotting the change in highest fitness over time, which I called velocity, but can also be considered efficiency.

Velocity of the fittest member of the population over time.

Here, we can see a lot of zero-efficiency/zero-velocity generations, where the fitness of the fittest member of the population did not change from the previous generation, specifically after about the 1000th generation. This shows us in another way that the algorithm is converging on a solution.

In fact, we could use this metric to improve our end condition function, i.e. stop the algorithm when the efficiency drops below a certain threshold, rather than blindly after a certain number generations or a particular fitness score. We can also use efficiency to compare the performance of different configurations.

That said, convergence on a solution doesn’t always mean that the solution is the best solution to be found, just that it’s the best so far. We should be careful not to prematurely stop the algorithm. For example, at around generation 4500, we see another significant spike in efficiency even after many hundreds of generations with little-or-no improvement.

Image Data

Subjectively, I thought that there was little to be gained after generation 2000, but based on the data from the logs, we know that the algorithm continued to find fitter members, almost linearly! Is generation 6198, with a fitness score of 2857.08, really 68% better than generation 2194 with a fitness score of 1700.74?

What does “68% better” look like?

Let’s start by taking a look at the error across individual pixels. If we remember back to our fitness function, we know that the fitness score was determined by the mean error of each pixel, i.e. how far off the grayscale value of each pixel was from the target image. We can visualize this by plotting the grayscale value of a few random pixel over time.

Five random pixels' grayscale values over time

Here, we’re plotting the same five pixels from each member as they journey towards their target (indicated by the dashed lines) over each generation. We can observe that these pixels remain relatively stable after just a few hundred generations, and even more so after about 2500 generations. We can also notice that while some pixels hit their target dead on, others get farther away over time.

Interestingly, we can see some pixels correlate to the global trends we saw in the log data earlier: pixel 4 makes a significant improvement around generation 2500, and pixel 1 makes a significant improvement around generation 4500.

10 random neighboring pixels' grayscale values over time moving towards grayscale 46

We see a similar story when we look at 10 pixels near the center of the image that are all moving towards the same grayscale value. Looking at neighboring pixels is interesting because the algorithm is evolving the population by moving the vertices of triangles around, and therefore, the grayscale values of neighboring pixels are likely to be correlated.

Again, we can see that the pixels stabilize after about 250 generations, even more so after around after 2500. And, we also clearly see the correlation to the large spike around generation 4500 that we saw in the log data. It appears that these pixels were somewhat stuck for over 1000 generations, and then suddenly jumped to their target grayscale value. Curiously, we can also see a divergence after 5500 generations, even though the overall fitness score continued to increase…

The differences between the target image and generations 0000, 0120, 2194, and 6198. Black pixels mean they're equal.

If we render the differences between the target image and a sample of generations, we get the images above. In these images, the darker the pixel, the closer the grayscale value of the pixel is to the target image.

This is a good way to visualize our fitness function as the algorithm is essentially trying to increase the number of black pixels and decrease the number of white pixels. We can see that the differences are, in fact, decreasing over time, but it’s still hard to see the 68% difference between generations 2194 and 6198.

The difference between generations 2194 and 6198.

If we look at the difference of the differences between generations 2194 and 6198, we see mostly dark colored pixels, indicating that the differences are small, and perhaps nearly imperceptible.

This is what “68% better” looks like!

Despite all the math and theory, throughout this exploration, we have aimed the power of evolutionary algorithms towards scratching the subjective search space surface of creativity and art. Algorithmic processes can produce results that are not only fascinating from a technical standpoint but also visually (and philosophically) compelling. At the end of analyzing our algorithm’s ability to near its target, we are left to contemplate how well the resulting image captures our aesthetics and creative vision.

Conclusion

Crafting a low-poly image representation is but one example of the myriad applications of evolutionary algorithms. The /examples directory within the Petri Dish repository showcases additional uses, including tackling the Travelling Salesperson Problem and a more straightforward task of text generation.

Despite the diverse applications, evolutionary algorithms generally follow a uniform framework, as discussed in Putting it Together. Interestingly, many of the design considerations we covered in this blog post are even more broadly applicable to the field of data engineering.

If you’re interested in exploring the use of genetic algorithms for creative purposes, I can recommend:

“The Nature of Code” by Daniel Shiffman, particularly the chapters on evolutionary algorithms.
“Generative Art” by Matt Pearson, which discusses the principles of algorithmic art creation.
The Processing community, which is rich with examples of generative art and creative coding.

And of course, feel free to checkout the code for Petri Dish on Github.

Footnotes

For academic study, we often use a language like Python or Julia or R. For production, we use C++ or Rust. But, for fun? For fun, we use Ruby ♥︎ ↩
Danny Guinther’s 2015 blog posts are a fantastic resource for learning about how exactly Ruby implements tail call optimization: Tail Call Optimization in Ruby: Background and Tail Call Optimization in Ruby: Deep Dive. ↩
Hash v. Struct v. Data v. Class Benchmark ↩
The normalized mean quantization error for any single pixel in the image. This distance measure is normalized to a range between 0 and 1. It is independent of the range of red, green, and blue values in the image. ↩
If we don’t invert the error, the algorithm will try to maximize the differences between the member and the target image. In the case of image reconstruction, this will result in an inverted image (black is white and white is black). This isn’t what we’re going for here, but it demonstrates the different ways we can be creative with evolutionary algorithms. ↩
0.1 is somewhat of a high mutation rate, but as we’ll see later, this value was chosen because it worked well for my particular configuration of the problem. ↩
Of course, this is all subjective. I’m sure there are some people who would say that the algorithm should have stopped after generation 1000, and others who would say that it should have continued to run for another 1000 generations. The measure of success in creative applications are arbitrary. That said, I think it’s important to have a way to measure the performance of the algorithm itself, and that’s what we’ll do next. ↩
Hi, Thomas here. I really wanted to finish this blog post. So as much as I want to write up my experience with tweaking the performance and hyperparameters, I’m going to leave that for another day. I hope you understand. ↩
We’ll use generation count as our time (t) instead of wall time or CPU time because it’s a more accurate representation of the algorithm’s performance. The algorithm is not guaranteed to run at a constant speed, so using generation count as our time allows us to compare only the algorithm’s performance of different configurations. ↩
Me again. CPU and memory performance tuning is an important aspect of developing evolutionary algorithms, especially considering that they are known for being resource intensive, but I’m going to leave that for another day as well. ↩

Utilizing Docker for Bundling in a Specific Ruby Environment

2023-07-06T00:00:00+00:00

One of the tasks we often find ourselves completing on Zendesk’s Core Ruby Engineering team is helping to keep a variety of projects up to date. This can involve running bundle update across dozens of repositories, each potentially requiring a different Ruby environment. Although we’ve developed tooling to accomplish this rather efficiently, there are often edge cases that require us to work in each repository locally. However, installing so many different versions of Ruby locally can be cumbersome and time-consuming.

In such scenarios, Ruby Docker images provide a lighter weight solution compared to running a project’s Dockerfile (which can contain dozens of dependencies) or spinning up our Kubernetes-based development tooling (which gives us access to an entire structured environment of all of Zendesk). Individual Ruby images allow us to run commands like bundle update in a specific Ruby environment, such as JRuby, without needing to install that Ruby version locally.

Let’s take an example of running bundle install in a project that requires JRuby. We’ll use JRuby as an example as it’s not always as straightforward to install as CRuby.

First (assuming you have Docker running locally), pull the Docker image from DockerHub for the Ruby version you need.

docker pull jruby:latest

Next, we’ll run the bundle install command within a Docker container using the JRuby image. If your project uses a private gem repository, you might need to pass in credentials. If you have these credentials stored in an environment variable locally, you can pass that to the Docker container. Docker will use the value from the local environment:

docker run --rm --volume "$(pwd)":/usr/src/app --workdir /usr/src/app --env PRIVATE_GEM_REPO_CREDS jruby:latest bundle update my_gem --conservative

Here’s a breakdown of the command:

--rm (-rm for short): This option removes the container after it exits.
--volume "$(pwd)":/usr/src/app (-v for short): This mounts the current directory (assumed to be your project directory) to /usr/src/app in the container.
--workdir /usr/src/app (-w for short): This sets the working directory inside the container.
--env PRIVATE_GEM_REPO_CREDS (-e for short): This passes the PRIVATE_GEM_REPO_CREDS environment variable from your local environment to the Docker container. Replace PRIVATE_GEM_REPO_CREDS with your actual environment variable that holds the credentials for your private gem repository.
jruby:latest: This is the Docker image we’re using.
bundle update my_gem --conservative: This is the command we’re running inside the container. The --conservative flag with instruct Bundler to update as few gems as necessary.

This command runs bundle update in the context of the Docker container, using your stored credentials for authentication and modifying your local directory. Your Gemfile.lock should be modified (if applicable) and any vendor-ized gems will be updated. You can now commit and push the changes.

Docker’s flexibility allows us to run bundle update in any Ruby environment, including but not limited to JRuby, without local installation, while also handling authentication via environment variables. This approach can be applied to similar scenarios where specific environments are required for package installation. It’s a streamlined, efficient alternative to more heavyweight solutions.

A Thought on Practice

2022-02-09T00:00:00+00:00

tl;dr Do a thing you already know how to do.

Our favorite psychologist, Mihaly Csikszentmihalyi, has taught us that Flow maneuvers between anxiety and boredom. I like to say that I err towards “anxiety” because I’m ambitious, I like to learn new things, I like a challenge. But, I like a bit of boredom too! The boredom range is where I find inspiration, curiosity, and discovery.

So, where is the line between the two? When I think of boring work, I think of work that does not require gaining new knowledge or skills, and therefore does not pose a challenge. For a Ruby on Rails engineer like me, this could be setting up a new CRUD (create, read, update, and delete) operation, TDD-ing (test-driven developing) a helper method, or changing a button from blue to red. These tasks, being tasks I already know how to do, ebb towards boring. Therefore, if given these tasks, I might find myself in boredom.

But, do I really know how to do these tasks?

It’s been so long since I’ve done them—and I’ve spent so much time scoffing at them—that figuring out how to edit CSS, writing unit tests for simple functions, and building a new simple Rails app are nearly foreign to me!

rails new leaves me with analysis paralysis, stubbing a method call fills me with doubt, and working with CSS might as well be working with live electrical wires!

This is because I’m out of practice.

I don’t spend time doing the things I think I know how to do because I’m so focused on learning new things—being pulled towards a challenge and away from “boredom.”

The things that I’ve written off as boring are ripe with new potential for growth and discovery.

That old rails new option that I’ve never really looked into is just as exciting as a new Rails feature! Building a simple command-line Ruby program is a brand new experience given how much I’ve learned since my first tic-tac-toe game! Front-end engineering is almost an entirely new discipline compared to when I was first starting out!

Instead of chasing the dopamine rush of buying new toys, I’m cracking open my old toy box with a new appreciation.

Københavns Luftkvalitet og Cykling

2021-08-02T00:00:00+00:00

Dataset retrieved from: Utrecht University & Google, 2021, via Google Environmental Insights Explorer (August 2021)

The Jupyter notebook used to do this analysis is available here as a Binder and here as a Gist for you to explore!

Introduction

I used to be a bicycle taxi driver, and I remember coming home after a day of work covered in soot from riding along the roads all day. It was a gross, yet kind of fascinating, experience to realize that not only was this stuff in the air covering my skin, I was also spending the day breathing it in.

Due to global warming, our consciousness is ever-focused on air pollution; especially within our cities. Many cities are, therefore, attempting to reduce emissions by reducing the amount of cars on the road and increasing the amount of bikes.

This is old news for cities like Copenhagen, Denmark. Their strong bike culture is only surpassed by their comprehensive bike infrastructure. So when The City of Copenhagen collaborated with Utrecht University and Google to map the city’s air pollution, I was eager to get my hands on the data.

Between November 2018 and February 2020, Google strapped three different air quality sensors on their Street View vehicles and took thousands of measurements throughout the city. Today, we’re going to take a look at the data and start to answer the question: what’s the healthiest bike route between A and B, based on air quality?

Data Ingestion & Initial Aggregation

The data can be found on Open Data DK’s website, and we can use the download link there to pull the .geojson file directly into our project.

url = "https://kkkortdata.spatialsuite.dk/airview/CAV_25May2021.geojson"
remote_data = urllib.request.urlopen(url)
raw_data = gpd.read_file(remote_data)

Let’s take a preview of the data.

raw_data.head()

FID	Shape_Leng	ROAD_FID	Mixed_NO2	Mixed_UFP	Mixed_BC	geometry
0	58.821698	1	-999999	-999999	-999999.0	LINESTRING (12.57828 55.69392, 12.57752 55.69423)
1	31.898519	2	10	18500	0.8	LINESTRING (12.62463 55.63461, 12.62473 55.63433)
2	36.276259	3	9	12800	0.8	LINESTRING (12.58604 55.60184, 12.58595 55.602…
3	40.416066	4	9	10700	0.7	LINESTRING (12.58629 55.59317, 12.58569 55.59305)
4	32.734397	5	13	14500	1.1	LINESTRING (12.58135 55.59200, 12.58130 55.59171)

The measurements have the following units:

Black Carbon: μg/m³
Nitrogen Dioxide: μg/m³
Ultrafine Particles: particles/cm³

The Open Data DK site mentions that “some missing data points are listed in datasets as -999999”, so we’ll go ahead and remove any rows in our data that contain this value.

Note that we could change this value to, let’s say, 0, however, 0 will mean something very specific when we look at the distribution later. We could also be sure not to remove the entire row if only one data point is missing, but for my initial analysis, I’m opting to more simply get rid of everything to keep all Series in the DataFrame the same dimension.

cleaned_data = raw_data.drop(raw_data[(raw_data.Mixed_BC == -999999) | (raw_data.Mixed_NO2 == -999999) | (raw_data.Mixed_UFP == -999999)].index)

Now we can look at the correlation between each measurement. I assume there’s a strong correlation: “where there’s high concentration of air pollution of type ‘x’, there’s probably a high concentration of air pollution of type ‘y’”, but it’s always good to check assumptions.

cleaned_data[["Mixed_BC", "Mixed_NO2", "Mixed_UFP"]].corr()

—	Mixed_BC	Mixed_NO2	Mixed_UFP
Mixed_BC	1.000000	0.938192	0.790519
Mixed_NO2	0.938192	1.000000	0.702058
Mixed_UFP	0.790519	0.702058	1.000000

And we can take a look at another statistic that might be interesting, the overall mean:

cleaned_data[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean()

Mixed_NO2       17.572950
Mixed_UFP    14280.222977
Mixed_BC         1.131806
dtype: float64

It’s worth noting that these averages are similar to the ones published on Google’s project site, which gives me a pretty good confidence that I’ve loaded the data correctly. The difference between my results and theirs may be that fact that I likely dropped some data while doing cleanup.

Google also shows the distributions of the amounts of pollutant per road section, so we can go ahead and look at that too.

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 7))

ax[0].title.set_text("Black Carbon")
cleaned_data.Mixed_BC.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[0], legend=True)

ax[1].title.set_text("Nitrogen Dioxide")
cleaned_data.Mixed_NO2.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[1], legend=True)

ax[2].title.set_text("Ultrafine Particles")
cleaned_data.Mixed_UFP.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[2], legend=True)

Now for the interesting bit: let’s take a look at a map of the data. GeoPandas quickly lets you plot a Series or DataFrame containing geometries that it created based on the geojson data we imported.

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 21))

ax[0].title.set_text("Black Carbon")
ax[0].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_BC", ax=ax[0], legend=True, legend_kwds={'label': "Black Carbon", "orientation": "horizontal"}, cmap="YlOrRd")

ax[1].title.set_text("Nitrogen Dioxide")
ax[1].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_NO2", ax=ax[1], legend=True, legend_kwds={'label': "Nitrogen Dioxide", "orientation": "horizontal"}, cmap="YlOrRd")

ax[2].title.set_text("Ultrafine Particles")
ax[2].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_UFP", ax=ax[2], legend=True, legend_kwds={'label': "Ultrafine Particles", "orientation": "horizontal"}, cmap="YlOrRd")

I love these maps! One, I think they’re beautiful (thank you matplotlib!) and two, It’s fascinating that there’s no underlying map of Copenhagen here. The streets represent the data points and only the streets represented in the data are projected onto the plot/map.

Route Querying & Data Joining

Querying for a route is done via the Google Maps API, or more specifically, the Google Maps Python client library. In this example, we’re being sure to use bicycling as the mode so that Google Maps knows to optimize for bicycling routes. It would also be interesting to query for walking routes!

gmaps = googlemaps.Client(key=GOOGLE_DIRECTIONS_API_KEY)
starting_location =  "Øste Gasvæk Teater"
ending_location = "Copenhagen Boulders"

try:
  directions_result = gmaps.directions(starting_location, ending_location, mode="bicycling")
except googlemaps.exceptions.ApiError:
  print("It's likely that Google cannot find one or both locations that you've entered")

if not directions_result:
  print("It's likely that Google cannot find one or both locations that you've entered")

The JSON returned from the API has a ton of information that you might need to display the route to the user: HTML snippets, warnings/cautions, street names, and even landmarks! However, we don’t need any of that. The only thing we’re interested in is the “polyline.” Google Polylines are encoded latitude/longitude pairs. This line encodes the geometries of the entire route. As an example, here is the polyline of the route we’ll look at below:

{x_sIskxkAAuFDKvAG^E?_@@mB?_BB]Lc@T_@POz@]dG{BrJyDnBu@f@Wz@o@fAo@^QdAYxBa@zCoAdGyDjDwBz@g@rAe@dAWjD_@jBOh@B`@FFHNJn@p@nB~CzAdCrBjD~AnClAxBpA~BTp@hApBPTTNLQb@cAZs@v@oBVqANe@p@{A|@kBP_@b@i@|@aAVMd@OpAk@hDsAfCaAjEcB|Ao@nB~ArBdBfCtBlCvBlFbEbDnC|CbCpB~APNNBJFl@b@LFt@C~@U|@e@zAY`AWJI|@W|Ae@DBF?`@KZCJ@DQBILUpAm@PMj@QNEjC{@b@Sd@QjA[n@NVTr@zAzA|DXhARdA`@rDZtCZzAfAbEd@~Ax@~ArAlCbFrKrAnCZl@JZVfBHt@Nb@pEtJbCdFbBvDnBhEdKtYzAlEp@fBTj@B\\Hp@`@hA`DdJx@lCt@tBj@`Br@|Bt@`CnApDDHLB`@BVf@T^b@n@`@b@bAn@h@Rr@Rd@FrAJhDJr@Jp@Vr@f@j@n@r@lAP\\Xt@\\xANxAd@nFZ`Ef@rGb@tGf@`GFf@NvATtDTfDHfAZ~Bl@rC`@tAn@rAj@v@nD`DbAv@xAbA`BvAl@j@rDjDz@v@\\RHDf@D~DcBhD{ArD_BjBy@p@QhBu@vBw@b@M|@g@Lt@r@z@fFvF|@t@jAv@z@`@n@Rp@PZDlAL`A?hAIlB]

We capture this in a variable

pline = directions_result[0]['overview_polyline']['points']

And then use the polyline library to decode it as geojson, or more specifically, as long/lat pairs that can be fed into a GeoPandas DataFrame:

decoded_polyline = polyline.decode(pline, geojson=True)
route_data = pd.DataFrame(decoded_polyline, columns=["x","y"])

route = gpd.GeoDataFrame(route_data, geometry=gpd.points_from_xy(route_data['x'], route_data['y']), crs=cleaned_data.crs)

We can then use the route data to overlay onto our original map. We do this by using a spatial index built by GeoPandas using RTree to quickly query based on a spatial predicate. In this case, we’re looking for all data points that are covered by a box that fits around our route.

route_boundary = shapely.geometry.box(*route.total_bounds).buffer(0.01)
overlap = cleaned_data.iloc[cleaned_data.sindex.query(route_boundary, predicate='covers')]

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(33,21))

ax[0].set_aspect('equal')
ax[0].set_facecolor("xkcd:grey")
cleaned_data.plot(ax=ax[0], column="Mixed_UFP", cmap="YlOrRd")
route.plot(ax=ax[0], marker='o', color='yellow', markersize=50)

ax[1].set_aspect('equal')
ax[1].set_facecolor("xkcd:grey")
overlap.plot(ax=ax[1], column="Mixed_UFP", cmap="YlOrRd")
route.plot(ax=ax[1], marker='o', color='blue', markersize=100)

plt.show()

This is primarily for visual reasons: to see the route placed onto the map, and to also provide a “zoomed in” version. A zoomed in version would be easier with an interactive mapping library, such as Plotly, but there’s no reason we can’t use a little bit of geometric querying to help us!

Next, for what I’m really interested in: which of our air quality data points intersect with our route. Here, I’m buffering the route geometry by a tiny amount to turn it from a 1D point geometry to a 2D polygon. This is because the precision on the longitude and latitude coordinates (and thus the geometry coordinates) is so high, I often found that the points from the route and the lines from the air quality data wouldn’t intersect at all!

intersections = cleaned_data.iloc[cleaned_data.sindex.query_bulk(route.geometry.buffer(0.0001), predicate='intersects')[1]]

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 21))

ax[0].title.set_text("Black Carbon")
ax[0].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_BC", ax=ax[0], legend=True, legend_kwds={'label': "Black Carbon", "orientation": "horizontal"}, cmap="YlOrRd")

ax[1].title.set_text("Nitrogen Dioxide")
ax[1].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_NO2", ax=ax[1], legend=True, legend_kwds={'label': "Nitrogen Dioxide", "orientation": "horizontal"}, cmap="YlOrRd")

ax[2].title.set_text("Ultrafine Particles")
ax[2].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_UFP", ax=ax[2], legend=True, legend_kwds={'label': "Ultrafine Particles", "orientation": "horizontal"}, cmap="YlOrRd")

Now with this data, we can aggregate the air quality data filtered by the route to get some meaningful statistics like how many times more polluted is our route compared to the average:

(intersections[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean() / cleaned_data[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean())

Mixed_NO2    1.637403
Mixed_UFP    1.432781
Mixed_BC     1.518163
dtype: float64

Or what’s the maximum average exposure along the route:

intersections[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].max()

Mixed_NO2       52.0
Mixed_UFP    38500.0
Mixed_BC         3.0
dtype: float64

Next Steps

The Jupyter notebook used to do this analysis is available here as a Binder and here as a Gist for you to explore!

Some interesting ideas:

Exposure amounts based on time. The Google Maps API returns a Polyline and duration for each part of the trip!
Using machine learning to fill in gaps in data, what about those -999999 data points or roads that aren’t represented at all?
Are there correlations between this data and any other data available at https://opendata.dk? Housing prices? Bike parking?

Sargable Queries & MUL Indexes; or Why My Query is Slow

2021-07-09T00:00:00+00:00

You’re on Ops. Debugging Error-s in Invoicing Rails app, as usual. They’re stored in MySQL, and accessed through ActiveRecord and some helper methods. They’ve already been updated with a status_message, so now it’s time to dig in and investigate.

irb(main):001:0> Error.column_names
[
    "id",
    "message",
    "backtrace",
    "status_message",
    "created_at",
    "updated_at",
    "runtime_environment",
    "error_type",
    "ticket_id"
]

The sun beams through your window on another record-high heat wave afternoon, but you’re cool as a cucumber: you brought your sunglasses and a tall glass of trusty H2O.

Let’s do this!

Error.where_message_like("NoMethodError: undfined method `is_approved_by_finance?\\' for nil:NilClass")

…

You check the time… it’s been over 45 seconds.

…

…maybe you shouldn’t have run this in the console??

…

Another 23.7 seconds pass and you decide to CTRL-C your way out of there before you cause a production incident. You decide to just pull the Error IDs from the JIRA ticket like a pleb. Suddenly the heat wave feels even hotter.

But this is nagging you. I mean, what gives?? Why doesn’t this method ever just do the thing it says on the tin? Sure, if we use the .no_status scope (WHERE status_message IS NOT NULL) it seems to work, but why can’t we get simple text search to work on this column? Sure there are a lot (a lot = 8612441) of rows, but I feel like I’ve seen text searching work before. Should we switch to Postgres?

Sargable & Non-sargable

Sargable is a gate keep-y portmanteau (<- also gate keep-y. A portmanteau is a word that smashes together the sounds and meaning of two words, like spoon + fork = spork) for “search argument-able.” We use the term to describe the ability of the database’s SQL optimizer to leverage an index. There are a few things that determine a query’s sargability (not sure if that’s a word), but a big one to look out for is using wildcards (%) when doing text searches.

Let’s look at the implementation of our where_message_like method to see if we might be running into something like this.

irb(main):001:0> ActiveRecord::Base.connection.to_sql(Error.where_message_like("NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass"))
=> "SELECT `errors`.* FROM `errors`  WHERE (message like '%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')"

Aha, so we have a wildcard (%) on either end of string in the WHERE clause:

'%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%'

To understand why this is a problem, let’s pretend you’re the SQL query optimizer. Let’s say that I ask you to find all of the words in the dictionary that contain the substring get:

SELECT * FROM entries WHERE entries.word LIKE '%get%'

You’ll have to scan through every word (row) in the dictionary (entries table) to find all of the instances:

...
- GETaway
- fidGETy
- veGETal
- exeGETe
- nugGETy
...

If, instead, I asked you to get all of the words in the dictionary that begin with get, you could just flip to the correct page and give me all the words! In this way, the dictionary being in alphabetical order is a type of index:

SELECT * FROM entries WHERE entries.word LIKE 'get%'

...
- GETaway
- GETable
- GETting
- GETters
...

“But, Thomas,” I hear you asking, “what if we want to do a substring search rather than search from the beginning of the string?”

That’s a great question! For our Error scenario, that’s seldom the case, but there are more intentional indexes we could use to get us a more robust sub-string search, like MySQL’s FULLTEXT or Postgres’ ts_vector. It’s worth noting that full-text indexes aren’t magic, and your tables and queries have to change to support them.

For our example, we don’t have these things readily at our disposal, so we’ll continue with the assumption that we’re searching for a substring at the beginning of the column’s contents.

`TEXT` & `VARCHAR`

So now, let’s see this sargable business in action by looking at the query plans!

First the non-sargable, double wildcard (%), query:

irb(main):002:0> puts ActiveRecord::Base.connection.explain(Error.where_message_like("NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass").to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (message like '%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 6739497 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Then the sargable, search from the beginning of the contents, query:

irb(main):003:0> puts ActiveRecord::Base.connection.explain(Error.where("message LIKE \'NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass%\'").to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (message LIKE 'NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 6739497 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Oh crap. What gives?! They look exactly the same! Why isn’t MySQL optimizing!? We sarged(?) it!

Unfortunately for us, our message doesn’t have an index. Further, the column is a TEXT column, and (given its extremely varied and undefined length) if we want to add an index, MySQL requires us to specify a prefix length, which tells MySQL how much of the beginning of the string should be indexed. Further further, only the InnoDB tables can be indexed this way. (Our errors table uses the InnoDB engine, so no problem there).

How it looks with an index

Even though our message TEXT column doesn’t have an index, our ticket_id column does:

irb(main):004:0> puts ActiveRecord::Base.connection.explain('errors')
  EXPLAIN (2.6ms)  EXPLAIN errors
+---------------------+--------------+------+-----+---------+----------------+
| Field               | Type         | Null | Key | Default | Extra          |
+---------------------+--------------+------+-----+---------+----------------+
| id                  | int(11)      | NO   | PRI | NULL    | auto_increment |
| message             | text         | YES  |     | NULL    |                |
| backtrace           | text         | YES  |     | NULL    |                |
| status_message      | varchar(255) | YES  | MUL | NULL    |                |
| created_at          | datetime     | YES  |     | NULL    |                |
| updated_at          | datetime     | YES  |     | NULL    |                |
| runtime_environment | text         | YES  |     | NULL    |                |
| error_type          | varchar(255) | YES  | MUL | NULL    |                |
| ticket_id           | varchar(255) | YES  | MUL | NULL    |                |
+---------------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

The ticket_id column is of type varchar(255), which means we have no problem putting an index on there. And that’s exactly what we did! In our case, it uses a MUL (or “Multiple”) key, meaning that its value is used at the beginning of a non-unique key (multiple records can have the same index).

Let’s look at how our knowledge of sargable wildcard queries plays a role with this type of column.

First the non-sargable, double wildcard (%), query:

irb(main):005:0> puts ActiveRecord::Base.connection.explain(Error.where('ticket_id LIKE \'%A00057440\'').to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (ticket_id LIKE '%A00057440%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 8612441 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Then the sargable, search from the beginning of the contents, query:

irb(main):006:0> puts ActiveRecord::Base.connection.explain(Error.where('ticket_id LIKE \'%A00057440\'').to_sql)
  EXPLAIN (1.2ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (ticket_id LIKE 'A00057440%')
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
| id | select_type | table          | type  | possible_keys                        | key                                  | key_len | ref  | rows | Extra                 |
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
|  1 | SIMPLE      | errors         | range | index_errors_on_ticket_id            | index_errors_on_ticket_id            | 768     | NULL |  214 | Using index condition |
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
1 row in set (0.00 sec)

Aha! In the first query plan, MySQL tells us it will scan through ALL 8612441 rows Using where. The second, sargable query plan tells us that it will only need to scan through a range of 214 by Using index condition!

Wow… a single % can make a huge difference.

I’ll leave it as an exercise for the reader to SET profiling = 1; to see the real cost. See documentation here.

Back on Ops

Ahhh, the sun is finally starting to set. We haven’t finished (or really even begun) our investigation, but we’re a little wiser about MySQL’s indexing patterns.

How can we use this knowledge to make our jobs a little easier and beat the heat?

I guess the first thing to know is that where_message_like could take a very long time if there are no other WHERE conditions. This is because it’s a TEXT column with no index; but even if it had an index, using a wildcard at the front of our search string isn’t doing us any favors.

Also, it’s nice to know that the ticket_id column is indexed, and if we want to search, we can use a sargable query to get our results much faster! Having a consistent format for ticket_id might be something to consider; if we can deduce what the beginning of the string is, finding all of the relevant records could be super efficient!

Lastly, if we did want to index the message column, we have a few options:

We could migrate it from TEXT to VARCHAR. The advantage there is gaining access to an easy index, but we give up the benefits of choosing TEXT in the first place.

We could use a column prefix key by specifying a prefix limit. That might get us some good bang for our buck if we use sargable queries.

Or, we could invest in a full-text index and really leverage what InnoDB can do! However, this will require us to alter our table a bit more and adjust our query patterns. Whether or not this plays nice with Rails is another question we’d need to answer.

Nevertheless, it’s 5:00 p.m. (4:56 p.m., but who’s counting), so it’s time to call it a day and grab the last few rays of sun down by the water.

Before you ⌘-Q out of Slack, you remember something. “…didn’t Brianna say that they were going to look at this issue??”

You refresh JIRA:

“Status: Done.”

You give a shout out to Brianna and ⌘-SHIFT-Q.

Rocket Lab Telemetry From Video

2021-05-15T00:00:00+00:00

The Anomaly

Early today, Rocket Lab launched their 20th Electron rocket, Running out of Toes from their Launch Complex 1 Mahia Peninsula in New Zealand.

On one of our toughest days, our team operated with professionalism and worked swiftly to ensure the anomaly was managed safely. Our team is resilient, and our top priority remains to safely and reliably return to flight for our customers. We will learn from this, and we’ll be back on the pad again. —Peter Beck, Rocket Lab founder and chief executive

Unfortunately, the mission ended in failure right around T+00:02:30 shortly after main engine cutoff and second stage separation. In the live stream, the second stage appears to ignite and then quickly cutoff.

Rocket Lab released a statement confirming that the anomaly had lead to the failure of the mission, but not yet releasing any preliminary data that might point to the cause of the failure, of course, pending the investigation.

Days like today are a magnet for armchair engineers…

The Data

This is a single frame from the live stream right at T-00:00:00. On the right hand side, you can see that they helpfully show some telemetry: flight time, speed, and altitude, along with the video feed itself, and some stages or “gates” on the left hand side.

When wanting to figure out what happened, besides the video, this was all the data we in the peanut gallery had.

So, I did what any sane person would do and started going frame by frame to record the telemetry by hand in a spreadsheet. After I realized that the area of interest was about three minutes and thirty second long, and the stream was at 30 frames/second, I was looking at transcribing thousands of frames! Given that the data wasn’t really updating every single frame, the number was likely to be a lot lower, but still… The band of DIY rocket scientists were waiting; this couldn’t take all day!!

I pulled the video down using youtube-dl just in case the video was taken down from Youtube. Rocket Lab isn’t known to do this, but SpaceX, for example, de-lists their videos and they can be difficult to find.

$ youtube-dl https://www.youtube.com/watch?v=Zw3sIUyfSfc

Google Video Intelligence API

Luckily, this is the exact kind of job that neural networks are suited for. In fact, the first algorithms you learn about are ones used for handwritten digit recognition. These weren’t even handwritten, so the job could end up being quite easy! One challenge was that we’re dealing with thousands of frames, not just a single image. My personal toolkit doesn’t support analyzing video, so I reached for Google Cloud Video Intelligence API

This service uses OCR (Optical Character Recognition) to detect and extract text from video. Perfect!

After configuring and deploying the resources in my Google Cloud account. I uploaded the video to Google Cloud Storage and called the REST API

// request.json
{
  "inputUri": "gs://thomascountz/rocket-lab-20-launch-anomaly.mp4",
  "features": ["TEXT_DETECTION"],
  "videoContext": {
    "textDetectionConfig": {
      "languageHints": ["en-US"]
    }
  }
}

curl -X POST \
  -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
  -H "Content-Type: application/json; charset=utf-8" \
  -d @request.json \
  https://videointelligence.googleapis.com/v1/videos:annotate

Data Cleanup/Presentation

What came back was exciting and a little unexpected. I guess I didn’t know what to expect (it’s not like I had read any documentation or anything).

Firstly, the API responds with a job name that then you can later use to query for the results and other metadata. I used that name to send another API request and I got back results almost instantly. The entire job took maybe 2 second to complete and I got something back like this:

{
  "name": "projects/REDACTED/locations/us-west1/operations/REDACTED",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "progressPercent": 100,
        "startTime": "2021-05-15T15:39:55.442511Z",
        "updateTime": "2021-05-15T15:41:17.361940Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "segment": {
          "startTimeOffset": "0s",
          "endTimeOffset": "218.100s"
        },
        "textAnnotations": [
          {
            "text": "2612",
            "segments": [
              {
                "segment": {
                  "startTimeOffset": "98.400s",
                  "endTimeOffset": "98.500s"
                },
                "confidence": 1,
                "frames": [
                  {
                    "rotatedBoundingBox": {
                      "vertices": [
                        {
                          "x": 0.7729167,
                          "y": 0.05277778
//... snip...

That’s a heavily truncated version, but the things I noticed were:

The result was so big, I was having trouble exploring it all at once.
The data was not in order by frame count the way a video is. This makes sense since OCR acts on a single image and Google Cloud most likely parallelized the work for each image before collecting it into a single result (fan-out/fan-in).
There wasn’t an obvious way to tell which frames contained the data I cared about (time, speed, altitude) and for each frame, it wasn’t entirely clear which bit of text mapped to which data point (altitude and speed are both numbers).

1. The results were big

Ha! 1116879 lines of unminified JSON never hurt anyone! Although my computer choked on opening the response in something like vim, I used jq to begin to explore the data.

After getting an understanding for the shape, I actually “trimmed” of the pieces of data I didn’t care about by mapping through each textAnnotation and only keeping the text found and the start and stop timestamps. (This ended up making my life more difficult… More on this later.)

jq '.response.annotationResults[0].textAnnotations[] |= { text: .text, start: .segments[0].segment.startTimeOffset, end: .segments[0].segment.endTimeOffset }' response.json > response_trim.json

This produced something a little more manageable:

{
  "name": "projects/REDACTED/locations/us-west1/operations/REDACTED",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "progressPercent": 100,
        "startTime": "2021-05-15T15:39:55.442511Z",
        "updateTime": "2021-05-15T15:41:17.361940Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "inputUri": "/rocket_lab_001/end-to-end-flight.mp4",
        "segment": {
          "startTimeOffset": "0s",
          "endTimeOffset": "218.100s"
        },
        "textAnnotations": [
          {
            "text": "2612",
            "start": "98.400s",
            "end": "98.500s"
          },
          {
            "text": "3467",
            "start": "110.300s",
            "end": "110.300s"
          },
          {
            "text": "1362",
            "start": "73s",
            "end": "73.100s"
          },
          {
            "text": "2392",
            "start": "94.800s",
            "end": "94.900s"
          },
          {
            "text": "2079",
            "start": "89.300s",
            "end": "89.300s"
          },
// ... snip ...

2. Data out of order

With the timestamps in place, the data being out of order was no longer an issue. I easily parsed the start value of each object into a float, and then sorted in ascending order.

Note that this was the time in seconds based on the video I uploaded to cloud storage which was only about three minutes of the hour+ Youtube stream.

3. Knowing what the text means for each timestamp

Now that things were sorted (pun intended), I had to then understand what each of the text values represented. For this, I moved to Ruby. (I told you that this was brute force data munging and worst practices. I hope you didn’t come here thinking I knew what I was doing).

In Ruby, I read in the trimmed file as a hash, removed the metadata, converted everything to floats, integers, and symbols, and grouped by the start timestamp.

Yes, there can be, and were, many annotations that started at the same time. Not all of the data on the video was updated in sync, especially because the flight time was only updated once per second, but the telemetry data was updated more often. (Thanks for the high fidelity Rocket Lab!).

I ended up with a hash that looked like this:

{
  # ...snip...
  205.6=>
   [{:text=>"7687", :start=>205.6, :end=>205.6},
    {:text=>"T+// OO:03:18", :start=>205.6, :end=>206.5}],
  205.7=>[{:text=>"7661", :start=>205.7, :end=>206.1}],
  206.3=>[{:text=>"7677", :start=>206.3, :end=>206.3}],
  206.4=>
   [{:text=>"110.0", :start=>206.4, :end=>207.8},
    {:text=>"7689", :start=>206.4, :end=>206.7}],
  206.6=>[{:text=>"T+// OO:03:19", :start=>206.6, :end=>207.5}],
  206.8=>[{:text=>"7676", :start=>206.8, :end=>206.8}],
  207.0=>[{:text=>"7656", :start=>207.0, :end=>207.0}],
  207.1=>[{:text=>"7647", :start=>207.1, :end=>207.2}],
  207.3=>[{:text=>"7649", :start=>207.3, :end=>207.3}],
  207.4=>[{:text=>"7657", :start=>207.4, :end=>207.4}],
  207.5=>[{:text=>"7669", :start=>207.5, :end=>207.5}]
  # ...snip...
}

Just taking a look at the :text values, we can see some integers, floats, and a string that starts with T+// ; this formatting happens to map to the speed, altitude, and flight time, respectively.

Remember earlier when I used jq to delete a bunch of data that I thought I didn’t need? It turns out that that data contained the bounding boxes for each annotation, that is for each piece of text found, the results give you four Cartesian coordinates that correspond to the x and y pixel locations of four corners of a square that describe where in the frame the text was found. In other words, instead of relying on each data point coincidentally having a different format, I could have used the bounding boxes to query for each data point in a given frame. (Worst. Practices. I literally warned you about this…).

I ran the hash through an each block that will make any programmer cry and ended up with four values for each frame: frame, which represents the frame in seconds, time, or flight time, speed and altitude.

result = []
data.each do |frame_data|
  f = frame_data[0]
  t = frame_data[1].find { |v| v[:text].include?("//") }
  t = t[:text] if t
  s = frame_data[1].find { |v| v[:text].to_i.to_s == v[:text] }
  s = s[:text] if s
  a = frame_data[1].find { |v| v[:text].include?(".") }
  a = a[:text] if a
  result << { frame: f, time: t, speed: s, altitude: a }
end

Looking at this feels embarrassing, but you know what? It worked!

Here’s the same slice of data from above:

[
  # ...snip...
    {:frame=>205.6, :time=>"T+// OO:03:18", :speed=>"7687", :altitude=>nil},
    {:frame=>205.7, :time=>nil, :speed=>"7661", :altitude=>nil},
    {:frame=>206.3, :time=>nil, :speed=>"7677", :altitude=>nil},
    {:frame=>206.4, :time=>nil, :speed=>"7689", :altitude=>"110.0"},
    {:frame=>206.6, :time=>"T+// OO:03:19", :speed=>nil, :altitude=>nil},
    {:frame=>206.8, :time=>nil, :speed=>"7676", :altitude=>nil},
    {:frame=>207.0, :time=>nil, :speed=>"7656", :altitude=>nil},
    {:frame=>207.1, :time=>nil, :speed=>"7647", :altitude=>nil},
    {:frame=>207.3, :time=>nil, :speed=>"7649", :altitude=>nil},
    {:frame=>207.4, :time=>nil, :speed=>"7657", :altitude=>nil},
    {:frame=>207.5, :time=>nil, :speed=>"7669", :altitude=>nil}
  # ...snip...
]

Now, all I had to do was clean up the T+// from the timestamp and ship this off to a CSV!

The Results

You can view the entire dataset on Google Sheets here.

This is kind of where my armchair engineering stopped. I mean, I don’t really know much about rockets, I just love data! (I also love rockets, but I just don’t know enough to make guesses as to what happened). I shared this data with my rocket-knowing friends to see if they could make anything of it.

What I do know is that things shouldn’t have ended around T+00:03:30 and that the green line (velocity) should have kept going up, but around T+00:02:30, right around main engine cut off and stage separation, it started going down and stayed down.

Conclusion

This was a fun project from a bad day. It really illustrates to me the power of motivation. This isn’t too dissimilar to my day job, but at work, I can’t just spin up cloud resources and the scale of the data is too large to try to manipulate in memory. That said, it was really fun to try to dive head first into a problem and come out on the other side having learned more about Google Cloud, jq, rocket trajectories, and basic data manipulation.

If I were to approach this problem again, I would keep the rotatedBoundingBox data and use it to identify the meaning of the data. SpaceX’s Falcon 9 streams, for example, contain two readouts each of speed and altitude, one for the first and second stages. It would be impossible to distinguish between the two on format alone.

Secondly, I’d get the data in SQL database right away (or at least store it somewhere where I can use a SQL-like API). Maybe I should up my data game, but my brain likes to think in SQL for problems like these; having to think procedurally in a loop was unintuitive for me.

Thanks for reading! Let me know what you think or if you have any questions. Would you be interested in having raw telemetry data formatted and presented after a launch stream? I’m certainly impressed by the quality of the data coming through! See if you can spot MAX-Q in the velocity graph!

Hexo - Schematic Capture, Mechanical Design, & PCB Layout

2021-05-13T00:00:00+00:00

Pictured above is the schematic for Hexo’s circuit. Like I mentioned earlier, reading and interpreting a schematic is a creative process akin to jazz… although in this case it is highly precise. Not only is this a graphical representation meant to aid human designers, each component and connection is backed by a computer model which can do things like check for output pin conflicts and missed connections, in a process called electrical rules checking. I use a free open source tool called KiCAD EDA (EDA: electrical design automation) for this work.

Schematic capture is about the process of encoding a circuit into an EDA. KiCAD has a very large library of components, but also has tools for building new components, cataloging datasheets (technical document and specification produced by component manufacturers), and exporting files to use in SPICE simulations.

After schematic capture, I moved onto mechanical design. All I knew was that I wanted Hexo to be worn as a pendant on a necklace. In electronics projects, mechanical design is concerned primarily with the shape of the PCB as it’s often mounted without an enclosure or case. For Hexo, the PCB itself would be exposed as the final form factor and I was inspired by game pieces from one of my husband and my (mine?) favorite games, Eclipse, pictured below. Eclipse is a sci-fi based game and besides the size and shape of its pieces, I was inspired by the futuristic and mysterious vibe of the game’s design.

The mechanical design was constrained primarily by the size of the battery I planned to use (or rather the size of the mechanical clip that holds the battery) and the user experience of pressing a small button and reading the numbers. The important constraints limited how small Hexo could be because I wanted Hexo to be as small as possible (because I thought it would look cooler).

Other than that, the form factor was left to my aesthetic sense and what was feasibly manufacturable (these constraints were helpfully available on the manufacturer’s website).

I went with a 30mm x 30mm square that had three corners cut away at 45º angles because… I thought it would look cooler (sensing a theme?) and because the final surface area was within the tolerances of the manufacturer’s cheapest options. (Definitely cooler).

Finally, the PCB Layout, or the process by which you plan where to mount each component on the final board, was all that was left.

During prototyping, we used the THT (through-hole technology) components that had leads that allow them to “plug-into” a breadboard. For PCBs, we can also use SMT (surface mount technology) components that, instead of having leads that go into a hole, they have pins that are soldered to pads.

These pads are where the electrical connection between the component pins and the rest of the circuit is made. Taken together, a group of pads make up a component’s footprint or the overall shape and dimensions of the electrical connections between the component and the PCB.

In the image above, you can see the various component footprints on the bare Hexo PCBs before they were assembled.

The battery clip datasheet, pictured above, shows the only footprint I had to design by hand in CAD for Hexo (the other components used standard footprints that were pre-designed in KiCAD). The footprint is a relatively simple three square design, but it was a lot of fun to get practice at taking specs and turning them into something that would leave the page.

You can see the battery clip’s footprint as the three green squares in the layout diagram below. Note that the diagram is showing both the front and the back of Hexo at the same time and that the battery clip was mounted on the back.

The final large technical constraint, after the board shape and the component footprints, is the management of electrical traces, or the copper “wires” that run (unseen) through the board and connect everything together.

Traces are the things that connect the LED to the battery and the battery to the switch, for example. The way this is done has to do with the manufacturing process, which is step 4, so I can’t go into it here, but I will leave you this video link if you’re interested.

There’s an art to PCB layout and there are tools to help make it easier. Like a lot of design, there’s a mix of technical and aesthetic attitudes that lead to a PCB being designed one way or another. When you begin the PCB layout process, you have, what is literally called, a rats nest of traces and components that need to be untangled, taking into account electrical requirements, interference, manufacturability, etc.

Now that Hexo was designed, and (nearly) every decision that could be made, was made, it was time to send the board to the manufacturer. After which, you’ll realize that you made a tiny mistake and hope that either the manufacturer will notice and fix it, or that it was tiny enough to have no impact.

Hexo - Prototyping, Design, Simulation

2021-05-13T00:00:00+00:00

Pictured above is a printed circuit board or PCB. Dubbed Hexo, a play on the prefix hexa-, meaning having six, it’s an electrically random “six-sided” die. I like to call it “electrically random,” because that sounds cool, and because the LED which lights up at the end is an output of a function whose inputs are voltages and time. When you push the button, Hexo cycles through its six LED, slower and slower, until eventually leaving just one LED on.

I built Hexo because I wanted to learn how I should go about bringing something into existence… something inanimate, that is. What I’m left with is a kind of systematic process that I hope will allow me to build bigger and more ambitious projects in the future!

This is what I’ve discovered so far; electronics projects like this can be broken down into the following steps:

Requirements Gathering
Prototyping, Analysis, & Simulation
Schematic Capture, Mechanical Design, & PCB Layout
Manufacture & Parts Ordering
Assembly
Validation, Testing, & Integration

In this post, I want to share with you my experience with prototyping, design, and simulation, the steps centered around implementing functional design. So grab a cup of tea or coffee and let’s geek out!

Prototyping, Analysis, & Simulation

I didn’t design Hexo’s primary circuitry; the design came from 555 Timer Circuits’ Roulette schematic and the many derivatives thereof. With a few very minor modifications, I was able to turn their 9-volt, 10 LED design into a 3-volt, 6 LED design with a similar effect. To read more about that circuit and how it works, you can read my writeup on Hackaday here.

With that said, in order for me to understand how to modify the circuit and to measure its performance, I had to build prototypes, do some pen-and-paper analysis, and (although perhaps a bit overkill), employ the use of circuit simulation tools. These things all happened in a happy nonlinear feedback loop during multiple iterations.

For prototyping, I use these things called breadboards (also known as solderless breadboards, it got its name from the old practice of using nails hammered into a cutting board to wire up temporary circuits), pictured below. A breadboard allows you to quickly “plug in” a bunch of through-hole (or THT, through-hole technology) components and test out designs.

Below, you can see a few THT components next to their equivalent SMT (surface mount technology) counterparts, which are often used on PCBs because of their small size. You can probably guess why it’s easier to prototype with through-hole components, though they aren’t only used for prototyping!

Prototyping in this way allows you to take the circuit off of the page and place it in your hands in order to answer questions like: “does this even work?”, and “is it supposed work like that?”, and “ugh… why isn’t this working?”, usually followed by, “oh wow, how am I such an idiot?” (I put in the LED backwards… again.)

Having a small country’s worth of components sitting around is the only way I’ve seen electronics hobbyists get through prototyping. (That’s why a lot of us join a hackerspace, a physical place where geeks unite and stockpile their trove of parts and tools). Before I set off on my electronics journey, I invested quite a bit in building out a workbench and I have more than enough to share!

If prototyping is taking a circuit off of the page, then circuit analysis is… writing more stuff on the page? Analysis allows you to understand the mathematical theory behind the circuit. Equations derived from the Laws of Ohm and Kirchhoff allow us to understand idealized circuits, that is, how the design would function in a vacuum without interference, resistive wires, and perfect electron flow. Combined with higher-level patterns, one can begin to read circuit diagrams, called schematics, like lead sheets and fake books in the world of jazz or design patterns in the world of code writing.

Having not formally studied electrical engineering, this was one of the more exciting parts of this build. I love seeing theory on the page become measurable and observable in real life. This is the part where we use math to affect decisions about design. It turned out that even though I was switching from using a 9-volt battery to a 3-volt battery, I didn’t need to change any of the other components in the circuit.

Lastly, simulation. Using CAD (computer aided design) software allowed me to more easily and efficiently do circuit analysis; particularly with circuits that contained integrated circuits, or the little black things that look like microchips.

This is where I learned about SPICE: Simulation Program with Integrated Circuit Emphasis, and its variants like LTspice, pictured above.

SPICE uses, what I’ve only seen described as, an ASCII-based declarative language to define component models, netlists (how components are connected), and initial condition directives to calculate the electrical state of a given circuit. In Hexo’s case, I used a transient analysis, which allowed me to observe the circuit’s behavior over time. This was particularly useful given that the LEDs would blink, and “blinking” is a function of time.

Pictured above, the top half of the image shows the state of each of the ten LEDs, in that particular Hexo iteration, plotted over time. Taken as a whole, you can see how the rate of flashing (defined by the period and duty cycle of the square wave) begins to slow or stretch out over time as the LEDs blink and cycle slower and slower, before ending with only one remaining lit.

I’m really excited about SPICE. I’m not sure where it sits in the professional electronics industry, but there’s not a lot of documentation for us mere hobbyists. I hope that we, as a maker community, can work together to change that.

A more intuitive type of circuit simulation can be done using the Falstad Circuit Simulator.

In fact, if you want to play with Hexo, you can take a look at the simulation with Falstad’s free browser-based app here. Although Falstad’s models are less realistic in a few ways, the drag-and-drop interface, the current flow visualizations, and real-time interactivity helped me build up a more intuitive understanding of Hexo’s design.

From bench, to paper, to screen. Back and forth, I’ve iterated and studied Hexo’s design until I felt confident that the circuit would work after I tweaked it a bit. I imagine a similar, if not more rigorous, process would exist for building an original design as well.

Now that we’ve experimented with prototypes and understood the theory, it’s time to design the physical thing!

Communication Superhighways

2021-03-13T00:00:00+00:00

Good communication in an emergency is vital: What is happening? Who is leading? Where are we meeting? Et al. Just like firefighters, who use highways instead of local roads to get to a burning building more quickly, our teams should leverage communication highways to respond efficiently in the face of an emergency.

This communication infrastructure isn’t just logistical: you may have a dedicated chat channel, video conference URL, or a paging system already. Designing communication superhighways is also about investing in our workplace environment and ultimately in each other.

Cognition and emotion are tightly intertwined, which means that [we] must design with both in mind. (Norman, 2013).

How we feel about our work with one another will greatly influence our ability to work together when things are difficult—if I don’t feel comfortable or confident communicating about our work when there isn’t an emergency, how do you think we’ll get along when there is one?

In practice, this looks like investing in maintaining a healthy workplace environment, team collaboration, transparency, accountability, and responsible incentives. Going into each of these is beyond the scope of this essay—instead, I’ll leave you with this: ethically, we should all be working towards keeping our team dynamic positive and healthy, but capitalistic incentives being what they are, we could all do well to remember that those team outings really do help your bottom line.

Norman, D. A. (2013). The design of everyday things. MIT Press.

Crafting an Invitation to Think

2021-03-08T00:00:00+00:00

Foreshadowing: signaling an outcome of chapter eight in the middle of chapter one, i.e. the original spoiler. Although, akin to the mean-spirited handiwork of a Reddit troll, wielded gracefully, foreshadowing helps an author build tension, introduce indirection, and drive home a butterfly-inducing denouement.

In non-fiction, we are afforded a similar device: we can build a compelling narrative or stack ever increasing layers of complexity onto a topic to help elevate our reader to a plane of inevitability. At this point, we want our readers to say, “Ah! Yes! This is so obvious!”

If we over-articulate the point, however, our audience disengages; like continually begging a friend to show up to your (virtual) party after they’ve already said yes. We want to posit an idea just enough to engage a reader’s curiosity and compel them to have thoughts of their own—the aim is to craft an invitation to think.

Repeated Signals Wear Grooves

2021-03-01T00:00:00+00:00

In The Design of Everyday Things, Don Norman shares an anecdote about “signifiers,” which are the design elements that communicate the purpose, structure, and operation of an object or system. The anecdote was about a wall onto which someone placed an empty bottle. Over time, more and more empty bottles showed up on this wall:

…as soon as one person discovers it can be used to dispose of empty drink containers, the discarded container becomes a signifier, telling others that it is permissible to discard their items there. (Norman, 2013).

This is a powerful message about the implicit, and often unconscious, signals we send one another on our teams.

For example, the boss may say “don’t feel like you have to work late,” but if they and other members of your team do, the implicit signal soon becomes louder than words.

This doesn’t always have to be negative, exhibiting healthy team customs, norms, and rituals also has the happy effect of instilling positive behaviors, too. If everyone is always on time to meetings, it can soon become a happy groove worn into the fabric of the team.

These grooves are worn in over time for better and for worse, however they aren’t a drop-in replacement for explicit team structures and boundaries. Signals help to reinforce healthy team habits and deter unhealthy ones, but they shouldn’t be relied upon to establish them.

Norman, D. A. (2013). The design of everyday things. MIT Press.

Arduino Tone for ESP32

2021-02-21T00:00:00+00:00

Arduino has a built-in tone() library which allows you to send a PWM frequency at 50% duty cycle to a specific pin in order to generate a tone on a piezoelectric buzzer with an optional duration.

tone(pin, frequency)
tone(pin, frequency, duration)

This functionality is famously unavailable in Espressif’s arduino-esp32 library and members of the community have found various work-arounds such as using the native LED Control functions to generate PWM signals.

However, looking more closely at arduino-esp32 library, not only has Espressif provided a clean API for generating tones, they’ve provided an interface for generating specific PWM frequencies for specific notes on the chromatic scale in different octaves.

double ledcWriteNote(uint8_t chan, note_t note, uint8_t octave){
  const uint16_t noteFrequencyBase[12] = {
  //   C        C#       D        Eb       E        F       F#        G       G#        A       Bb        B
      4186,    4435,    4699,    4978,    5274,    5588,    5920,    6272,    6645,    7040,    7459,    7902
  };

  if(octave > 8 || note >= NOTE_MAX){
      return 0;
  }
  double noteFreq =  (double)noteFrequencyBase[note] / (double)(1 << (8-octave));
  return ledcWriteTone(chan, noteFreq);
}

Although not directly compatible with Arduino’s tone(), the function provides a dedicated interface for producing named frequencies out of the box via the note_t type.

typedef enum {
    NOTE_C, NOTE_Cs, NOTE_D, NOTE_Eb, NOTE_E, NOTE_F, NOTE_Fs, NOTE_G, NOTE_Gs, NOTE_A, NOTE_Bb, NOTE_B, NOTE_MAX
} note_t;

Example Usage

I’ll take an example from @lbernstone’s Tone32 library. Their library is a great solution for providing cross-compatibility between code written for an Arduino and code written for an ESP32.

void loop() {
  tone(BUZZER_PIN, NOTE_C4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_D4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_E4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_F4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_G4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_A4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_B4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
}

In this example, a major C scale is played while holding each note for 0.5 seconds. The BUZZER_CHANNEL argument is optional. In the case of an ESP32, there are 16 PWM channels which can generate independent waveforms which need to be explicitly assigned to any PWM-capable pin.

Here is the equivalent scale programmed using ledcWriteNote().

void loop() {
  ledcAttachPin(BUZZER_PIN, BUZZER_CHANNEL);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_C, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_D, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_E, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_F, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_G, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_A, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_B, 4);
  delay(500);
  ledcDetachPin(pin)
}

The tradeoff here is that this is not cross-compatible with an Arduino, however, it means not having to import an external library or having to define note frequencies yourself. Similar to Arduino’s tone(), each signal is produced at 50% duty cycle. Under the hood, both implementations are calling ledcWriteTone.

References

https://randomnerdtutorials.com/esp32-pwm-arduino-ide/
https://github.com/espressif/arduino-esp32/issues/1720
https://community.platformio.org/t/tone-not-working-on-espressif32-platform/7587

SQL Injection Overview

2020-10-24T00:00:00+00:00

An SQL Injection occurs when untrusted input is used directly in the construction of an SQL query. This attack is commonly executed by introducing a meta character (such as a comment) into a data plane in such a way that allows an attacker to add commands to the control plane. Essentially, when building a SQL query from user input, an attacker can insert SQL instructions that cause the application to behave in unintended ways.

SQL Injection flaws are introduced when software developers create dynamic database queries that include user supplied input.

—OWASP SQL Injection Prevention Cheat Sheet

Authentication Form Example

Let’s take a login form that requires a user enter a username and password to log in.

In the happy-path case, a user enters in their details, presses “Login,” and a request is sent to a server where we take what the user enters, search the database for that user, and then log that user into our web application.

# SQL Injection Vulnerable Pseudocode
# -- snip --

let user = ORM.query(
  "SELECT * 
    FROM users 
    WHERE username = '#{params['username']}'
    AND password = '#{params['password']}';"
).execute()[0];

# -- snip -- 

log_in_user(user)

Using a tool like SQL Fiddle, you can see how a query like this will behave in PostgreSQL 9.6. Here’s is the schema and seed data for our example:

CREATE TABLE users(
   id        INT PRIMARY KEY   NOT NULL,
   username  VARCHAR           NOT NULL,
   password  VARCHAR           NOT NULL
);

INSERT INTO users (id, username, password) 
VALUES (1, 'sarah', 'good-password');

INSERT INTO users (id, username, password) 
VALUES (2, 'thomas', 'password123');

INSERT INTO users (id, username, password) 
VALUES (3, 'patrice', md5('palm-kumquat-futon-padden'));

After the user submits the username:

thomas

and password:

password123

the resulting query will look like this:

SELECT * 
FROM users 
WHERE username='thomas' AND password='password123';

and the following results will be returned:

| id | username |    password |
|----|----------|-------------|
|  2 |  tcountz | password123 |

As expected, our backend code then takes this result, and calls the log_in_user() function which authorizes the user to access certain parts of the application.

Authentication Attack

The vulnerability here will allow an attacker to log in as any user.

Because user input is interpolated directly into the SQL query, we can have the server execute arbitrary SQL statements. Let’s take a look at an example that would allow us to log into any user, given we have their username.

After the user submits the username:

patrice'; --

and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username='patrice'; --' AND password='<anything>'

and the results:

| id | username |                         password |
|----|----------|----------------------------------|
|  3 |  patrice | 26e9053a783f364d949b4e400dd2f68c |

Now, our application, again, takes the 0th results (this time the user patrice) and logs them in.

What Happened?

Even though we hashed Patrice’s password, our query was vulnerable to interpreting the ', ;, and -- PostgreSQL meta characters.

Firstly, the single quote: ', ends the string (or VARCHAR) that we’re searching for; in this case the username patrice. Next, the semi-colon ; represents the end of the SQL statement. Finally, the -- represents a comment and tells PostgreSQL to ignore everything that comes after.

This effectively makes our query look like this:

SELECT * 
FROM users 
WHERE username='patrice';

Which, as we’ve seen, and as we expect, returns the patrice user and logs them in. Now our attacker has been authenticated and has access to Patrice’s account!

This combination: '; --, and others like it, show up often in SQL injection attacks and it works by prematurely ending a SQL statement.

Data Destruction Attack

In the example above, the 0th row of the results returned from the query will be passed into the log_in_user() function, but the scope of this attack vector isn’t limited to logging in.

As an example of how we can attack the server to execute any SQL, take this example where we destroy the users table.

If we enter a username of:

'; DROP TABLE users; --

and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username=''; DROP TABLE users; --' AND password='<anything>'

The user-facing effect of this query might not tell us exactly what has happened, but a developer might see something like this show up in the logs:

ERROR: relation "users" does not exist

What Happened?

Similar to the first attack, we’ve cut the original query short and this time, we’ve injected our own query to drop the users table.

Prevention

Parameterized Queries

The use of prepared statements with variable binding (aka parameterized queries) is how all developers should first be taught how to write database queries.

Parameterized queries force the developer to first define all the SQL code, and then pass in each parameter to the query later. This coding style allows the database to distinguish between code and data, regardless of what user input is supplied.

— OWASP SQL Injection Prevention Cheat Sheet

Our code was vulnerable because we use string interpolation to build an SQL statement directly from user input. Instead, we should “parameterize” our query by using whatever tools our language gives us to separate the data plane (input) from the control plane (SQL). This is the idea of using variable binding (placing user input into a type of variable) with prepared statements (the rest of the SQL that we don’t want the user to be able to alter).

The way to code this varies depending on the language you’re working with, so check out the OWASP SQL Injection Prevention Cheat Sheet section on parameterized queries.

Other Defenses

Another defense against SQL injection are stored procedures, which are predefined SQL statements stored in the data table. These procedures can have parameters and can effectively be similar to constructs from different languages. Read more here.

To read more about allow-listing or escaping user input, see the rest of the OWASP SQL Injection Prevention Cheat Sheet.

Next Steps

Research SQL injection prevention for your ORM/database
- e.g. https://guides.rubyonrails.org/security.html#sql-injection
Aim to separate queries and data (parameterized queries)
- e.g. https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Query_Parameterization_Cheat_Sheet.md
Consider SQL Injection when reviewing code

Code Review

Aim to validate user input by testing type, length, format, and range.
Avoid building SQL statements directly from user input.
Implement multiple layers of validation.
Avoid concatenating user input that is not validated; this is the primary point of entry for script injection.
You should review all code that calls execute(), exe(), and any SQL calls or commands that can call out outside resources or the command line.

Example Code Snippets

Ruby/ActiveRecord

# SQL Injection Vulnerable Ruby/ActiveRecord
# -- snip --

user = User.where(
    "username = #{params[:username]} AND " \
    "password = #{params[:password]}"
).first

# -- snip --

# SQL Injection Safe Ruby/ActiveRecord
# -- snip --

user = User.where(
    username: params[:username], password: params[:password]
).first

# -- snip --

Elixir/Ecto

# SQL Injection Vulnerable Elixir/Ecto
# -- snip --

query = """
  SELECT *
  FROM users
  WHERE username = \'#{params["username"]}\'
  AND password = \'#{params["pasword"]}\';
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, []
)[:rows][0]

# -- snip --

# SQL Injection Safe Elixir/Ecto
# -- snip --

query = """
  SELECT * 
  FROM users 
  WHERE username = $1 
  AND password = $2;
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, params["username"], params["password"]
)[:rows][0]

# -- snip --

Java

// SQL Injection Vulnerable Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = "
    + username + " AND password = " + password + ";";

Statement statement = connection.createStatement();
Object user = statement.executeQuery(query).getObject(0);

// -- snip --

// SQL Injection Safe Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = ? AND password = ?";
PreparedStatement pstmt = connection.prepareStatement(query);
pstmt.setString( 1, username);
pstmt.setString( 2, password);

Object user = pstmt.executeQuery().getObject(0);

// -- snip --

Resources

Dare to be Good Enough—You'll be Happier for it

2020-10-06T00:00:00+00:00

Originally published on 8th Light’s blog

As I sit down to write, I noticed a tug at the back of my neck—a tension forming as I begin editing my words obsessively. It’s a habit of writing characterized by my attempts at achieving the best possible outcome. “I know that I can write well,” I tell myself, “I’ve just got to strike the right tempo, tone, voice…”

I often have the same sensation when I’m writing code. The giants of our industry, whose shoulders I stand upon, tell me all sorts of things that should be true about my code—the way it reads, the way it’s coupled to other parts of a system, and the amount of complexity it’s allowed to have—it’s all qualified and quantified.

This tension, of balancing all of the small discrete decisions that crop up when implementing a software feature, causes me to try to painstakingly eke out the “correct” way to do things. It’s as if when faced with a choice, I scan through an endless tree of outcomes trying to game out what my best option would be. “Do I extract a shared setup for these tests, or do I duplicate it for each assertion?” “Under what circumstance should I choose the builder pattern over a factory method?” “Should this method be called is_ready or is_prepared?” The list goes on and on.

If this sounds familiar to you, you’re not alone. People who tackle decision-making in this way are called maximizers.

In order to determine their optimal decision outcome, maximizers feel compelled to examine each and every alternative available, which is often infeasible in reality due to the limitations in human cognition (Roets, et al., 2012).

In theory, maximizers make the best choices, but research shows that, in practice, maximizers aren’t ever sure that the choices they make are the best choices, and it’s this “not knowing” that leads to analysis paralysis, and ultimately, regret.

Satisficers, on the other hand, are happy with whatever “good enough” choice they make:

Several options may fall within a satisficer‘s threshold for acceptability, providing greater flexibility and latitude in achieving a desired decision outcome. As soon as [they encounter] a good enough option, the satisficer can easily ignore the addition of new choices to the decision domain (Peng, 2013).

Satisficers are pragmatists at their core; the solution that works is the solution they commit to.

This way of decision-making is embedded in the fabric of agile methodologies, which gives us permission to make small incremental changes over time, rather than grasping for the perfect solution from the start. From this, we know that iterative design can help us build tighter feedback loops, deliver value to stakeholders sooner, and ultimately keep our development reactive and responsive to the needs of our customers.

Our projects’ bottom line isn’t the only beneficiary to thinking like a satisficer. Even though the research is still out over whether maximizers or satisficers achieve objectively better decision-making (Peng, 2013), what is clear is that satisficers are happier. In fact, even if a better option occurs to them later, satisficers are less likely to experience regret (Schwartz et al., 2002).

If you find yourself like me, trying to game out the best future outcomes based on limited knowledge and with bounded cognition, remember the ways of the satisficers. Make a good enough decision today, and remain flexible and forgiving in order to make another good enough decision tomorrow. Of course, we all have metaphorical hills on which we may choose to die, but choose your metaphorical battles wisely and dare to be good enough.

References

Roets, A., Schwartz, B., & Guan, Y. (2012). The tyranny of choice: a cross-cultural investigation of satisficing effects on well-being. Judgment and Decision Making.

Peng, Starry. “Maximizing and Satisficing in Decision-Making Dyads.” ScholarlyCommons, 2013, https://repository.upenn.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=1101&context=wharton_research_scholars

Schwartz, B., Ward, A., Monterosso, J.,Lyubomirsky, S., White, K., & Lehman, D. R. (2002). Maximizing versus satisficing: happiness is a matter of choice. Journal of personality and social psychology.

Agile Alliance. “What is Incremental Development?”, 2019, https://www.agilealliance.org/glossary/incremental-development

Specify Bundler Version

2020-09-18T00:00:00+00:00

If you’re like me, you may have versions of both Bundler 1 and Bundler 2 installed on your system. This can make it difficult to manage different codebases.

Say one requires an older version, like 1.13.6. You can run the following command to install it,

gem install bundler -v 1.13.6

but when you install gems,

bundle install

you might end up seeing something like

BUNDLED WITH
   2.1.4

in your Gemfile.lock file… which is not the same as bundling with 1.13.6.

You can tell Bundler that you’d like to use a specific version of Bundler by specifying the exact version number before the command you wish to run. Like so:

bundle _x.x.x_ install

To install gems for your project using Bundler 1.13.6, you can use this command to force Bundler to use the correct version:

bundle _1.13.6_ install

And you should see

BUNDLED WITH
   1.13.6

at the end of your Gemfile.lock file.

Hooray!

Types of Notetaking

2020-08-30T00:00:00+00:00

Earlier this week, my pair and I struck upon something interesting, and I stopped to take notes.

Every time I see you taking notes, you’re using a new method!

— Pair

She’s not wrong. Over the past several years, I’ve tried many different ways of capturing my thoughts: yellow legal pads, Basecamp, Notion, Vimwiki, Org Mode, Tyke, index cards, Gitbook, blogging with Jekyll and Medium, Slack/Emails to self, Notability, and voice notes. I didn’t switch note taking styles because I liked to try new things, I switched because I hadn’t found a note-taking tool that fit all of my needs. I’ve researched and tried different note taking systems like Getting Things Done and Building a Second Brain, but as soon as a system or tool caused me to think more about note taking and less about what I was taking notes on, I struggled to keep focused and motivated.

I realized that I should drop all of that and let my intuition guide how I record information best. It turns out that trying to force everything I wanted to write down into one system caused me to feel frustrated and confused. For some things, Vimwiki was perfect! For others, it felt clunky and difficult. A pocketbook was awesome for a while, then suddenly I felt like I could never keep things straight.

Once I stopped fighting this battle, I found that there were three types of notes that I was taking and therefore there were three different tools/methods I could use to take them.

Actionable TODOs

In Situ Documentation

Long-form Reference

Actionable TODOs

Buzz. Buzz.

I check my phone. It’s a text from my husband:

If you stop by the grocery store, I just used the last few onions. Can you grab some?

TODOs and short-term tasks come at me from all directions and at any time. Old me would just try to remember everything, but I’ve realized just how much bandwidth and brain cycles that takes up, so I’ve taken to writing these things down. I write down things that are “actionable,” and are “obvious when they’re done,” like “take out trash,” or “fill out form DL-80.”

My criteria for a TODO tool:

Readily available — I need access to my TODO list at all times. I don’t know when I’ll have the opportunity to check something off my list and I don’t know where I’ll be when I need to add a new task.
Low friction — Capturing a TODO needs to be a quick activity so that I can continue on with my day. Having to also schedule a reminder, add a category, a priority value, and a description, before I could “save” a task limited my willingness to commit to writing down my TODOs
Non-ceremoneous — similar to being low friction, I don’t want any other features tied to my TODO notes besides creating a task and marking as task a finished. For example, one TODO tool generated a weekly chart displaying how many tasks I completed compared to last week. I started re-structuring my TODOs to effect these outcomes and meant that I was no longer focusing on the tasks themselves.

My solution: a passport-sized notebook and pen

I can carry this notebook and pen most places with me, and I’m un-apologetic about what I write in it. I mix work-related things with personal task, and I scribble quick “…can you write down this number…” notes, too. Even though it’s for “tasks,” I occasionally find myself taking temporal notes like, “finished reading Convenience Store Woman” and ideas like “write blog post about note taking.”

A pocketbook is an always-charged smartphone.

In Situ Documentation

Do you remember what we ended up deciding on Friday?

In a world of synchronous-first communication, after an hours long meeting, the last thing I want to do is write down everything we just talked about. More than that, after an hours long meeting, the last last thing I want to do is feel like we forgot everything we talked about.

Sometimes the meetings I’m in are a bit chaotic and cover a lot of ground. (Often more like mob programming than a meeting). In these cases, I’ve found that rapid-fire note taking has allowed me to both 1) keep a reference of what we may have decided and 2) free up my brain to keep listening.

Contemporaneous Work Notes look like this for me:

Draft-zero — I think of these notes as casting a very wide net. I scribe almost continuously and more often than not, half of it is garbage. With that said, I need a tool that can help me create private unedited drafts that allow me to hone the 50% later.
Adaptable — I might be taking notes during a meeting, while pairing, during code review, etc. I need a system that adapts to however my brain feels like organizing information in the moment. Besides a date, these notes can end up looking very different: a doddle here, a line reference there, a TODO might sneak in—who knows?!
Disposible — These notes depreciate in value over time. Ideally, I reference these notes shortly after taking them in order to formalize something that may need to be shared. When I’m done, the in situ notes shouldn’t contain any important information that isn’t captured in a different form elsewhere.

My solution: yellow legal pad and mechanical pencil*

Yellow legal pads are beloved (and hated) by many for their accessibility and ease of use, see: The Illustrious History of the Yellow Legal Pad. I fell in love with them because they feel limitless. I found that when taking notes in beautiful leather-bound $40 notebook, I became a bit too precious about what I would write in it and began to edit my notes before I put pencil to paper! On the contrary, yellow legal pads are cheap, thin, and always open. I prefer a mechanical pencil because I feel as though I can write cleaner lines with a 0.5mm lead than I can with an equally fine pen. Also, I do erase things sometimes, especially if I’m trying to capture a concept in an illustration.

A yellow legal pad is a private portable whiteboard.

*In an effort to minimize my impact on the environment, I’ve moved to using Notability on the iPad with an Apple Pencil or folding yellow legal pad pages into four—using one section per day and both sides of the page.

Long-Form Reference

CUT TO: Thomas is, yet again, digging through the pages of Building Git by James Coglan to remember how Ruby’s open3 module exposes child processes to IO streams.

I love diving deep into different topics, often topics that I may not directly visit or use again in my day-to-day work. Take machine learning for example. When I first studied it, I learned quite a lot! But, just a few months later, when I revisited data science, I felt as though I was back to square one. Not only can I have notes to come back to, I’ve noticed that my comprehension increases when I’m able to write new information down in my own words.

My solution to avoid this type of thing is to take long-form reference notes:

Reference-able — As you might be able to guess, these notes have to be notes that I can come back to and understand. That means that these aren’t draft-zero notes, they’re well crafted and continuously editable as I learn and discover new things. They’re written with the reader in mind, even on the first pass.
Structured — These notes should be both hierarchical and linkable. If I taking notes on material that is related to notes I’ve taken in the past, I want to connect them somehow. This helps with referencing things later, but also helps as I’m editing.
Pleasurable — It’s important to me that these notes be enjoyable to write and to read. This means the ergonomics have to be comfortable and familiar and help me to focus on the content, rather than the styling.

My solution: Gitbook/personal wiki

This is a new venture for me, but you can find the beginnings of this journey at research.thomascountz.com. Gitbook has switch from self-hosted to a SAAS offering in the past few years, but still offer free services for individuals. I enjoy Gitbook because the online editor is intuitive and comfortable, and all of my notes are backed by git/Github, which means exportability. Also, I like that these notes are public. They’re not quite a blog post, but being able to link directly to specific notes is helpful when sharing ideas.

I’ve also taken to recording personal project documentation in wiki format. Even when I’m working alone, I prefer to write asynchronous documentation as though I’m together with a team. These project docs help me link thoughts and ideas together that would otherwise decay over time as other priorities come into focus.

A personal wiki is the brain’s external hard drive.

Your Notes, Your Way!

It’s with this combination of a small notebook, yellow legal pad, and personal wiki that I declutter my brain and keep focused. What’s your preferred note-taking method/tool? Do you have one that fits all your needs or are you like me: different tools for different situations? I’d love to hear your thoughts!

FastBook Chapter 4 Questions & Notes

2020-05-09T00:00:00+00:00

Fastai, known for it’s MOOCs, is working on a book, Fastbook to go along with their new MOOC starting July 2020. In my eagerness, I’ve been going through the draft of the book (linked above, though they may remove it after publication) and have been coding alongside on Kaggle. At the end of each chapter of the book is a list of questions for the reader/students to answer. I’ve found these questions to be rigorous and useful to deepen my understanding.

This blog post might not be useful to anyone besides myself, but I want to keep my answers to these questions as a reference somewhere other than a Kaggle notebook.

How is a greyscale image represented on a computer? How about a color image?

A greysale image is represented by a matrix/rank 2 tensor/grid of pixels with a value between 0 and 255. Color images are a rank 3 tensor, height and width along two dimensions and third dimension representing values between 0 and 255 for redness, greenness, and blueness.

How are the files and folders in the MNIST_SAMPLE dataset structured? Why?

There are two directories: /train and /valid, which each contain two directories: /3 and /7. Inside those directories are images of the respective digit. The train/ directory contains the majority of images to be used to train a model likewise the /valid directory contains images to validate a model. There’s also a labels.csv file which likely contains a mapping between file names and digit label.

Explain how the “pixel similarity” approach to classifying digits works.

The idea behind this approach was to look at all of the images for a given digit and compute the average value for each pixel across all images. Then, by comparing an unlabeled digit to that “average” digit, we could determine how similar or dissimilar the unknown image was to the known average.

What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.

It’s a way of mapping over an enumerable object in Python

[i * 2 for i in range(20) if i % 2 == 0]

What is a “rank 3 tensor”?

A rank 3 tensor is a tensor with three dimensions, such as a color photo.

What is the difference between tensor rank and shape? How do you get the rank from the shape?

A tensor’s rank is equivalent to the number of dimensions a tensor has. It’s shape is how many values exist in each dimension. Because the shape tells us the number of values per dimension, we can determine the number of values returned to determine the rank.

What are RMSE and L1 norm?

RMSE, or root mean square error is the average difference of the prediction and the label squared and then 2nd rooted?

def mse(y, yhat): ((y - yhat)**2).mean().sqrt()

L1 norm is the average of the absolute value of the difference between y and yhat

def l1(y, yhat): (y - yhat).abs().mean()

How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?

Broadcasting.

Create a 3x3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom right 4 numbers.

(torch.reshape(torch.arrange(1, 10), (3, 3)) * 2)[-2:,-2:]

What is broadcasting?

Broadcasting is traditionally used to support computation between two unequal tensor with the side effect of not needing to copy any data. This side effect can be taken advantage of for applying calculation to two tensors even if they are equal in rank.

This happens implicitly in Pytorch when operating of two vectors, for example:

Non-broadcasting: slow

>>> a = [1, 1, 1]
>>> b = [1, 1, 1]
>>> c = [x + y for x, y in zip(a, b)]
[2, 2, 2]

Broadcasting: fast

>>> a = tensor([1, 1, 1])
>>> b = tensor([1, 1, 1])
>>> a + b
tensor([2, 2, 2])

Are metrics generally calculated using the training set, or the validation set? Why?

A metric is a function that measures quality of the model’s predictions using the validation set, and will be printed at the end of each epoch. —Fastbook, Ch. 4

We use a metric to judge our network’s ability to accurately predict outputs for data is hasn’t seen before.

What is SGD?

Gradient Descent is the process by which we update the weights/parameters/coefficients of our model in order to minimize the loss function. By taking the gradient/derivative of our loss function, we can determine how a change in our parameters would result in a change to the output. We can use that gradient to update our weights in proportion to the learning rate. Stochastic just means measure loss and update our weights in in “batches.” Some definitions say that “Stochastic” means we calculate and update for the entire training set.

Why does SGD use mini batches?

Updating parameters after every training example takes a long time and can mean sporadic jumps in parameters after each example as the network tries to optimize for each individual example. Waiting until after going through the entire dataset can be impossible for large datasets that don’t fit into memory. It’s inefficient and doesn’t necessarily provide better results. Mini-batches solve both of these problems. We reduce the variance between each parameter update which can smooth out convergence, and we don’t run into memory issues waiting for the entire training to complete.

What are the 7 steps in SGD for machine learning?

Initialize weights, Predict, calculate loss, determine gradient, update parameters, repeat and stop.

How do we initialize the weights in a model?

Randomly. There are other methods, but randomly is a good starting point.

What is “loss”?

Loss is the function that tells us how well or how poorly our model predicted an output for a example.

Why can’t we always use a high learning rate?

The learning rate is what we use gently adjust the parameters. A high learning rate can cause our loss to jump around sporadically as we attempt to minimize it.

What is a “gradient”?

The gradient of a function, denoted as follows, is the vector of partial derivatives with respect to all of the independent variables, aka, the parameters.

The derivative, it’s a function that describes the slope of another function at a given point. It tells us how “quickly” a function changes at a certain input.

To calculate the partial derivative of a single parameter, you hold all other parameters constant. After computing all of the partial derivatives, they’re collected into a vector call the gradient.

Do you need to know how to calculate gradients yourself?

Nope. Thanks Pytorch!

Why can’t we use accuracy as a loss function?

Accuracy is for humans to consume. It tells us how well a model is at prediction examples that it has never seen before (in the validate set) overall. Accuracy isn’t necessarily a function from which we can calculate a gradient/derivative in order update our weights. In the MNIST model in this chapter, a small nudge in a parameter won’t necessarily affect accuracy, unless that small nudge changes a prediction from a 0 to a 1 or vice versa.

Draw the sigmoid function. What is special about its shape?

It looks like an S on it side with horizontal asymptotes a y=0 and y=-1. The smooth curve makes it special because it helps us to gently calculate derivatives of our loss.

What is the difference between loss and metric?

A we use a metric to determine human-interpretable accuracy, the model uses loss to determine how to update weights.

What is the function to calculate new weights using a learning rate?

In this example: Stochastic Gradient Descent

What does the DataLoader class do?

DataLoader will return an (x, y) tuple for our model, divide the data into training and validation sets, and create mini batches

Write pseudo-code showing the basic steps taken each epoch for SGD.

prediction = model(x, params)
loss = loss(prediction, label)
loss.backward()
for p in params:
    p.grad.data += lr * p.grad.data
    p.grad.data = None

Create a function which, if passed two arguments [1,2,3,4] and ‘abcd’, returns [(1, ‘a’), (2, ‘b’), (3, ‘c’), (4, ‘d’)]. What is special about that output data structure?

def f(a, b): return list(zip(a, list(b)))

It returns a list of tuples.

What does view do in PyTorch?

From the docs:

PyTorch allows a tensor to be a View of an existing tensor. View tensor shares the same underlying data with its base tensor. Supporting View avoids explicit data copy, thus allows us to do fast and memory efficient reshaping, slicing and element-wise operations.

For example, to get a view of an existing tensor t, you can call t.view(…).

>>> t = torch.rand(4, 4); t
tensor([[0.2108, 0.4824, 0.4418, 0.9436],
        [0.9554, 0.5866, 0.7631, 0.2809],
        [0.2934, 0.7608, 0.7741, 0.6948],
        [0.0813, 0.5682, 0.8023, 0.3858]])
>>> b = t.view(2, 8); b
tensor([[0.2108, 0.4824, 0.4418, 0.9436, 0.9554, 0.5866, 0.7631, 0.2809],
        [0.2934, 0.7608, 0.7741, 0.6948, 0.0813, 0.5682, 0.8023, 0.3858]])
>>> t.storage().data_ptr() == b.storage().data_ptr()  # `t` and `b` share the same underlying data.
True
# Modifying view tensor changes base tensor as well.
>>> b[0][0] = 3.14
>>> t[0][0]
tensor(3.14)

What are the “bias” parameters in a neural network? Why do we need them?

Bias terms are additional constants attached to neurons and added to the weighted input before the activation function is applied. Bias terms help models represent patterns that do not necessarily pass through the origin. For example, if all your features were 0, would your output also be zero? Is it possible there is some base value upon which your features have an effect? Bias terms typically accompany weights and must also be learned by your model. —https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html?highlight=bias#bias

What does the @ operator do in python?

Matrix multiplication ( * is element-wise multiplication)

What does the backward method do?

loss.backward() computes dloss/dx for every parameter x which has requires_grad=True. These are accumulated into x.grad for every parameter x. In pseudo-code:

x.grad += dloss/dx

https://medium.com/@zhang_yang/how-pytorch-tensors-backward-accumulates-gradient-8d1bf675579b

Why do we have to zero the gradients?

See link above, pytorch will accumulate the gradient from all operations.

What information do we have to pass to Learner?

The DataLoader that contains the data, the model, the optimization function (e.g. SGD), loss functions (e.g. MSE), and any metrics to print

learn = Learner(dls,
                nn.Linear(28*28,1),
                opt_func=SGD,
                loss_func=mnist_loss,
                metrics=batch_accuracy)

Show python or pseudo-code for the basic steps of a training loop.

for _ in range(epoch):
    prediction = model(x, params)
    loss = loss(prediction, label)
    loss.backward()
    for p in params:
        p.grad.data += lr * p.grad.data
        p.grad.data = None

What is “ReLU”? Draw a plot of it for values from -2 to +2.

A ReLU, or rectified linear unit, replaces every negative number with 0

What is an “activation function”?

I understand an activation function as being a non-linear layer in a neural network. This has the effect of allowing or preventing a certain “neuron” from activating depending on a threshold. For a ReLU, only positive values in a previous layer would be passed forward to the next layer. Similarly, a unit step function only passes values greater than 0 to the subsequent layers, but all positive values are passed on as 1.

What’s the difference between F.relu and nn.ReLU?

F.relu is the function, nn.ReLU is the Pytorch module module.

The universal approximation theorem shows that any function can be approximated as closely as needed using just one nonlinearity. So why do we normally use more?

Efficiency and performance. With deeper models, we don’t need as many parameters. Smaller matrices with more layers > larger matrices with fewer layers.

Thomas Countz

Job Scheduling with at(1)

Quick Start

Good Citizen Syndrome

Event Streaming vs Background Jobs

Log of Events

To-Do List

The Confusion

The Conclusion

Chrome's SSL Bypass Cheatcode

This is Unsafe

History of the Bypass Code

Danger (2014)

Bad Idea (2015)

This is not Safe (2018)

dGhpc2lzdW5zYWZl (2018 - Present)

Is this Unsafe?

Debouncing API Calls

Hardware

Software

Debouncing with Delays

Delays and Cancellations

Where Does Apple Hide Your Voice Memo Transcripts?

Step 0: The Problem

Step 1: Find the Transcripts

Step 2: Explore the Files

Step 3: JSON Extraction

Containers

Atoms

Accessing the Transcript Atom

Postscript

Footnotes

Forth Feedback Loop

Color Strings Refinement

Good Morning, Team!

Run Commands on File Change

Ruby Bug #21117: Numbered Block Params

tl;dr

The Bug

The Fix

What About it?

Routine Complications

Hill Chart Web App

Capture ActiveRecord Queries

Ruby CLI Progress Bar

Checklist App Prototype

Interactive jq

ActiveRecord Models to a Mermaid ERD

Adding Journals

Low-Poly Image Generation using Evolutionary Algorithms in Ruby

Table of Contents

Evolutionary Algorithm Overview

The Petri Dish Framework

Understanding the Objective

Member Representation

Input Target Image

Member Genes

Reconstructed Output Image

Population Initialization

Fitness Function

Genetic Operators

Parent Selection Function

Crossover Function

Mutation Function

Replacement

End Condition

Putting it Together

Final Configuration

Results

Subjective Analysis

Data Analysis and Interpretation

Log Data

Image Data

Conclusion

Footnotes

Utilizing Docker for Bundling in a Specific Ruby Environment

A Thought on Practice

Københavns Luftkvalitet og Cykling

Introduction

Data Ingestion & Initial Aggregation

What About `it`?

`TEXT` & `VARCHAR`