Posts on erenon.hu

Summary of Readings

Thu, 11 Dec 2025 00:00:00 +0000

A non-exhaustive list of non-fiction books in no particular order I read recently, with some recommendation. This post was inspired by the similar posts of Eli Bendersky.

Faith

The Old Testament. Reading it was a challenge of the faith. Long and often difficult to understand. It is interesting to perceive how the faith of authors change over time, as they refine their understanding of their Lord, in the context of their history. My understanding was greatly helped by also reading concurrently:
Az Ószövetség keletkezése I.-II. by Rózsa Huba. A comprehensive review of the studies of every book in the Old Testament. For each book, it explains its origins, genre, historical background, content, relation to the other books, and the views and theories of different scholars. This book is its own illustration in a fascinating and unique way.

History

The Jewish War by Flavius Josephus. An account centered on the Jewish rebellion against the Roman around A.D. 66, starting before Herod the Great, ending after the destruction of Jerusalem. Long and detailed, but with many factual errors (and suspected cosmetics), from Josephus, once a Jewish general of the rebellion, turned to writing after falling captive to Vespasianus. Take away: apparently, people changed little in 2000 years: greed, hubris, envy, cruelty: nothing new. Also: Herod had every reason to get mad.

Science

Breaking Through: My Life in Science by Katalin Karikó. Autobiography of the Nobel laureate mRNS pioneer, who played a great role in developing a COVID-19 vaccine. Also a story of how an extremely determined girl from a family of simple means achieves great success, through sheer will a continuous work - for decades. I’d also like to acknowledge the seemingly unwavering support of her family. There’s not much about COVID itself, books are referenced instead that explain it in details. The second part (after Penn) felt rushed, the development mostly unclear to me. I suppose Kati had some reading due… The writing is not great - it is clearly written by a scientist.

Finance

Straight to Hell by John LeFevre. Terrific stories about terrible people, written by someone who portrays himself as a similarly terrible person, with a value system completely foreign to me. The (supposedly true) stories are entertaining in a cringe way. Going through this book was the reading equivalent of the guilty pleasure inducing overeating of chocolate cookies.

Multidimensional Make Targets

Tue, 18 Mar 2025 00:00:00 +0000

This is about describing targets in GNU Make that depend on many parameters that need to be sampled.

Example use-case 1: There’s a simulation with many input parameters, e.g: weather simulation, with current temperature, humidity, date, duration parameters. You want to run the simulation for some combinations of these parameters.

Example use-case 2: There’s a remote database the can be queried. You want to sample larger batches, and apply a pipeline of transformations on them, iteratively, as you figure out what’s really needed.

Describing these tasks in a Makefile is helpful because:

It makes dependencies and data-flow explicit (vs. manual changes and ad-hoc intermediate files)
Slow steps can be cached (simulation, database download)
make can execute independent steps parallel, greatly reducing execution time.

This post is about handling intermediate outputs, that depend on many parameters. Consider this motivating example:

DATE ?= 2025-03-15
LONGITUDE ?= 19.0527693
LATITUDE ?= 47.4943593
RESOLUTION ?= min

temperature: download_temperature.py
  ./$< --date $(DATE) --longitude $(LONGITUDE) --latitude $(LATIDUTE) --resolution $(RESOLUTION) > $@

In this case, by default, make temperature downloads the measurements for a pre-defined location and day. The user can override the default using environment variables, e.g: DATE=2025-03-16 make temperature will download it for the next day. However, there are multiple problems here, including:

After a successful download, changing the variables will not prompt make to re-execute the download. It doesn’t know that the target depends on those variables, and it will consider it up-to-date.
We can’t have multiple measurements (with different parameters) in our tree at the same time, e.g: to compare them.
If we have a similar “humidity” target, it gets easy to accidentally combine different measurements that are otherwise unrelated.

To fix these, we need to make the dependency of the target on the variables explicit. The key idea is that we need to encode the variables in the target name. For example, instead of simply having “temperature”, we’ll have “o/2025-03-15/19.0527693/47.4943593/min/temperature”. This immediately solves all of the above.

But how do we describe this in a Makefile? We can’t enumerate all the possible combinations. Ideally, make pattern rules could support something like this:

# not actual make syntax
o/%DATE%/%LONGITUDE%/%LATITUDE%/%RESOLUTION%/temperature: download_temperature.py
  ./$< --date $(DATE) --longitude $(LONGITUDE) --latitude $(LATIDUTE) --resolution $(RESOLUTION) > $@

but it doesn’t:

A pattern rule contains the character ‘%’ (exactly one of them) in the target.

Instead, we’ll emulate this, using Pattern-specific Variable Values:

# when requesting "o/2025-03-15/19.0527693/47.4943593/min/temperature",
# split the target name ($@) into a list, and set the variables using the directory names.
o/%: override DATE = $(word 2,$(subst /, ,$@))
o/%: override LONGITUDE = $(word 3,$(subst /, ,$@))
o/%: override LATITUDE = $(word 4,$(subst /, ,$@))
o/%: override RESOLUTION = $(word 5,$(subst /, ,$@))

o/%/temperature: download_temperature.py
  ./$< --date $(DATE) --longitude $(LONGITUDE) --latitude $(LATIDUTE) --resolution $(RESOLUTION) > $@

o/%/humidity: download_humidity.py
  ./$< --date $(DATE) --longitude $(LONGITUDE) --latitude $(LATIDUTE) --resolution $(RESOLUTION) > $@

To hang an Icon

Fri, 14 Mar 2025 00:00:00 +0000

I wanted to hang a picture, in a way that it doesn’t touch the wall directly, using a protruding hanger. If the hanger is difficult to observe, it makes a nice floating effect.

I couldn’t find a “µ” shaped piece of metal, and I don’t have right drills for metal either, so I decided to 3D-print the appropriate hanger. This would have been my first time doing this.

First, I needed a 3D modelling software, that is free and Linux compatible. From the Arch Linux offering, I quickly selected OpenSCAD.

OpenSCAD is a wonderful software - instead of moving the model elements with a mouse, it lets you describe the model using code, something I’m familiar with. It is lightweight, with a clear, responsive UI. By skimming the wiki and looking at examples, I was able to create a simple model in less than 50 minutes. A testament of a great piece of software indeed.

The code producing this model is short and simple. The parametric nature allows quick changes, e.g: of thickness or depth. After exporting the model to .stl, I asked a friend to print it (thanks!). The Prusa machine did a beautiful job:

Finally, after a bit of measuring and drilling, the final composition: modern and antique co-existing in perfect harmony.

Restore file from editor memory

Mon, 14 Oct 2024 00:00:00 +0000

I was editing a large text file over sshfs using nvim. Probably due to a lose network cable, during save, the connection broke, and only half of the data reached the remote computer, the other half remained in the send buffers of the kernel.

The editor was configured to use no swapfiles, therefore it wrote the target file directly: at this point the second half of the file was lost, and the editor stuck in disk sleep.

The situation did not improve after the network connection was restored and the share re-mounted (after it was forcefully unmounted…). The last backup was too old, and I needed the data.

Solution

Get the PIDs of the currently running editors:

$ ps -A | grep nvim
<PID>s

Let’s find which editor instance was writing the file:

$ ls -l /proc/<PID>/fd/
13 -> path/to/our/file

We can check that this editor is indeed stuck, waiting for the kernel to complete a write operation, that’ll never happen:

cat /proc/<PID>/status | grep State
State: D (disk sleep)

In this state, the process cannot be attached to, gdb or gcore will hang if tried. Fortunately, /proc/<PID>/mem is still readable. The file is sparse, only offsets mapped by that process are readable. /proc/<PID>/maps gives the valid offsets. A python script from StackExchange will save the readable sections into a file. Slightly improved:

# Original https://unix.stackexchange.com/questions/6301/how-do-i-read-from-proc-pid-mem-under-linux/6302#6302
import re

pid = '123456' # change this
maps_file = open(f"/proc/{pid}/maps", 'r')
mem_file = open(f"/proc/{pid}/mem", 'rb', 0)
output_file = open("self.dump", 'wb')
for line in maps_file.readlines():  # for each mapped region
    m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])', line)
    if m.group(3) == 'r':  # if this is a readable region
        start = int(m.group(1), 16)
        end = int(m.group(2), 16)
        mem_file.seek(start)  # seek to region start
        try:
            chunk = mem_file.read(end - start)  # read region contents
            output_file.write(chunk)  # dump contents to standard output
        except OSError:
            print("failed to read from ", start)
maps_file.close()
mem_file.close()
output_file.close()

Running this script will save the memory of the editor, that contains the complete file – in chunks. Fortunately, the internal structure is line-oriented, and tends to follow the time of insertions. The lines in each chunk are in reverse order, use tac to fix that:

strings nvim.dump | tac > nvim.strings

Takeaway

set swapfile, when editing over sshfs
More frequent backups
Don’t panic.

No Miracles

Thu, 04 Apr 2024 00:00:00 +0000

One would argue that miracles do not exist.

Long time ago, a prominent member of a now mostly abandoned web forum offered to lend a technical book (JavaScript: The Good Parts) to me, an absolute unknown nobody. For free. No strings attached. Obviously, it isn’t a miracle, just a selfless act of kindness. I was still flattered.

15 years later, I was wondering what happened to him. His blog had a very memorable domain name. Lacking a device, I couldn’t check it out immediately, and the memory faded. The very next day, a post from that very blog popped up in my RSS reader, coming through the HackerNews “best” feed. A feed that selects the top few posts of thousands of submissions all over the world. A lucky coincidence, we can conclude, no miracle.

The article is about a scientific experiment (if you squint a bit). Given the connection above, I felt compelled to verify the results by reproduction, especially because I had all the required resources:

Some colored pieces of DUPLO
A xylophone of matching (somewhat) colors
An interested toddler

Please find the anonymized results of the reproduced experiment in the diagram below (Figure 1):

I can confirm that, in accordance with the original research:

Toddler beating the xylophone can instruct papa correctly to find the daughter
Toddler listening to the xylophone can move the daughter correctly to find papa
and have fun in the meantime

In addition, we found that:

Collectible flowers can be added to the map, and a mini inventory (separate flat piece), so the daughter can find and properly greet mama
Introducing windows that can be opened and climbed through works
Only listening to the xylophone without looking at the colors doesn’t work

Apparently, you can have this simple, no-computer equipment, and still have fun with your kid, science concur. So we experiment, explore, build and laugh together, freely.

And one might argue, that miracles do not exist.

Summary of Readings

Fri, 20 Oct 2023 00:00:00 +0000

A non-exhaustive list of non-fiction books in no particular order I read recently, with some recommendation. This post was inspired by the similar posts of Eli Bendersky.

Adventure

Endurance: Shackleton’s Incredible Voyage by Alfred Lansing. The true story of how the members of a failed Antarctic expedition struggle for years to get home. Based on journals and interviews with the survived. Excellent writing, recommended.

Biography

Surely you are joking, Mr. Feynman by Richard P. Feynman. Edited personal stories told by the Nobel prize laureate author, presented in an informal, entertaining style, without any forced political correctness. The focus is on physics, education, social interactions and girls, strongly from the authors point of view.
Failure is Not an Option by Gene Kranz. Memoir, focusing on the U.S. space program, from Mercury through Gemini to Apollo - from the first lauch to the first manned Moon landing and more. It is especially about the Mission Control Center. To work is insightful as it shows how the space program (the preparations and missions) was operated. Marta is a real hero. The writing is sometimes unclear and overall a bit long.

Sociology

A közösség nélküli ember, Körmendy Lajos. Első (jelentősen nagyobbik felében) áttekintést ad a nyugat-európai közösségek (család, falu, céh, rend) fejlődéséről, azok szerepéről, gazdaság és személyiségformáló hatásáról. Tárgyalja az individualista-kollektivista emberek hatását a társadalom különböző rétegeiben, a modernizáció, szekularizáció, racionalizmus, kapitalizmus dinamikáját, kölcsönhatását az egyénnel és közösségeivel. A felállított modellek, az ismertetett fogalomkészlet szemléletformáló hatású. A második részben értelmezi a modern társadalom egyes jelenségeit (pl.: tömegember, divat, ellenkultúra), és becsléseket tesz a változó demográfiájú, kulturális összetételű Európa jövőjére vonatkozóan.

Software

Stay Awhile and Listen: Book I by David L. Craddock. An overview of of Condor (later Blizzard North) was founded, how Diablo I was developed. The book is mainly a big collection of quotes, compiled from interview conducted by the author with the original developers.
Stay Awhile and Listen: Book II by David L. Craddock. The story of how Diablo II was produced, with additional info about other Blizzard Entertainment games of the same time, especially Starcraft I. It is a better edited collection of quotes, but the cohesion is still pretty weak. Like the first one, this book is also way too light on technical details. The author does not assume that the reader played any of the games. I’m otherwise grateful to the author for the interviews.

Selfhosted RSS Reader

Tue, 01 Aug 2023 00:00:00 +0000

For some time I was using a simple RSS reader on my phone. However, I wanted to:

Read the same set of feeds on multiple devices
Archive the scraped content
Search the archived content

So I looked around for different options. I ruled out proprietary options, fearing their end of life (e.g: Google Reader) or gradual degradation - I wanted something stable.

The chosen architecture consists of (1) a server component, that periodically scrapes and archives the configured feeds, exports the content to authorized clients, and tracks read status (So if I read an article on one device, it’ll appear as read on all the connected devices), and (2) client programs on different devices with screens.

For the server side, I chose miniflux. It is easy to deploy (requires PostgreSQL), sufficiently fast on a Raspberry Pi4 (written in Go), and provides a pleasingly simple web interface, for desktop use. Note that miniflux offers a paid, hosted service.

For mobile, there are several options with different qualities. On Android, I found FeedMe really good. Enable the Fever API in miniflux (Settings/Integrations/Fever) to make them connect.

On iOS, NetNewsWire is an absolute treat. Enable the Google Reader API in miniflux, and use the FreshRSS option in the client to make them connect.

All of these components (miniflux, FeedMe, NetNewsWire) can also get the original HTML content and convert it to simplified text (with images), in case the content in the RSS feed is missing or incomplete. A non-trivial, and really useful feature.

The server runs on a network that doesn’t have a public IP address. I made it available to other devices using tailscale, that is also warmly recommended, and surprisingly easy to set up. It uses various tricks to bypass NAT based firewalls, and works in every scenario I tested so far.

The result is a securely available, robust, fast, clean news feed.

Considered Alternatives

FreshRSS: I didn’t like the HTML interface
Microflux: Slow and confusing UI
Miniflutt: Does not store content locally, always needs to re-fetch everything on start (no offline mode)

Why Bazel

Tue, 18 Jul 2023 00:00:00 +0000

Some time ago I was tasked with improving the build system of a large company. Requirements: build (compile and test) a large amount of C++ code (n*100k+ lines), targeting various versions of Linux and Windows, scattered in several repositories, using an enormous number of dependencies (mostly stored as binaries), mixed with Python tests, with the possibility of extending it to Java.

This post explains why Bazel was chosen for this particular task. This is not a general introduction for Bazel, nor a thorough comparison against other build systems.

The Problem

The then current build system was based on a fork of a lesser known Makefile generator. It had the following issues:

The only form of caching was with ccache, that stores object files only, and has its own limitations.
The generated Makefiles allowed one to change a dependency without triggering a re-build (this is because make uses mtime for change detection), causing hard to debug issues, causing developers to lose trust in the system, causing them to regularly make clean first, further increasing resource usage.
The development of the tool was halted, there was no hope to get C++ Modules support for it.
The build system did not cover running tests. Even if the smallest change was made in a very large repository (think monorepo), all the tests must be re-run, taking a lot of time and resources.
Because running all the tests was slow, developers skipped them, or guessed, failing in CI much later, losing time.
Multiplatform builds were time consuming and difficult, making developers to push it on the CI, and running long cycles on failure.
Getting a project and all its dependencies build with a particular set of compiler flags (e.g: -fsanitize=address) was next to impossible, turning easy-to-find bugs to terribly difficult to debug failures.

The Solution

I chose to migrate to Bazel, that offers a set of self-reinforcing benefits. To understand these benefits, we need to understand how Bazel works (supposed to work). I’m going to simplify things for brevity.

Bazel is similar to make. The user defines a set of rules, that describe how the project can be built. Each rule has a set of inputs and an expected set of outputs. The input set contains e.g: a command, that defines what should be executed to turn the inputs into the output. The input set also contains a program (e.g: a compiler) that transforms the input files into the output files. The output of one rule can be the input of another, forming a tree. To build a particular output, first the system builds the dependencies of the rule (the inputs that are outputs of other rules), then it executes the rule.

Bazel is an improvement over make. First, it enforces (tries hard to) that rules are only allowed to read their declared inputs, to produce the output, and cannot depend on anything else (e.g: system time, network, etc). A build executed in this fashion is called a Hermetic Build.

If a build is really hermetic, executing it on the same source always yields to the same output, therefore it is a Reproducible Build.

For the first full build, there’s little difference compared to make. For subsequent, incremental builds however, there are plenty.

Because of the reproducible property, if the inputs of a rule did not change, the output wouldn’t change either, therefore the rule does not need to re-run. For example, if a comment is removed, but the compiler produces the same object file, the link step will not be executed again. Bazel manages tests, similar to compilation, therefore, if the test binary (and its inputs) did not change, the test doesn’t need to re-run either.

To decide whether the input has changed (compared to the previously built output), Bazel uses the hash of the input (compared to make, that uses ctime). This fixes the correctness issues, and a key piece for distributed execution and caching.

It is possible to setup a remote executor cluster (e.g: buildbarn), that accepts rules to be executed, along with the inputs. If the rule for the given set of input was already executed, it returns the cached result, otherwise it executes the rule, caches then returns the result. This includes not just compilation, but also running the tests (where the test results are cached).

This improves the developer workflow on several points:

Fetching and building the main branch of a repository is quick, as that state is probably already built and cached.
After a change, it is to run all the affected tests, and only those, implcitly caching the results at the same time.
The CI gets also fast, because it just fetches the repository, and probably gets back the green results of the tests, produced from the previous step.
The remote executor can support multiplatform builds, saving the programmer from shipping the code manually to multiple platforms for building.

It also greatly reduces the overall resource usage.

Because of its efficiency, Bazel allows one to put more and more code (also dependencies) into a single repository, building them together. This unlocks further opportunities: fully static builds for better optimization and simplicity, build governed flags (e.g: sanitizers).

Bazel is not without difficulties: making builds hermetic, maintaining proper state (the input hash, output storage), operating the remote executor and cache, etc., is a demanding task. Therefore Bazel is better suited for larger projects with more developers. During selection, CMake was considered and evaluated, but apart from it being a popular build system, it does not solve any of the issues Bazel does.

Summary of Readings

Mon, 10 Apr 2023 00:00:00 +0000

A non-exhaustive list of recently read non-fiction books in no particular order.

Hit

Böjte Csaba füveskönyve 2., Böjte Csaba. Rövid, tömör, önmagukban megálló, bölcs gondolatok. Csaba atya férfias szeretete átsugárzik a sorain. Szülőknek és útkeresőknek kifejezetten ajánlom.

Industrial Design

The Design of Everyday Things, by Donald A. Norman. A few good ideas on how to design user interfaces (or in general, anything). Overall, the information density of this book is very low, it was a huge drag. I recommend to read a summary instead.

Psychology

Ami igazán számít by Pál Ferenc. A könyv hátáról: “Ez a könyv szándékom szerint az eddigi szakmai tanulmányaim és emberi tapasztalataim összefoglalását is jelenti. Arra vágytam, hogy amit a különböző humán területek tudásban, kutatási eredményekben kidolgoztak, az eljuthasson azokhoz, akiket érint, akiknek a legnagyobb szükségük van rá. Hozzátok. Ezért több mint tíz nagy téma került a kötetbe, reményem szerint összegyűjtve a leghasznosabb információkat egy-egy területre vonatkozóan.” Jól felépített, fontos témákat érintő, tudományos eredményeket és személyes sztorikat jól vegyítő, olvasmányos könyv. Ajánlom.
The Choice by Edith Eva Eger. Autobiography, from coming of age, through Auschwitz and other camps, to a new land. From trauma to healing, through acceptance and love. Excellent writing, with suspense and drama, pierced through by an ever positive attitude.
Men’s Search for Meaning by Viktor Frankl. A short, two parts book from a famous Holocaust survivor. In the first part, the author describes his concentration camp experiences from an analytical perspective, focusing on the behaviour of inmates and guards. The second part is a short introduction to Logotherapy, a psychotherapy school founded by the author that emphasises the importance of finding the meaning of life, and actualizing possibilities presented by it. Theories of the second part are underpinned by examples from the first part, making the book whole. It has a lot of food for thought, e.g: “It’s not what you expect from life, but what life expects from you.”.

Economy

Misbehaving: The Making of Behavioral Economics by Richard Thaler. Part professional autobiography, part the story of how behavioral economics was developed and popularized. The mainline is illustrated with memorable stories (mainly experiments and surprising incidents), generous praise of fellow behavioral economists, and an endless stream of jabs pointed at classical economists. A shorter summary focusing on the presented findings would have been appreciated to save pages: it is too shallow for a biography, and too verbose otherwise, while still thought provoking. Read Thinking, Fast and Slow first.

Honorable Mentions

Agamemnon by Aeschylus. I barely understand the English version. I also tried an older Hungarian one, with similar results. It appears that bit rot applies to literature as well.

LedgerSplit vim function

Thu, 09 Mar 2023 00:00:00 +0000

I manage my finances using ledger. Periodically, I export my bank statements and push them through a make pipeline to get a ledger file. The conversion assigns transactions to accounts based on a set of rules, e.g: If the payee is “Foo Shop”, the account is going to be “Expenses:Food”:

2023-03-09 Foo Shop
    Expenses:Food           1000 HUF
    Assets:Card

However, sometimes I buy e.g: toothpaste in “Foo Shop”, that I don’t want to eat. I do not want to contaminate my ledger, so I save the receipt and mark the item. During the periodic export, I manually fix the ledger file in vim:

2023-03-09 Foo Shop
    Expenses:Food            750 HUF
    Expenses:Health          250 HUF
    Assets:Card

Since I have an engineering degree, most of the time I get this subtraction right. For the same reason, I made a tiny vim script, that automates some of this. Standing on this line:

    Expenses:Food           1000 HUF

Issuing this command:

:LedgerSplit 100+200

Modifies the posting like this:

    Expenses:Food           700 HUF
    Expenses:Food           300 HUF

It splits the given amount from the current posting, and moves it to the next line. It handles expressions (as shown) and decimal fractions. After this I just need to fix the second account name.

Installation

To .vimrc, add:

au BufRead,BufNewFile *.ledger set filetype=ledger

To .vim/after/ftplugin/ledger.vim:

function! LedgerSplit(amount) abort
  let pat = "[0-9\\.]\\+"
  let line = getline(".")
  let old = str2float(matchstr(line, pat, 0))
  let new = substitute(printf("%.2f", old-a:amount), "\\.00$", "", "")
  let split = substitute(printf("%.2f", a:amount), "\\.00$", "", "")
  call setline(".", substitute(line, pat, new, ""))
  call append(".", substitute(line, pat, split, ""))
endfunction

command! -nargs=1 LedgerSplit call LedgerSplit(<args>)

The disgust of sequential processing of binary WebSocket messages in JavaScript

Thu, 09 Mar 2023 00:00:00 +0000

Let’s process binary WebSocket messages in the browser, in the order the server has sent them. The Writing WebSocket client applications MDN article is a reasonable place to start:

const ws = new WebSocket('ws://foo.bar/endpoint')
ws.onmessage = (event) => {
  console.log(event.data)
}

If we were to receive text messages, this’d be all we need. However, with binary messages, we’ll get:

Blob { size: 16, type: "" }

Ugh. So we need to extract the received bytes from this blob first. Also note that, about event.data MDN says:

The data sent by the message emitter; this can be any data type.

Therefore, a correct program should check if it really received a blob. Unfortunately, Blob does not provide a synchronous way to get to its data, only through an asynchronous promise (and this will be a pain later):

const ws = new WebSocket('ws://foo.bar/endpoint')
ws.onmessage = (event) => {
  event.data.arrayBuffer().then(function (ab) {
    console.log(ab)
  })
}

At this point, we have an ArrayBuffer. It still does not allow reading its data, we have to use another wrapper:

const ws = new WebSocket('ws://foo.bar/endpoint')
ws.onmessage = (event) => {
  event.data.arrayBuffer().then(function (ab) {
    const dv = new DataView(ab)
    const i = dv.getUint32(0 /* offset */, true /* little endian */)
    console.log(i)
  })
}

Finally, we managed to read a 32 bit unsigned number from the binary payload. The DataView documentation says:

The DataView view provides a low-level interface for reading and writing multiple number types in a binary ArrayBuffer, without having to care about the platform’s endianness.

Obviously, we still need to care about the endinanness of the message!

Ordering

So this was it? It wasn’t that hard! Yes, and this solution is also wrong. We wanted to process the messages in the order the server has sent them. However, while we wait for the array buffer promise that fetches our data (see: then), the onmessage handler yields to the runtime, that is free to schedule another task. It is even free to schedule another onmessage invocation, and that promise might be ready earlier. Therefore, it is possible, that we’ll process messages out of order. If the message rate is high enough, this possibility becomes a certainty.

One possible solution is to queue the promises, and process them in queue order, instead of resolution order:

const Sequencer = () => {
  const xs = []
  let x0id = 0

  return {
    read: () => {
      if (xs[0] !== undefined) {
        x0id++
        return xs.shift()
      }
      return undefined
    },
    reserve: () => {
      xs.push(undefined)
      return x0id + xs.length - 1
    },
    write: (id, x) => {
      xs[id - x0id] = x
    }
  }
}

const processNextMessage = (q) => {
  for (let msg = q.read(); msg !== undefined; msg = q.read()) {
    const dv = new DataView(ab)
    const i = dv.getUint32(0 /* offset */, true /* little endian */)
    console.log(i)
  }
}

const q = Sequencer()
const ws = new WebSocket('ws://foo.bar/endpoint')
ws.onmessage = (event) => {
  const mi = messages.reserve()
  event.data.arrayBuffer().then(function (ab) {
    q.write(mi, ab)
    processNextMessage(q)
  })
}

Conclusion

A tiny change in the requirements (binary messages instead of text messages) inflicted a great deal of complexity on the correct implementation. It is sad to see how seemingly simple tasks can turn out to be not so simple.

Sway sessions à la tmux

Fri, 20 Jan 2023 00:00:00 +0000

A window manager allows the user to organize windows of running programs on a desktop. Desktops are mapped to outputs (i.e: monitors). Most window managers support Virtual Desktops, allowing the user to move windows to separate desktops and quickly switch which desktop is shown on a given output (how quickly depends on the window manager: on popular commercial systems, not so quickly).

The used workspaces might be shown to the user in a taskbar-like fashion, e.g: on Sway:

[1] [2] [3]

Virtual desktops (or workspaces, in the i3/sway terminology) enlarge the available screen estate. By putting certain windows on well-known workspaces, one can quickly find them without cycling through all the windows by Alt+Tab-ing (or even…). For example, I always put a browser to workspace 1, an editor to workspace 2, and a shell to workspace 3. This way I can always quickly jump to any of them, by pressing Mod+1/2/3.

What happens if I have to work on two things (foo and bar) concurrently? I can open 3 more workspaces:

[1] [2] [3] [4] [5] [6]

Now this is a bit unfortunate. If the workload is dynamic enough, I’ll have a hard time quickly telling what is running where. It can be improved by renaming some workspaces, e.g:

$ swaymsg rename workspace 1 to 1:foo
$ swaymsg rename workspace 4 to 4:bar

So it’ll look like this:

[1:foo] [2] [3] [4:bar] [5] [6]

While it is certainly better, it isn’t ideal. Muscle memory doesn’t help any more: while editing the second project (workspace 5), pressing the usual Mod+3 will jump to the shell of the first project (workspace 3), not to workspace 6, that hosts the shell of the second project.

Also, if I want to add a new workspace to the working set of foo… I can’t, without shifting subsequent workspaces to the right first.

Furthermore, pressing Mod+6 with left hand only is not comfortable to me.

The famous tmux terminal multiplexer offers a solution: it calls them sessions. Each session holds a distinct set of workspaces (windows, in the tmux terminology). Sessions can be named and easily switched to. Only the workspaces of the selected session are shown, and pressing the right combination selects the numbered workspace of the active session. This is (almost) exactly what I wanted. Sway does not provide this functionality out of the box, so I created some scripts to emulate it. Let’s see:

Scripts to emulate sessions

First, there’s sway_rename_workspace.py, that simply wraps swaymsg rename workspace:

#!/usr/bin/python3

# Prompts for a name and renames the current workspace to it, ESC cancels.
# It preserves the number of the workspace - therefore the ordering.

import json
import subprocess

def ui_input(prompt):
    s = subprocess.check_output(["/usr/bin/dmenu", "-b", "-p", prompt], input="", encoding="ascii")
    return s.strip()

def focused_workspace_num():
    workspaces = subprocess.check_output(["/usr/bin/swaymsg", "-r", "-t", "get_workspaces"])
    workspaces = json.loads(workspaces)
    for ws in workspaces:
        if ws["focused"]:
            return ws["num"]
    return ""

name = ui_input("rename workspace")
num = focused_workspace_num()

subprocess.check_output(["/usr/bin/swaymsg", "rename", "workspace", "to", f"{num}:{name}"])

It prompts for a name using dmenu. Perhaps dmenu is not the most precise choice, being an X application, but it was already installed and perfectly matches my use-case.

Second, there’s sway_select_session.py, slightly more than a one-liner:

#!/usr/bin/python3

# Prompts for a workspace name, ESC cancels.
# If a name is given, it looks for the selected workspace.
# If a workspace is found, it moves that workspace, and every
# workspace to the right -- until a different named-workspace is found --
# to the beginning, and moves every other workspace to the end,
# starting at workspace number 10.
# If no workspace matches the given name, it will be created,
# and every other workspace will be moved to the end.
# It does not touch workspaces given in `protected_workspaces`.

import json
import subprocess

# do not move these workspaces
protected_workspaces = set([1, 9])

def ui_input(prompt, selection):
    s = subprocess.check_output(["/usr/bin/dmenu", "-b", "-p", prompt], input=selection, encoding="ascii")
    return s.strip()

def strip_before(s, d):
    a,_,b = s.partition(d)
    return b or a

def selected_range(workspaces, s):
    first = 0
    for ws in workspaces:
        if ws["name"] == s:
            break
        first += 1
    last = first + 1
    for ws in workspaces[last:]:
        if ':' in ws["name"]:
            break
        last += 1
    return first, last

def renumber(i, ws):
    tag = ws["name"].partition(":")[2]
    return f"{i}:{tag}" if tag else str(i)

# select a session, find the range of workspaces that belongs to it
workspaces = subprocess.check_output(["/usr/bin/swaymsg", "-r", "-t", "get_workspaces"])
workspaces = json.loads(workspaces)

sessions = []
for ws in workspaces:
    session = ws["name"].partition(":")[2]
    if session:
        sessions.append(ws["name"])

selected_session = ui_input("select session", "\n".join(sessions))
first, last = selected_range(workspaces, selected_session)

# move selected worspaces to si..., deselected workspaces to di...
si = 1
di = max(last-first, 10)

renames = []

for (i, ws) in enumerate(workspaces):
    if ws["num"] in protected_workspaces:
        continue
    if first <= i and i < last:
        while si in protected_workspaces:
            si += 1
        renames.append((ws["name"], renumber(si, ws)))
        si += 1
    else:
        while di in protected_workspaces:
            di += 1
        if len(renames) == 0 and not ":" in ws["name"]:
            # first deselected workspace is unnamed, name it "main" to avoid
            # merging it with the selected session
            renames.append((ws["name"], str(di) + ":main"))
        else:
            renames.append((ws["name"], renumber(di, ws)))
        di += 1

# move them to tmp names to avoid clashes
for (src, dst) in renames:
    subprocess.call(["/usr/bin/swaymsg", "rename", "workspace", src, "to", f"_{dst}"])

# move them back
for (_, dst) in renames:
    subprocess.call(["/usr/bin/swaymsg", "rename", "workspace", f"_{dst}", "to", dst])

# finally, jump to the leading workspace if the selected session
# (this also creates the session if it is a new one)
tag = strip_before(selected_session, ":")
i = 1
while i in protected_workspaces:
    i += 1

subprocess.call(["/usr/bin/swaymsg", "--", "workspace", "--no-auto-back-and-forth", f"{i}:{tag}"])

So it prompts for a name (a session name), identifies the workspaces that belong to that session, moves them to the beginning, and moves the others to the end. Let’s see an example. We start with:

[1:foo] [2] [3] [4:bar] [5] [6]

Then when we select “4:bar”, we’ll end up with:

[1:bar] [2] [3] [10:foo] [11] [12]

…where “1:bar” was “4:bar”, “2” was “5”, “3” was “6”, “10:foo” was “1:foo”, “11” was “2” and “12” was “3”. This new layout allows muscle memory to kick in, keeps frequently used workspace numbers small, and also adds a gap between the first and the other sessions: if needed, workspace 4 can be quickly added to the bar session.

A personal twist appears here: 1 and 9 are protected_workspaces, virtually part of every every session – I never want to move them. I keep there communication tools, a journal, a music player, etc.

Finally, if you want to stop sessioning, there’s sway_flatten_sessions.py:

#!/usr/bin/python3

# Assign each workspace the lowest number still available.

import json
import subprocess

# do not move these workspaces
protected_workspaces = set([9])

renames = []
index = 1

workspaces = subprocess.check_output(["/usr/bin/swaymsg", "-r", "-t", "get_workspaces"])
workspaces = json.loads(workspaces)
for ws in workspaces:
    if ws["num"] in protected_workspaces:
        continue
    session = ws["name"].partition(":")[2]
    if session:
        renames.append((ws["name"], f"{index}:{session}"))
    else:
        renames.append((ws["name"], f"{index}"))
    index = index + 1
    while index in protected_workspaces:
        index = index + 1

for rn in renames:
    subprocess.call(["/usr/bin/swaymsg", "rename", "workspace", rn[0], "to", rn[1]])

This simply renumbers the workspaces in order, putting them next to each other, removing any gap. To tie them together, I’m using the following sway piece of config:

set $mode_session session [s]elect [r]ename [f]latten
bindsym $mod+s mode "$mode_session"
mode "$mode_session" {
    bindsym s exec --no-startup-id ~/bin/way_select_session.py, mode "default"
    bindsym r exec --no-startup-id ~/bin/way_rename_workspace.py, mode "default"
    bindsym f exec --no-startup-id ~/bin/way_flatten_sessions.py, mode "default"

    bindsym Escape mode "default"
    bindsym Return mode "default"
}

This allows very quick session renaming (Mod+sr) and selection (Mod+ss). The presented code is also available on GitHub.

Summary of Readings

Tue, 27 Dec 2022 00:00:00 +0000

A list of non-fiction books in no particular order I read recently, with some recommendation. This post was inspired by the similar posts of Eli Bendersky.

Energy

How to Drive a Nuclear Reactor by Colin Tucker. A high level tour for laymans on how a specific kind of nuclear reactors (PWR) work. Goes from the reactor, through the two circuits, to the turbine and the grid. Deftly written, makes you believe you actually understand what’s going on, but still highlights the enormous complexity of such an installation. While reading, I recommend to keep a list of abbreviations, they come up frequently, but only explained once.
The Future Of Fusion Energy by Jason Parisi, Justin Ball. A long introduction to fusion energy, goals and challenges, prospects. The book covers the past, present, and the expected future of the field (no, we’ll not expected to have commercial fusion reactors at least until 2100, sorry). It attempts to explain things without mathematics. The authors could use a bit fewer footnotes¹. The second chapter of the book could stand on its own: a comprehensive, but accessible, list of energy sources of the Earth.

Environment

Limits To Growth by Donella H. Meadows, Dennis L. Meadows, Jørgen Randers. A long time ago, a group of scientists built a computer model to guess how the apparently limitless growth of the economy will affect the future of humankind. The results were published in 1972, urging profound and immediate changes. Not much change was made. The 30 year update of the book confirms that we are on track to our predicted doom, and again proposes reduction of consumption, waste, greed; a fair society and more birth control as a solution. A depressing read, forecasting the untimely death of billions in this century.

Software

The Making of Prince of Persia by Jordan Mechner. Relevant journal entries verbatim, from the author of the famous computer game, around 1985-1993. Shows how Jordan struggles between game development and screenwriting, and the fascinating self-doubt of an extremely talented and professionally successful youth. Easy to read, exciting, inspirational.
The Making of Karateka by Jordan Mechner. A prequel to The Making of Prince of Persia, this time about a less well-known game, Karateka. Similarly cute and easy to read. The amount and quality of output this young boy produces is fascinating.
Masters of Doom by David Kushner. A documentary on how the computer game developer company id was founded and operated, from Commander Keen through Doom to Quake I-II, mostly focusing on John Carmack and John Romero, their different personalities and motivations – without technical details.

These (sometimes) (slightly) funny footnotes do not really add value, but break the flow of reading. ↩︎

Summary of Readings

Thu, 08 Sep 2022 00:00:00 +0000

A non-exhaustive list of non-fiction books in no particular order I read recently, with some recommendation. This post was inspired by the similar posts of Eli Bendersky.

Engineering

Kelly: More Than My Share of It All by Clarence “Kelly” Johnson. The autobiography of a aeronautical engineer genius, the founder of the Skunk Works. Concise and honest, inspirational.
Skunk Works: A Personal Memoir of My Years at Lockheed by Ben Rich. A story of the development the U2, Have Blue and F-117, the SR-71 Blackbird and more. Filled with personal anecdotes, not too technical but still informational. A tale of amazing engineering prowess.
Mars Rover Curiosity: An Inside Account from Curiosity’s Chief Engineer by Rob Manning. An other story of an elite engineering group, told by a lead engineer. A super expensive project with tight deadlines, much excitement, and no ordinary problems.
Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age by Michael Hiltzik. The history of how the PARC research center was founded, the inventions that came out of it, and how Xerox capitalized on this precious gem. Somewhat longer, chapters dedicated to distinct researchers, management being the overarching cohesion.
The Soul of a New Machine by Tracy Kidder. A detailed account of how a particular computer was developed in the late 1970s in the U.S, mostly by enthusiast, young professionals – with a twist at the end. The author is a master of his craft (journalism).

Software

UNIX: A History and a Memoir by Brian Kernighan. A very interesting story of how the original Unix was invented - as seen by Brian, and how now-familiar tools were developed alongside. Recommended for the users of UNIX derivative systems, and tools like bash, make, awk, etc.
Just for Fun: The Story of an Accidental Revolutionary, by Linus Torvalds and David Diamond. An easy, funny, part auto-biography book about Linus Torvalds, and the early years of Linux. Not too technical, but still interesting and insightful.
A Philosophy of Software Design by John Ousterhout. An attempt to present abstract software design ideas, generic rules? My takeaway: shallow abstractions are wrong. Otherwise, not so much, unfortunately.

History

The Art of War by Sun Tzu. Written in 5th century BC, a collection of military maxims, commented by later interpreters, then again translated and commented by many, including Lionel Giles.
Fermat’s Last Theorem by Simon Singh. The author tells how the theorem was finally solved, starting with the ancient greeks and ending up at modern mathematics. Unfortunately, to remain comprehensible for casual readers, there are not much mathematical details. I think this is not the authors fault – modern mathematics simply transcended most of us.

Thinking, Fast and Slow by Daniel Kahneman. A collection of important psychological phenomena, that helps to understand why do we feel, think, decide, and get things wrong the way we do. Very long, but extremely insightful.
The Formula: The Universal Laws of Success by Albert-László Barabási. An analysis of how valuation of achievements at the highest level does (not) work. An interesting, data driven, simple message, in a bit lengthy and sensationalist form.

Money and Economy

A Man for All Markets by Edward O. Thorp. An autobiography telling a story of how a poor, but super intelligent boy discovers the world, and denies the “it cannot be done” sentiment. From university through Las Vegas to Wall Street. Recommended for aspiring quants and hedge fund operators.
Flash Boys: A Wall Street Revolt by Michael Lewis. A fun, easy to read, short story about high frequency trading in the US, based on actual events. Unfortunately, some argue it is incorrect, factually wrong, written in bad faith, or simply an ad of a specific exchange. There’s even a rebuttal book. Still, it is a fun read.

Parenting

How to Talk So Kids Will Listen & Listen So Kids Will Talk by Adele Faber, Elaine Mazlish. A great book with practical tips on difficult but everyday situations when dealing with children. Half of the book is tips - just the right amount, half of it is case-studies. The presented techniques do work. Highly recommended for parents with small children.
Raising An Emotionally Intelligent Child by John Gottman. The author argues that understanding, reflecting and labeling the child’s emotions is beneficial in may ways. True. The book information/letter ratio is a bit low.

The Unreasonable Effectiveness of Doing Nothing

Thu, 08 Sep 2022 00:00:00 +0000

Static storage shared between compilation units

Wed, 30 Mar 2022 00:00:00 +0000

Suppose you want to create a static container compilation time, that can be inserted into by different translation units, and can be iterated runtime. For example:

// a.cpp
void a() { STORE("names", "alice") }

// b.cpp
void b() { STORE("names", "bob") }

// c.cpp
void c() { STORE("names", "charlie") }

int main() {
  for (auto name : fetch("names"))
  {
    std::cout << name << ",";
    // prints alice,bob,charlie, in unspecified order
  }
}

I know of no way in standard C or C++ that’d allow this. However, with platform specific tools, it is possible to implement STORE and fetch - this post describes how. The complete source code is available in the companion repository.

Theory

The following discussion is constrained to the linux, macOS and windows platforms. Each platform in scope has a mainstream execution format, that defines how an executable binary should look like. ELF for linux, Mach-O for macOS, PE for windows. Each of these formats define headers (metadata) and sections (data). Sections can contain e.g: program code, exported symbol information debug symbols, or even arbitrary data. We are going to create and populate a custom section (STORE), and read to contents of that section runtime (fetch).

Put Data into Custom Sections

On each platform, a different construct can be used to put a given static variable into a specific section:

linux/ELF: section attribute:
```
#define STORE(name, str) {               \
  __attribute__((section(name), used))   \
  static constexpr const char s[] = str; \
}
```
The used attribute instructs the compiler not to remove s, even if it appears unused. The curly braces around the definition prevent name conflicts (e.g: if there are multiple STORE invocations in the same scope).

macOS/Mach-O: sections are grouped into segments, the segment name must be also specified to the attribute:

#define STORE(name, str) {                               \
  __attribute__(("__DATA_CONST," section(name), used))   \
  static constexpr const char s[] = str;                 \
}

windows/PE: allocate specifier. I couldn’t find the equivalent of the used attribute, so you have to convince the compiler otherwise, that the variable is used (e.g: by actually using it) - not shown here. Also each section need to be initialized first with a const_seg pragma. The pragma must be in global/namespace scope, it can’t be added to the STORE macro.
```
#define STORE(name, str) {               \
  __declspec(allocate(name))             \
  static constexpr const char s[] = str; \
}

#pragma const_seg("names") // user code, required once for each segment/translation unit
```

This definition makes it possible to use STORE inside functions. Similarly, a STORE_GLOBAL macro can be created, that works in global/namespace scope.

One might be tempted to prefix the custom section name with a leading dot (ELF/PE) or double underscore (Mach-O) to match conventions of the well known sections (e.g: .text or .rodata). However, that’d be incorrect: the leading decoration is there to separate platform specified sections from user defined custom sections. For example:

Section names with a dot (.) prefix are reserved for the system, although applications may use these sections if their existing meanings are satisfactory. Applications may use names without the prefix to avoid conflicts with system sections.
Executable and Linkable Format, 1-16

To verify that STORE works, create a test binary that invokes it, and run a platform specific tool that dumps the contents of a specified section of the specified binary:

linux/ELF: readelf: readelf -x names binary
macOS/Mach-O: otool: otool -s __DATA_CONST names binary -V
windows/PE: dumpbin: dumpbin.exe /SECTION:names binary /RAWDATA

The strings we invoked STORE with should appear in the output of these commands.

GCC Complications

All trivial so far. Unfortunately, if STORE is used in functions with different linkage (e.g: in a normal function and in an inline or template function), GCC will produce a section type conflict. The situation is analyzed in a stackoverflow question.

The issue stems from the fact that GCC must give special care to some function local static variables:

Function-local static objects in all definitions of the same inline function (which may be implicitly inline) all refer to the same object defined in one translation unit, as long as the function has external linkage.
cppreference

To ensure uniqueness, those variables are put into a comdat section, other variables are not, and GCC is unable to resolve this situation. To avoid GCC messing up the section type, it has to be specified explicitly. Unfortunately, it cannot be specified using the section attribute, and I found no way to convince GCC to forget about the standard requirements, so inline assembly needs to be used:

#define STORE(name, str)                              \
  __asm__ (                                           \
    ".pushsection \"" name "\",\"?\",@progbits" "\n"  \
    ".asciz \"" str "\""                        "\n"  \
    ".popsection"                               "\n"  \
  );

The .pushsection and .asciz assembler pseudo directives are used to put a string value into a specific section, without actually creating a variable that is restricted by C++ rules. The quoting mess is a bit scary, but there’s nothing too complicated here.

Get data from custom sections

We have a STORE macro, that takes two string literals, and puts one into the section indicated by the other. Now we only need to extract the data runtime that was inserted compile time. To do that, we just need to replicate a small portion of what readelf/otool/dumpbin does. The implementation is platform specific, let’s see the linux/ELF case for illustration:

// Get every string STOREd in `name` in the binary at `path`.
std::vector<std::string> fetch(
  const char* name,
  const char* path = current_binary_path())
{
  // Create a random access view of `path` (error handling on read included)
  const MemoryMappedFile f(path);

  // Read the ELF header.
  // ElfW(Ehdr) expands to Efl32_Ehdr or Elf64_Ehdr,
  // according to the current platform.
  ElfW(Ehdr) ehdr;
  f.read(0, sizeof(ehdr), &ehdr);

  // Read the section headers from the location
  // indicated by the ELF header.
  std::vector<ElfW(Shdr)> shdrs(ehdr.e_shnum);
  f.read(ehdr.e_shoff, shdrs.size() * sizeof(ElfW(Shdr)), shdrs.data());

  // The section headers do not contain their name,
  // only an offset into the section header string table.
  // Locate this string table:
  const ElfW(Off) string_table_offset = shdrs.at(ehdr.e_shstrndx).sh_offset;

  std::vector<std::string> result;

  // For each section
  for (const ElfW(Shdr)& shdr : shdrs)
  {
    // Get the section name from the string table
    const std::string_view section_name = f.string(string_table_offset + shdr.sh_name);
    if (section_name == name)
    {
      // This is the section we are looking for.
      // Get all the strings.
      for (ElfW(Xword) offset = 0; offset < shdr.sh_size;)
      {
        const std::string_view str = f.string(shdr.sh_offset + offset);
        result.emplace_back(str);
        offset += str.size() + 1;
      }
      // no break or early return here.
      // section names do not need to be unique.
    }
  }

  return result;
}

The full source is available in the companion repository.

Use case

This looks cool (?), but why? The binlog high performance log library uses a more advanced variation of this technique to store the metadata of log invocations in the binary, avoiding one-time setup for each invocation that’d be otherwise necessary.

Preprocessor foreach compile time

Mon, 07 Mar 2022 00:00:00 +0000

Assume you want to create a preprocessor macro that takes a macro F and a variable number of arguments. When expanded, it invokes F with each argument:

#define FOREACH(F, ...) //...

FOREACH(G, a, b, c) // expands to: G(a) G(b) G(c)

Such a macro can be useful for example for a log library. There are several ways to implement it. Here, we compare three different implementations, with regard to complexity, ease of change, and compile time.

The complete source code used for testing is available at erenon/pp-foreach-comparison.

Contenders

The first implementation is taken from binlog. It consists of a macro that counts the number of arguments, and N foreach macros, one for each arity. It is generated by a python script, binlog.py. N is a parameter of the generator script, here, N=512. This version is slightly different from the original one used in binlog, to match the capabilities of the other contenders, for a fair comparison.

import sys

N = 512

print('#define CAT(a, b) CAT_I(a, b)')
print('#define CAT_I(a, b) a ## b')

sys.stdout.write('#define COUNT(...) COUNT_I(__VA_ARGS__')
for i in reversed(range(N)):
    sys.stdout.write(f',{i}')
print(')')

sys.stdout.write('#define COUNT_I(')
for i in range(1, N+1):
    sys.stdout.write(f'a{i},')
print(f'...) a{N}')

print('#define FOREACH(F, ...) CAT(FOREACH_, COUNT(__VA_ARGS__)) (F, __VA_ARGS__)')

for i in range(1, N+1):
    sys.stdout.write(f'#define FOREACH_{i}(F')
    for j in range(i):
        sys.stdout.write(f',a{j}')
    sys.stdout.write(')')
    for j in range(i):
        sys.stdout.write(f' F(a{j})')
    sys.stdout.write("\n")

For N=3, it looks like this:

#define CAT(a, b) CAT_I(a, b)
#define CAT_I(a, b) a ## b
#define COUNT(...) COUNT_I(__VA_ARGS__,2,1,0)
#define COUNT_I(a1,a2,a3,...) a3
#define FOREACH(F, ...) CAT(FOREACH_, COUNT(__VA_ARGS__)) (F, __VA_ARGS__)
#define FOREACH_1(F,a0) F(a0)
#define FOREACH_2(F,a0,a1) F(a0) F(a1)
#define FOREACH_3(F,a0,a1,a2) F(a0) F(a1) F(a2)

Disclaimer: I’m the author of binlog.

The second implementation is taken from swansontec. It is a complex, but documented solution. It works with up to N=364 arguments. The idea can be expanded to support more arguments. The entry point of the solution is renamed to FOREACH in the snippet below:

/* Created by William Swanson in 2012. Public Domain. */

#define EVAL0(...) __VA_ARGS__
#define EVAL1(...) EVAL0(EVAL0(EVAL0(__VA_ARGS__)))
#define EVAL2(...) EVAL1(EVAL1(EVAL1(__VA_ARGS__)))
#define EVAL3(...) EVAL2(EVAL2(EVAL2(__VA_ARGS__)))
#define EVAL4(...) EVAL3(EVAL3(EVAL3(__VA_ARGS__)))
#define EVAL(...)  EVAL4(EVAL4(EVAL4(__VA_ARGS__)))

#define MAP_END(...)
#define MAP_OUT
#define MAP_COMMA ,

#define MAP_GET_END2() 0, MAP_END
#define MAP_GET_END1(...) MAP_GET_END2
#define MAP_GET_END(...) MAP_GET_END1
#define MAP_NEXT0(test, next, ...) next MAP_OUT
#define MAP_NEXT1(test, next) MAP_NEXT0(test, next, 0)
#define MAP_NEXT(test, next)  MAP_NEXT1(MAP_GET_END test, next)

#define MAP0(f, x, peek, ...) f(x) MAP_NEXT(peek, MAP1)(f, peek, __VA_ARGS__)
#define MAP1(f, x, peek, ...) f(x) MAP_NEXT(peek, MAP0)(f, peek, __VA_ARGS__)

#define MAP_LIST_NEXT1(test, next) MAP_NEXT0(test, MAP_COMMA next, 0)
#define MAP_LIST_NEXT(test, next)  MAP_LIST_NEXT1(MAP_GET_END test, next)

#define MAP_LIST0(f, x, peek, ...) f(x) MAP_LIST_NEXT(peek, MAP_LIST1)(f, peek, __VA_ARGS__)
#define MAP_LIST1(f, x, peek, ...) f(x) MAP_LIST_NEXT(peek, MAP_LIST0)(f, peek, __VA_ARGS__)

/**
 * Applies the function macro `f` to each of the remaining parameters.
 */
#define FOREACH(f, ...) EVAL(MAP1(f, __VA_ARGS__, ()()(), ()()(), ()()(), 0))

/**
 * Applies the function macro `f` to each of the remaining parameters and
 * inserts commas between the results.
 */
#define MAP_LIST(f, ...) EVAL(MAP_LIST1(f, __VA_ARGS__, ()()(), ()()(), ()()(), 0))

The third implementation is taken from hobbes. It is a complex solution. It works with up to N=513 arguments, probably it can be changed to support more. The entry point of the solution is renamed to FOREACH in the snippet below:

// Copyright 2016 Morgan Stanley
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
// https://github.com/morganstanley/hobbes
#define HOBPP_FIRST(a, ...) a
#define HOBPP_SECOND(a, b, ...) b
#define HOBPP_CAT(a,b) a ## b
#define HOBPP_IS_PROBE(...) HOBPP_SECOND(__VA_ARGS__, 0)
#define HOBPP_PROBE() ~, 1
#define HOBPP_NOT(x) HOBPP_IS_PROBE(HOBPP_CAT(HOBPP_SNOT_, x))
#define HOBPP_SNOT_0 HOBPP_PROBE()
#define HOBPP_BOOL(x) HOBPP_NOT(HOBPP_NOT(x))
#define HOBPP_IF_ELSE(condition) HOBPP_SIF_ELSE(HOBPP_BOOL(condition))
#define HOBPP_SIF_ELSE(condition) HOBPP_CAT(HOBPP_SIF_, condition)
#define HOBPP_SIF_1(...) __VA_ARGS__ HOBPP_SIF_1_ELSE
#define HOBPP_SIF_0(...)             HOBPP_SIF_0_ELSE
#define HOBPP_SIF_1_ELSE(...)
#define HOBPP_SIF_0_ELSE(...) __VA_ARGS__
#define HOBPP_EMPTY()
#define HOBPP_EVAL(...) HOBPP_EVAL256(__VA_ARGS__)
#define HOBPP_EVAL256(...) HOBPP_EVAL128(HOBPP_EVAL128(__VA_ARGS__))
#define HOBPP_EVAL128(...) HOBPP_EVAL64(HOBPP_EVAL64(__VA_ARGS__))
#define HOBPP_EVAL64(...) HOBPP_EVAL32(HOBPP_EVAL32(__VA_ARGS__))
#define HOBPP_EVAL32(...) HOBPP_EVAL16(HOBPP_EVAL16(__VA_ARGS__))
#define HOBPP_EVAL16(...) HOBPP_EVAL8(HOBPP_EVAL8(__VA_ARGS__))
#define HOBPP_EVAL8(...) HOBPP_EVAL4(HOBPP_EVAL4(__VA_ARGS__))
#define HOBPP_EVAL4(...) HOBPP_EVAL2(HOBPP_EVAL2(__VA_ARGS__))
#define HOBPP_EVAL2(...) HOBPP_EVAL1(HOBPP_EVAL1(__VA_ARGS__))
#define HOBPP_EVAL1(...) __VA_ARGS__
#define HOBPP_DEFER2(m) m HOBPP_EMPTY HOBPP_EMPTY()()
#define HOBPP_HAS_PARGS(...) HOBPP_BOOL(HOBPP_FIRST(HOBPP_SEOAP_ __VA_ARGS__)())
#define HOBPP_SEOAP_(...) HOBPP_BOOL(HOBPP_FIRST(HOBPP_SEOA_ __VA_ARGS__)())
#define HOBPP_SEOA_() 0
#define FOREACH(f, VS...) HOBPP_EVAL(HOBPP_MAPP(f, VS))
#define HOBPP_MAPP(f, H, T...)        \
  f(H)                                \
  HOBPP_IF_ELSE(HOBPP_HAS_PARGS(T))(  \
    HOBPP_DEFER2(HOBPP_SMAPP)()(f, T) \
  )(                                  \
  )
#define HOBPP_SMAPP() HOBPP_MAPP

Compile Time Comparison

To measure the time it takes to preprocess a solution, for each solution, a source file is generated, that includes the implementation, and invokes FOREACH with M arguments, 32 times. Then the time the compiler spends in user space, while preprocessing the file is measured. Finally, measured time is plotted against the number of arguments:

The measurement was done using gcc 11.1.0 and clang 13.0.0, on Linux 5.16, running on a x86_64 AMD machine. From the plots we can infer that:

Preprocessing binlog takes a small, constant amount of time, regardless the number of FOREACH arguments.
The preprocessing time of swansontec and hobbes linearly increases as the number of arguments increases. The hobbes solution is roughly two times slower than swansontec.
There is no meaningful case where swansontec or hobbes would be faster than binlog.
clang is almost three times slower to preprocess these files, than gcc.

Implementation Comparison

The three solutions are also different implementation wise. I’m giving here a comparison along a mix of subjective and objective axes:

	binlog	swansontec	hobbes
Complexity	low	high	high
Extensibility	easy	difficult	difficult
Documentation	no	yes	yes
File Size (N<=512)	1.6 MB	2 kB	2 kB
Generated Code	yes	no	no
License	Apache 2	Public Domain	Apache 2

By Complexity, I mean the effort required to understand and verify the correctness of the implementation. By Extensibility, I mean the effort required to change the behavior of the implementation, e.g: support 0 arguments, pass along a constant data value, increase N, inject a token between F calls, etc.

Verdict

Interestingly, less code doesn’t mean faster to compile (preprocess) code. Despite that the generated source code of the binlog solution is almost a 1000 times bigger, it is still favourable compile time performance wise. Furthermore, the straightforward, simple solution is easier to understand and modify. Therefore, if code generation or having large source files is not an issue, I see no reason to use a more complicated solution.

Snapcast on Demand

Mon, 21 Feb 2022 00:00:00 +0000

Snapcast is a wonderful piece of software: a multiroom client-server audio player, where all clients are time synchronized with the server to play perfectly synced audio. It allows you to play the same music in different rooms, at the same time, using multiple devices.

Normally, I listen to music on my desktop. Sometimes, however, I’d like to broadcast music to another room. In that room, there’s a raspberry pi connected to a set of speakers. I wanted to make the switch between local and multiroom playback as easy as possible. This post describes my approach.

Normally, my music player uses the pulse-audio interface to interact with the audio system (PipeWire, currently) that uses ALSA to talk to the actual device:

cmus -> pulse -> pipewire -> alsa -> device

We are going to add a virtual sink to pulse audio, a file sink, where the Snapcast server will pick up the music from:

cmus -> pulse -> file -> snapcast server -> snapcast client -> alsa -> device

To this chain, we can add multiple snapcast clients, to achieve a multiroom sound system. It is important that the snapcast client should not use the pulse audio interface, to avoid infinite loops. Read more about these pieces in this post from badaix. By changing the active pulse sink, we can switch between local and multiroom audio. The advantage of this setup is that it is player agnostic: it works with every music player that can talk to pulse, and requires no additional player configuration.

Theory aside, let’s get to work. First, build snapcast from AUR, with a change to minimize dependencies:

diff --git a/PKGBUILD b/PKGBUILD
index e5033a8..5efe228 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -7,7 +7,7 @@ pkgdesc="Synchronous multi-room audio player"
 arch=('x86_64' 'armv6h' 'armv7h' 'aarch64')
 url="https://github.com/badaix/snapcast"
 license=('GPL')
-depends=(alsa-lib avahi libvorbis flac opus expat libsoxr libpulse)
+depends=(alsa-lib)
 makedepends=(cmake alsa-utils boost)
 install="snapcast.install"
 backup=('etc/default/snapserver' 'etc/default/snapclient' 'etc/snapserver.conf')
@@ -25,6 +25,12 @@ build() {
     cmake -B build -S . \
           -DCMAKE_BUILD_TYPE=None \
           -DCMAKE_INSTALL_PREFIX=/usr \
+          -DBUILD_WITH_FLAC=Off \
+          -DBUILD_WITH_VORBIS=Off \
+          -DBUILD_WITH_TREMOR=Off \
+          -DBUILD_WITH_OPUS=Off \
+          -DBUILD_WITH_AVAHI=Off \
+          -DBUILD_WITH_EXPAT=Off \
           -Wno-dev
     make -C build
 }

Create systemd units:

# .config/systemd/user/snapserver.service
[Unit]
Description=Snapcast server
Documentation=man:snapserver(1)
After=network-online.target time-sync.target

[Service]
ExecStart=/usr/bin/snapserver --logging.sink=system --server.datadir=/tmp/ --stream.codec=pcm --http.enabled=false --tcp.enabled=false
Restart=on-failure

# .config/systemd/user/snapclient.service
[Unit]
Description=Snapcast client
Documentation=man:snapclient(1)
After=network-online.target time-sync.target sound.target

[Service]
ExecStart=/usr/bin/snapclient --logsink=system -h my_desktop_hostname
Restart=on-failure

Copy the snapclient service file to the raspberry pi. To allow running user services via ssh, enable PAM:

# /etc/ssh/sshd_config
UsePAM yes

Add the virtual pulse audio sink:

# .config/pulse/default.pa
load-module module-pipe-sink file=/tmp/snapfifo sink_name=Snapcast format=s16le rate=48000
set-default-sink alsa_output.pci-0000_00_14.2.analog-stereo

Create a script that switches to the virtual sink and starts the required services, then undoes everything on exit:

#!/bin/sh

set -xeu

LOCAL_SINK=$(pactl get-default-sink)

trap cleanup EXIT INT

cleanup() {
  ssh my_remote_host 'systemctl --user stop snapclient.service'
  systemctl stop --user snapserver.service snapclient.service
  pactl set-default-sink $LOCAL_SINK
}

systemctl start --user snapserver.service snapclient.service
ssh my_remote_host 'systemctl --user start snapclient.service'
pactl set-default-sink Snapcast

read

I decided not to run snapclient on the pi all the time: if the server is down (i.e: I’m not using it, or the desktop is powered off), it tries to reconnect every second, as expected, and I considered that a waste of resources.

To avoid systemd stopping the remote service after ssh disconnects, on the remote machine, do:

$ sudo loginctl enable-linger $USER

Review of Thriving in a Crowded and Changing World

Sun, 09 Jan 2022 00:00:00 +0000

Thriving in a Crowded and Changing World is a bulky paper written by C++ author Bjarne Stroustrup for the History of Programming Languages ACM conference. I collected a few quotes below that I found interesting.

I wanted that tool to write a version of the Unix kernel that could work on multiple processors connected through a local area network or a shared memory.

What happened to that vision?

In the late 1980s, the power of computers increased dramatically, I added Templates.

This solved the too much computer power issue for a very long time.

iostreams – an elaboration by Jerry Schwartz and the standards committee of my simple 1984 streams library to handle a wide variety of character types, locales, and buffering strategies.

I wonder what he means by “elaboration”…

I was highly amused to find that representatives from Intel and IBM effectively vetoed that idea by pointing out that by adopting the Java memory model for C++ we would slow down all JVMs by a factor of at least two. Consequently, to preserve the performance of Java, we had to adopt a far more complex model for C++. Ironically and predictably, C++ was then criticized for having a more complicated memory model than Java.

Unfortunately, I couldn’t find a reference for that. I admire tough that the committee has such high ethics.

This is not just “design by committee;” it is “design by a confederation of committees.”

I consider the threads-and-locks level of concurrency the worst model for application use of concurrency

Why did we require that programmers should use constexpr to mark functions that can be executed at compile time? In principle, the compiler can figure out what can be computed at compile time, but without an annotation, users would be at the mercy of variations in cleverness of compilers and a compiler would need to keep bodies of all functions around “forever” in case they were needed for constant expression evaluation

I always wondered, didn’t know that. Unfortunately, I still do not really understand why.

Unfortunately, beginners flock to examine the most horrendously specialized code and take great pride in explaining it (often erroneously) to others. Bloggers and speakers enhance their reputations by showing off hair-raising examples. This is a major source of C++’s reputation for complexity.

I don’t get what’s wrong with using swallow = int[]; (void)swallow{0, (f(t), int{})...};. Rolls off the tongue.

On exception specifications:

As I feared, that led to maintenance problems, run-time overhead as an exception was repeatedly checked along the unwinding path, and bloated source code. For C++11, exception specifications were deprecated and for C++17, we finally removed them (unanimously).

Thank you guys for using using namespace std;, forcing the committee to chose the technically correct names.

They are called unordered (e.g., unordered_map) to distinguish them from the older, ordered, standard containers (e.g., map) and because the obvious names (e.g., hash_map) had been used extensively in other libraries before C++11.

On digit separators:

Many, including me, argued for using the underscore as a separator (as in several other languages). Unfortunately, people were using the underscore as part of user-defined literal suffixes

Past forced mistakes:

The prefix template<...> syntax was not my first choice when I designed templates. It was forced upon me by people worried that templates would be misused by less competent programmers, leading to confusion and errors. The heavy syntax for exception handling, try { ... } catch( ... ) { ... }, was a similar story.

The bashing of C++17 is a recurring theme:

I have not been able to identify any unifying themes for the C++17 features. These features look to be merely a set of “bright ideas” thrown into the language and standard library as voting majorities could be found.

Now I understand why:

auto {x ,y ,z} = f (); // proposed
auto [x ,y ,z] = f (); // accepted

This is a minor change, but I think a mistake.

Regarding optional/variant/any:

I see these three variants of the idea of a discriminated union as a stop-gap measure. Functional- programming-style pattern matching is a far more elegant, general, and potentially more efficient solution to the problems with unions.

I consider operator dot a prime example of the members of the committee lacking a shared view of what C++ is and what it should become. Thirty years, six proposals, many discussions, much design and implementation work, and we have nothing.

In retrospect, I don’t think that the object-oriented notation (e.g., x.f(y)) should ever have been introduced. The traditional mathematical notation f(x,y) is sufficient. As a side benefit, the mathematical notation would naturally have given us multi-methods, thereby saving us from the visitor pattern workaround.

Regarding the spaceship operator <=>:

So, instead of a facility that made trivial examples work as expected for novices, we got an elaborate facility that allows experts to carefully craft subtle comparisons.

Expectations:

Coroutines - should be very fast and simple.

At this point in the paper, you just know the final result:

a complete explanation of the proposals written, presented, and discussed is beyond the scope of this paper. Here, I present only an outline. There are simply too many complex details for anything else; the papers alone run into many hundreds of pages

The conclusion was that it might be possible to get the best of both approaches, but that would require serious study. That study took years, yielded no clear results, and in the meantime more proposals emerged.

How dare you, Dennis:

At the time, I heard the priceless comment: “Dennis isn’t a C expert; he never comes to the meetings”.

The only thing that increases faster than hardware performance is human expectation.

I don’t think this is true. The perceived latency of computers just got worse in general in the past decades. I blame bloated software. Humans expect a fluent user interface.

Nextcloud on RPi4 with Arch Linux

Tue, 12 Oct 2021 00:00:00 +0000

I experimented with Nextcloud 22 on a Raspberry Pi 4 8 GB, running the aarch64 build of Arch Linux ARM. I found the setup too slow to be useful: even with caching, switching between tabs on the web interface takes seconds, while top shows php-fpm consuming a lot of CPU. Either further tuning would be required, or this hardware is simply too slim for this kind of workload, or more efficient software needs to be produced.

For the reference, I share my configuration below. It is a summation of the Arch Linux nextcloud documentation, and of the pages referenced by it, including the official documentation, choosing one particular configuration, instead of presenting many alternatives: PostgreSQL, nginx, php, php-fpm and nextcloud.

PostgreSQL

Install the package, init a user and database:

# pacman -Syu postgresql
# su -l postgres
[postgres]$ initdb --locale=en_US.UTF-8 -E UTF8 -D /var/lib/postgres/data
[postgres]$ createuser -h localhost -P nextcloud
[postgres]$ createdb -O nextcloud nextcloud

Make it listen on a local UNIX domain socket only:

# vim /var/lib/postgres/data/postgresql.conf
listen_addresses = ''

nginx

Install the package and setup a site, according to the nextcloud nginx docs:

# pacman -Syu nginx
$ cat /etc/nginx/nginx.conf
user http;
worker_processes  4;

events {
    worker_connections  128;
}

http {
    include       mime.types;
    default_type  application/octet-stream;


    sendfile        on;
    keepalive_timeout  65;

    # https://wiki.archlinux.org/title/Nginx#Warning:_Could_not_build_optimal_types_hash
    types_hash_max_size 4096;
    server_names_hash_bucket_size 128;

    upstream php-handler {
        server unix:/run/nextcloud/nextcloud.sock;
    }

    server {
        listen       80;
        server_name  <your_hostname>;
        root /usr/share/webapps;

        location ^~ /.well-known {
          root /usr/share/webapps/nextcloud;

          # The rules in this block are an adaptation of the rules
          # in the Nextcloud `.htaccess` that concern `/.well-known`.
          location = /.well-known/carddav { return 301 /nextcloud/remote.php/dav/; }
          location = /.well-known/caldav  { return 301 /nextcloud/remote.php/dav/; }
          location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
          location /.well-known/pki-validation    { try_files $uri $uri/ =404; }
          # Let Nextcloud's API for `/.well-known` URIs handle all other
          # requests by passing them to the front-end controller.
          return 301 /nextcloud/index.php$request_uri;
        }

        location ^~ /nextcloud {
          # set max upload size
          client_max_body_size 10M;
          fastcgi_buffers 64 4K;

          # Enable gzip but do not remove ETag headers
          gzip on;
          gzip_vary on;
          gzip_comp_level 4;
          gzip_min_length 256;
          gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
          gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

          # Pagespeed is not supported by Nextcloud, so if your server is built
          # with the `ngx_pagespeed` module, uncomment this line to disable it.
          #pagespeed off;

          # HTTP response headers borrowed from Nextcloud `.htaccess`
          add_header Referrer-Policy                      "no-referrer"   always;
          add_header X-Content-Type-Options               "nosniff"       always;
          add_header X-Download-Options                   "noopen"        always;
          add_header X-Frame-Options                      "SAMEORIGIN"    always;
          add_header X-Permitted-Cross-Domain-Policies    "none"          always;
          add_header X-Robots-Tag                         "none"          always;
          add_header X-XSS-Protection                     "1; mode=block" always;

          # Remove X-Powered-By, which is an information leak
          fastcgi_hide_header X-Powered-By;

          # Specify how to handle directories -- specifying `/nextcloud/index.php$request_uri`
          # here as the fallback means that Nginx always exhibits the desired behaviour
          # when a client requests a path that corresponds to a directory that exists
          # on the server. In particular, if that directory contains an index.php file,
          # that file is correctly served; if it doesn't, then the request is passed to
          # the front-end controller. This consistent behaviour means that we don't need
          # to specify custom rules for certain paths (e.g. images and other assets,
          # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
          # `try_files $uri $uri/ /nextcloud/index.php$request_uri`
          # always provides the desired behaviour.
          index index.php index.html /nextcloud/index.php$request_uri;

          # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
          location = /nextcloud {
              if ( $http_user_agent ~ ^DavClnt ) {
                  return 302 /nextcloud/remote.php/webdav/$is_args$args;
              }
          }

          # Rules borrowed from `.htaccess` to hide certain paths from clients
          location ~ ^/nextcloud/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)    { return 404; }
          location ~ ^/nextcloud/(?:\.|autotest|occ|issue|indie|db_|console)                  { return 404; }

          # Ensure this block, which passes PHP files to the PHP process, is above the blocks
          # which handle static assets (as seen below). If this block is not declared first,
          # then Nginx will encounter an infinite rewriting loop when it prepends
          # `/nextcloud/index.php` to the URI, resulting in a HTTP 500 error response.
          location ~ \.php(?:$|/) {
              fastcgi_split_path_info ^(.+?\.php)(/.*)$;
              set $path_info $fastcgi_path_info;

              try_files $fastcgi_script_name =404;

              include fastcgi_params;
              fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
              fastcgi_param PATH_INFO $path_info;
              #fastcgi_param HTTPS on;
              fastcgi_param HTTPS off;

              fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
              fastcgi_param front_controller_active true;     # Enable pretty urls
              fastcgi_pass  php-handler;

              fastcgi_intercept_errors on;
              fastcgi_request_buffering off;
          }

          location ~ \.(?:css|js|svg|gif|png|jpg|ico)$ {
              try_files $uri /nextcloud/index.php$request_uri;
              expires 6M;         # Cache-Control policy borrowed from `.htaccess`
              access_log off;     # Optional: Don't log access to assets
          }

          location ~ \.woff2?$ {
              try_files $uri /nextcloud/index.php$request_uri;
              expires 7d;         # Cache-Control policy borrowed from `.htaccess`
              access_log off;     # Optional: Don't log access to assets
          }

          # Rule borrowed from `.htaccess`
          location /nextcloud/remote {
              return 301 /nextcloud/remote.php$request_uri;
          }

          location /nextcloud {
              try_files $uri $uri/ /nextcloud/index.php$request_uri;
          }
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
}

php

Install php, enable the required modules and enable caching:

# pacman -Syu php php-apcu php-gd php-imap php-intl php-pgsql php-sqlite

$ snip /etc/php/php.ini
memory_limit = 1024M
extension=bz2
extension=gd
extension=gmp
extension=imap
extension=intl
extension=pdo_pgsql
extension=pgsql

$ cat /etc/php/conf.d/apcu.ini
extension=apcu.so
apc.ttl=7200
apc.enable_cli=1

Opcache is enabled by default.

php-fpm

This is a daemon that takes requests from nginx and executes them with php. Install and configure:

# pacman -Syu php-fpm

$ snip /etc/php/php-fpm.conf
systemd_interval = 1h

$ cat /etc/php/php-fpm.d/nextcloud.conf
[nextcloud]
user = nextcloud
group = nextcloud
listen = /run/nextcloud/nextcloud.sock
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp

; should be accessible by your web server
listen.owner = http
listen.group = http

pm = dynamic
pm.max_children = 15
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

$ cat /etc/systemd/system/php-fpm.service.d/override.conf
[Service]
ReadWritePaths=/var/lib/nextcloud/data

nextcloud

Install the php files, setup the database:

# pacman -Syu nextcloud
# /usr/bin/occ maintenance:install --database pgsql --database-name nextcloud --database-host /run/postgresql --database-user nextcloud --database-pass nextcloudpw --data-dir /var/lib/nextcloud/data -vvv

Enable APCu:

$ snip /usr/share/webapps/nextcloud/config/config.php
'memcache.local' => '\OC\Memcache\APCu',

Start the required services and check their status:

# systemctl start postgresql nginx php-fpm
# systemctl status postgresql nginx php-fpm

The nextcloud web UI should be accessible at http://<your_hostname>/nextcloud.

Appendix

To see the difference between a current file and the package maintainers original version, use (source):

function qkkdiff-file {
  if [[ "$1" == "" ]];
  then
      echo "Example usage: qkkdiff-file /etc/pacman.conf"
  elif [[ -f "$1" ]];
  then
      pkg="$(pacman -Qo $1 | awk '//{printf "%s-%s", $(NF-1), $NF;}')"
      bsdtar -xOf /var/cache/pacman/pkg/${pkg}-$(uname -m).pkg.tar.xz "${1/\//}" | diff - "$1"
      return 0
  else
      echo "The provided file ${1} does not exist."
      return 1
  fi
}

Input Latency

Mon, 11 Oct 2021 00:00:00 +0000

This post was inspired by Dan Luu: Computer Latency. I felt that my work setup (see below) is sluggish, even though I got used to it. I wanted to quantify how sluggish it really is. Also, VSCode was advertised to offer a lower latency experience (compared to ssh+vim), as it runs and renders on the local machine: I didn’t feel it - I wanted to verify my observation, knowing that I use vim for too long to acknowledge its shortcomings.

Results

Program	Latency (ms)
kitty	90
gvim	100
vscode	140
remote² vim	195
remote vscode	200
remote Visual Studio	210

These are the test results of the latency between a keypress and the display of a character (see below for details). Results are rounded to the nearest 5 ms to avoid the impression of false precision.

The measurement confirms that typing on this particular remote setup is twice as bad as locally, and that the additional latency of vscode (compared to vim) is greater than the latency of ssh.

Experimental setup

A local computer was filmed with a 240 FPS camera. Latency is defined as the time (number of frames) elapsed between a keyboard key starts moving, and the matching character appears on screen. The declared latency is a rounded average of several key presses, without much deviation. No particular care was taken to achieve the lowest latency possible (e.g: CPU pinning, interrupt pinning, reducing workload) – as that’s not how I normally work – but the computers used were under no particular load in general.

The local computer runs on a AMD FX 8320, with a standard HP KU-0316 keyboard attached via USB, and a 4K 60Hz AOC monitor attached via DisplayPort. The operating system is Linux 5.14.7, running the latest software from the ArchLinux distribution, including the i3 window manager.

The remote computer is a dedicated machine, hosted in a datacenter, ~1500 kilometers away from the local computer, with a modern Intel CPU and plenty of RAM. It is accessed with xfreerdp, and runs Windows 10.

The remote server is a server class Intel machine of unspecified details, runs Linux 3, and somewhat older versions of software. Regarding physical location, it is close to the remote computer described above. It is shared between different users, but during the test, no particular load was observed.

Contenders

kitty is a powerful terminal emulator, version 0.23.1 was used. Runs on the local computer.

gvim is the GTK GUI version of the venerated editor, version 8.2. Runs on the local computer.

vscode is a recent Microsoft product, version 1.60.2. Runs on the local computer.

remote vscode runs on the remote computer, version 1.53.2.

remote Visual Studio runs on the remote computer, version 16.9.2.

remote² vim is version 8.2. It runs in a tmux instance on the remote server, accessed via PuTTY, that runs on the remote computer (that is accessed via xfreerdp, from the local computer), totalling to 2 logical hops in one direction. Contorted? Yes. Unrealistic? Well, that’s how I have to work…

Bootstrap aarch64 image on X86

Sun, 21 Mar 2021 00:00:00 +0000

The current (2021-Feb-05) aarch64 Arch Linux ARM image has issues on 8GB Raspberry Pi 4 models: It does not bring up the network interface, and the USB ports are not working, making any interactive fix difficult (without serial cable).

A possible fix is to replace the default kernel with a Raspberry specific one. It still results in a 64bit system, only with a non-upstream kernel. If there’s a different ARM system available, it is trivial to mount/boot the SD card there and apply the required changes. The solution below, however, only requires a single working, linux supported hardware, e.g: x86. This description is Arch Linux specific. For ubuntu, see this blogpost on Disconnected Systems.

Install Qemu

Qemu makes it possible to run arm binaries on x86 (and many more combinations). Install qemu-user-static from AUR. To make the resulting package smaller, add --target-list=aarch64-linux-user to the configure invocation in the PKGBUILD file. This package requires glib-static and pcre-static, the former takes some time to build. Due to the number of dependencies, consider building in a clean chroot. (Do not forget to set -j in MAKEFLAGS)

Install the built qemu-user-static package. Read the introduction of binfmt, then setup the config accordingly:

$ cat /etc/binfmt.d/qemu-static.conf
:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-static:CF

Finally, restart the service that - if there’s a config file - mounts the filesystem described by the doc above:

$ sudo systemctl restart systemd-binfmt

Now the system can run aarch64 binaries using qemu.

Install Arch Linux ARM

Follow the install guide. Use the ArchLinuxARM-rpi-aarch64-latest.tar.gz archive, but do not change /etc/fstab! Also, before unmounting root and boot, do:

$ sudo arch-chroot root

This will drop you into the installed ARM system. Add the alarm repos, and install the raspberry kernel:

$ pacman-key --init
$ pacman-key --populate archlinuxarm
$ pacman -S --needed linux-raspberrypi4 raspberrypi-bootloader raspberrypi-bootloader-x raspberrypi-firmware firmware-raspberrypi

This is a good time to also complete the official Installation Guide. After done, copy root/boot over boot again, and unmount the partitions. Again, do not modify etc/fstab, contrary to the alarm docs.

With everything done correctly, using the patched image, the network and USB ports of the 8GB Raspberry Pi 4 are fine again!

Automated, encrypted b2 backups with rclone

Wed, 03 Mar 2021 00:00:00 +0000

Here’s how to create automated, secure backups on blackblaze using rclone and systemd.

First, register a new blackblaze b2 account if you don’t have one already. Make sure you select the right region during sign-up - it cannot be changed later. Create a new bucket, name it after your hostname. Create a b2 rclone remote (see doc). Create a crypt overlay over the b2 remote (see doc). Generate long passwords.

The resulting config is something like this:

$ cat ~/.config/rclone/rclone.conf
[b2]
type = b2
account = *redacted*
key = *redacted*

[b2_HOSTNAME_crypt]
type = crypt
remote = b2:HOSTNAME
filename_encryption = standard
directory_name_encryption = true
password = *redacted*
password2 = *redacted*

Try rclone listremotes, rclone copy, rclone mount to see if it works. Use the b2 online browser to verify that the uploaded data is encrypted.

If you want to backup your home directory, chances are, you don’t want to backup it all (e.g: ignore .cache). Create an rclone filter file that selects the files to sync:

$ cat ~/.config/rclone/filter.txt
- .cache/**
+ **

Use rclone ncdu --filter-from .config/rclone/filter.txt . to verify that the right files are selected.

Automate with systemd

To sync the directory, use rclone sync. To avoid overwriting previous backups, use --backup-directory. The following systemd service can be used:

$ cat ~/.config/systemd/user/rclone.service
[Unit]
Description="Sync selected parts of home directory to b2 encrypted remote storage"

[Service]
Type=oneshot
ExecStart=%s -c 'rclone sync --filter-from %E/rclone/filter.txt %h b2_%H_crypt:home --b2-hard-delete --fast-list --transfers 32 --backup-dir b2_%H_crypt:home-$$(date +%%u)'

This will keep 1 week worth of data without overwriting. A simple systemd timer can make it run daily:

$ cat ~/.config/systemd/user/rclone.timer
[Unit]
Description="Sync daily"

[Timer]
OnCalendar=*-*-* 12:15:00
FixedRandomDelay=true
Persistent=true

[Install]
WantedBy=timers.target

Enable and start the timer:

$ systemctl --user enable rclone.timer
$ systemctl --user start rclone.timer

Backup the config

If the rclone encryption passwords are stored in the directory that is being backed up, in case of a local failure, the remote content cannot be restored. Therefore, the rclone config must be stored separately. The Paperkey page offers a couple of ideas. For example, to get a printable version of an encrypted config, do:

$ qrencode --8bit -t svg -lH -o config.svg < encypted_config

To recover:

$ zbarcam -1 --raw -q -Sbinary

The encryption can happen with e.g: gpg, that requires a similar treatment of the private key.

Note about browsers

Browsers tend to write their persistent storage frequently, making it more difficult to create a consistent snapshot of them. To reduce the chance that a browser write conflicts with the sync action, use Profile Sync Daemon It will move the browser storage to memory, and sync it back periodically.

Store SSH key passphrase in password manager

Mon, 08 Feb 2021 00:00:00 +0000

The ssh-agent program can be used to manage unlocked SSH keys. The ssh-add program can be used to add keys to a running ssh-agent. By default, ssh-add uses TTY to ask for a password, if the selected key is password protected.

The pass program is a password manager that follows the Unix philosophy. If you want to use pass together with ssh-add, first add the private key password to it:

$ pass insert ssh/identityname_rsa

Then create this wrapper program /home/user/bin/askpass.sh:

#!/bin/sh
# $1 ~= Enter passphrase for '.ssh/$KEYNAME':
exec /usr/bin/pass $(grep -o "ssh/[^:']\+" <<< "$1")

Export the following variables, e.g: by adding them to you shell rc:

export SSH_ASKPASS_REQUIRE=force
export SSH_ASKPASS="/home/user/bin/askpass.sh"

Now whenever an SSH key is needed (e.g: for git or ssh client), ssh-add will turn to pass, giving it the key name to be unlocked, and in case the gpg key is not unlocked at the moment, pass will prompt you to enter your password, then safely transmit the key passphrase to ssh-add, that will add the key to the running ssh-agent.

If git does pop up the gpg unlock dialog, but prints “gpg: decryption failed: No secret key” instead, make sure the GPG_TTY environment variable is exported:

export GPG_TTY=$TTY

If git still does not ask for a password, and no such error is printed: If the private key is not the default one (e.g: .ssh/id_*), the host-key association must be configured in .ssh/config, e.g:

Host github.com
  IdentityFile ~/.ssh/your_nondefault_key

Useful FreeRDP flags for good performance

Mon, 25 Jan 2021 00:00:00 +0000

I need to use RDP to connect to a Windows 10 machine. Equipped with a 4k display, on Arch Linux, with FreeRDP 2.2.0, the following combination of flags turned out to be the best: -grab-keyboard /gfx-h264:avc444 +gfx-progressive.

To make it work, first make sure hardware acceleration is enabled. With AMD:

$ sudo pacman -Sy xf86-video-amdgpu libva-mesa-driver freerdp

Then add this .desktop file:

$ cat .local/share/applications/xfreerdp.desktop
[Desktop Entry]
Name=XFreeRDP
Comment=Connect to remote desktops using RDP
Exec=xfreerdp %U -grab-keyboard /gfx-h264:avc444 +gfx-progressive
Terminal=false
Type=Application
StartupNotify=false
MimeType=application/x-rdp-type;

Without /gfx-h264:avc444, certain UI elements (e.g: red text on dark background in PuTTY) are rendered unreadable. Without +gfx-progressive, scrolling is extremely laggy, a full redraw (switching window) can take a second.

-grab-keyboard is used to let i3 key bindings take effect even in full screen mode. If this doesn’t work for you, take a look at this post and the freerdp man page.

Overloading by concept without concepts in C++ part II

Thu, 14 Jan 2021 00:00:00 +0000

In the previous post, we’ve seen how to select an overload from a set of functions based on concepts matching the arguments, without actual concept support in the C++ language. The described solution allows a library to define a single set of overloads.

However, how to allow the client to override the library provided overload for a given concept? Simply adding a specialization would result in ambiguous function calls. If not today, probably later, when the library changes - making the solution hard to maintain.

A conflict free solution simply adds an indirection:

If user defined overload exists, call that
Otherwise call the library defined function

This clearly models what we want to achieve. This is how it looks in code:

template <typename, typename = void> struct CustomF {
  CustomF() = delete; // detection will use is_constructible
};

template <typename, typename = void> struct BuiltinF; // default case undefined
// Specializations for supported concepts as shown before follow...

template <typename T> struct F {
  using type = std::conditional_t<
    std::is_constructible<CustomF<T>>::value,
    CustomF<T>,
    BuiltinF<T>
  >;
};

The type member of F will be either the user defined custom logic (if the type is constructible), or the built-in logic. The wrapper that hides the invocation remains the same:

template <typename T>
void f(const T& t) { detail::F<T>::f(t); }

That’s it! Now clients can easily add CustomF specializations, without risking conflicts with the present or future specializations of BuiltinF.

This is the system that the binlog serializer library, mserialize uses.

Overloading by concept without concepts in C++

Thu, 12 Sep 2019 00:00:00 +0000

Given the following function:

void f(int);

Suppose you want to make the behaviour of this function to depend on the argument type. A simple way to use overloading:

void f(int);
void f(double);

What if you want to define an overload for a set of types, which satisfy a given concept? As of C++20, concepts can be used for overloading. Before that, the function has to be converted to a template. However, since C++ does not allow specialization of function templates, first change the function into a type template with a single static member:

template <typename> struct F; // default case undefined
template <> struct F<int> { static void f(int); };
template <> struct F<double> { static void f(double); };

This is equivalent with the above, but now we can apply various kind of SFINAE magic to enable different specializations for different kind of types. Common techniques use std::enable_if, expression SFINAE and others. Here a simple, yet powerful solution is presented, which is based on void_t:

template <class... >
using void_t = void;

This innocent looking type alias resolves to void, if the template arguments are valid, otherwise it becomes a substitution failure.

To use it, add an extra, defaulted template argument to F:

template <typename, typename = void> struct F; // default case undefined
template <> struct F<int, void> { static void f(int); };
template <> struct F<double, void> { static void f(double); };

So far, there’s no behavioural change. However, no we can easily add overloads which apply to a set of types, e.g: types which has a specific nested type:

template <typename Container>
struct F<Container, void_t<typename Container::value_type>> {
  static void f(Container const&);
};

A different example, specialize for types with a specific member:

template <typename Lockable>
struct F<Lockable, void_t<decltype(Lockable::lock)> {
  static void f(Lockable const&);
};

This technique also integrates well with standard type predicates. First, define the following helper:

template <typename Cond>
using enable_spec_if = void_t<std::enable_if_t<Cond::value>>;

Usage:

template <typename Trivial>
struct F<Trivial, enable_spec_if<std::is_trivial<Trivial>>> {
  static void f(Trivial const&);
};

Given C++17 is available, a similar, slightly longer solution is:

template <typename Trivial>
struct F<Trivial, std::enable_if_t<std::is_trivial_v<Trivial>, void>> {
  static void f(Trivial const&);
};

A limitation of this technique (and also several others), that specializations must be mutually exclusive for the types they are used.

To make usage easier, we can add a function template, making it the only entry point of the operation, while hiding F in the detail namespace:

template <typename T>
void f(const T& t) { detail::F<T>::f(t); }

In the next part, we’ll cover how to allow third parties to define additional specializations, without conflicting with built in ones.

Applied Ephemeralization

Sun, 28 Apr 2019 00:00:00 +0000

Ephemeralization: Do more and more with less and less until eventually you can do everything with nothing
Ephemeralism: Do more with less, until you can do everything with nothing

Also see Wikipedia on Ephemeralization

How to share a calendar entry

Wed, 30 May 2018 00:00:00 +0000

Suppose you’d like to share a calendar event over the internet, e.g: on your website. The goal is to be support as many platforms as possible:

Desktop calendars (Thunderbird, macOS iCal, Outlook)
Web applications (Google Calendar)
Smartphone applications (Google Calendar App, iOS iCal)

The most intuitive way is to create the event in your calendar of choice, and send the invites in the native way provided by the application. However, such methods tend to work only if the recipient uses the same system, which isn’t a plausible assumption – we need something platform independent.

Using iCalendar

Wikipedia indicates that iCalendar is well supported format, which allows Internet users to send meeting requests and tasks to other Internet users by sharing or sending files in this format through various methods. – exactly what we are looking for. Let’s create an .ics file:

BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
X-WR-TIMEZONE:Europe/Budapest
BEGIN:VEVENT
UID:your_eventname@yourdomain.com
DTSTART:20180530T130000Z
DTEND:20180530T143000Z
DESCRIPTION:The description of your event
LOCATION:The location of your event
SEQUENCE:0
SUMMARY:The summary of your event
END:VEVENT
END:VCALENDAR

There are other fields. The fields are mostly self describing. In free form text values, , and ; characters must be escaped by a backslash. The tzselect command can be used to get the name of your timezone. The DTSTART and DTEND fields are in UTC, 20180530T130000 is 2018. 05. 30, 13:00:00. If all this sound difficult, create your event in a suitable calendar application and export it to .ics from there.

Sharing this file will cover desktop calendars associated with the ics extension and the iOS calendar. It probably will not work with the Google Calendar website, and – surprisingly – it will not work on Android as expected.

Fixing Android

On Android, after download, a toast message will be shown, saying: “Unable to launch event”. Internet wisdom blames various components. On the phones I had access to, Logcat show that the Calendar app does not have permission to read downloaded files.

Fortunately, there’s a way around: Android allows emitting intents by links, in some cases. The proper syntax and the allowed fields can be discovered by looking at the source, or StackOverflow. Here’s an example:

<a href="https://www.google.com/calendar/event?action=TEMPLATE&text=your_event&dates=20180530T130000Z/20180530T143000Z&location=location of the event&sprop=yourdomain.com">Calendar event</a>

Values need to be urlencoded. Clicking on this link on Android will open the Google Calendar application, if installed, with the specified event shown to be saved. Fortunately, this link will work for the browser based Google Calendar as well!

Solution

Let’s put all this together. By default, we should link to the .ics file, as it works on most platforms. On Android, the link has to be changed:

<a href="event.ics" data-gcal="https://www.google.com/calendar/event?..." class="ics-link">Calendar event</a>

var replaceIcs = function() {
  var as = $('.ics-link');
  for (var i = 0; i < as.length; i++) {
    as[i].href = as[i].getAttribute('data-gcal');
  }
};

document.addEventListener("DOMContentLoaded", function(e) {
  // NB: there are more robust ways to check for android
  var userAgent = navigator.userAgent.toLowerCase();
  if (userAgent.indexOf("android") >= 0) {
    replaceIcs();
  }
});

What about desktop clients using the Google Calendar web application? It is difficult if possible to tell what kind of calendar solution such users would like to use. However, there’s a hack, which detects if the user is logged in to a google account. If desired, as an imperfect solution, ics links can be also replaced, if such a login is detected.

Summary

It is more difficult to share a calendar entry over the internet as expected. Application native ways do not interoperate, while some platforms are hard to target by standard solutions. While the solution shown above covers a large percentage of use-cases, it still might fail in others. A free web needs more support of open standards!

The sorcerers frog

Sat, 29 Mar 2014 00:00:00 +0000

Once there was a sorcerer who had the most shining gold. A simple and mild man came to him and wanted to buy this gold from him. The sorcerer said to the simple man: “You will not receive this gold, unless you give me better gold and in larger quantity.” The man said: “I have such a great desire for your gold that I will give you what you want rather than losing it.” He then gave the sorcerer better gold and in larger quantity and received the shining gold from him and put it in a casket, thinking of making himself a ring from it for his finger.

After a short time, the sorcerer approached that simple man and said: “The gold you bought from me and laid in your casket is not gold, as you thought, but the most ugly frog. It has been fostered in my chest and fed with my food. And in order for you to test and know that this is true, you may open the casket and you will see how the frog will jump to my chest where it was fostered.” When the man wanted to open it and find out if it was true, the frog appeared in the casket. The cover of the casket was hanging on four hinges that were about to break and fall off soon. Immediately when the cover of the casket was opened, the frog saw the sorcerer and jumped into his chest.

When the servants and friends of the simple man saw this, they said to him: “Lord, this most fine gold is in the frog, and if you want, you can easily get the gold.” The man said: “How can I get it?” They replied: “If someone took a sharp and heated spear and thrust it into the hollow part of the frog’s back, he would quickly get the gold out. But if he cannot find any hollow in the frog, he should then, with the greatest force and effort, thrust his spear into it, and this is how you will get back the gold you bought.”

Celestial Revelations

Bootstrap Typeahead submit on select

Sat, 02 Mar 2013 00:00:00 +0000

The Twitter Bootstraps Typeahead is a very nice way to provide autocomplete functionality on your text inputs. However, the default configuration might be a bit confusing. When the user clicks on a suggestion in the dropdown menu, the utility populates the input but doesn’t submit the form. It’s usually ok, but sometimes (e.g: search boxes) it’s frustrating. Here’s how to change it:

var input = $('#your-input-box');
input.typeahead({
    'source' : ['foo', 'bar', 'baz'],
    'updater' : function(item) {
        this.$element[0].value = item;
        this.$element[0].form.submit();
        return item;
    }
});

This snippet autosubmits the form if the user clicks on a suggestion or selects it by keyboard.

Imprisoned Toroid - Raytrace on CUDA

Sun, 25 Nov 2012 00:00:00 +0000

The following video is created by a C++/CUDA program using the theory of raytracing which was the most interesting piece of knowledge I learned on the university this semester.

Because of its as-good-as-it-should-be quality of the source, I can’t opensource it at this precise moment. Meanwhile, take a look at OptiX, which I found a month later I had completed this.

The video was rendered at 24 FPS, render time appr. 3 hours.

YouTube

Posts on erenon.hu

Summary of Readings

Faith

History

Science

Finance

Multidimensional Make Targets

See Also

To hang an Icon

Restore file from editor memory

Solution

Takeaway

No Miracles

Summary of Readings

Adventure

Biography

Sociology

Software

Selfhosted RSS Reader

Considered Alternatives

Why Bazel

The Problem

The Solution

Summary of Readings

Hit

Industrial Design

Psychology

Economy

Honorable Mentions

LedgerSplit vim function

Installation

The disgust of sequential processing of binary WebSocket messages in JavaScript

Ordering

Conclusion

Sway sessions à la tmux

Scripts to emulate sessions

Summary of Readings

Energy

Environment

Software

Summary of Readings

Engineering

Software

History

Social and Psychology

Money and Economy

Parenting

The Unreasonable Effectiveness of Doing Nothing

Static storage shared between compilation units

Theory

Put Data into Custom Sections

GCC Complications

Get data from custom sections

Use case

Preprocessor foreach compile time

Contenders

Compile Time Comparison

Implementation Comparison

Verdict

Snapcast on Demand

Review of Thriving in a Crowded and Changing World

Nextcloud on RPi4 with Arch Linux

PostgreSQL

nginx

php

php-fpm

nextcloud

Appendix

Input Latency

Results

Experimental setup

Contenders

Bootstrap aarch64 image on X86

Install Qemu

Install Arch Linux ARM

Automated, encrypted b2 backups with rclone

Automate with systemd

Backup the config

Note about browsers

Store SSH key passphrase in password manager