smhk

Using Dear ImGui from C

Wed, 29 Apr 2026 16:00:00 +0000

These notes cover how to add Dear ImGui into an existing C project using the SDL3_Renderer backend. Since Dear ImGui is C++, we use the C bindings provided by Dear Bindings. The end result is a C application with a couple of example GUI windows.

Bindings

Since Dear ImGui is written in C++, we need some bindings to enable us to use it from C.

Two popular options are:

Both of these take the Dear ImGui C++ source as input, and perform some processing to output C bindings.

Firefox Profile Nomenclature: Traditional vs Fancy

Fri, 28 Nov 2025 18:00:00 +0000

As someone who has been using Firefox Profiles for years, I was initially excited by the announcement of the new profile management feature. It sounded like this was promoting the existing feature with a UI refresh.

Confusingly, it’s a completely separate feature that shares the exact same name!

Let’s invent some terminology so that we can be clear:

Traditional Profiles
- These have been around for years.
- They are managed through about:profiles.
Fancy Profiles:
- These began a gradual rollout¹ on 14th October 2025. If not yet enabled on your Firefox, go to about:config and enable browser.profiles.enabled.
- Once enabled, these are managed through about:profilemanager and about:editprofile.

What do these two profiles look like?

Pixel Perfect Rendering in SDL2

Fri, 10 Oct 2025 17:45:00 +0100

TL;DR: To ensure SDL2 rendering is pixel perfect, you must use “nearest pixel sampling” for the render quality, and mark the process as “DPI aware” on Windows.

Enable nearest pixel sampling

The scale quality must be set to "0", which is nearest pixel sampling.

SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "0");

If set to "1" ("linear") or "2" ("best"), SDL2 will use linear filtering when scaling textures, resulting in antialiasing.

Set DPI aware on Windows

Windows 10 may automatically apply a scaling of 125% to processes which do not declare themselves “DPI aware”. This will break any attempt at pixel perfect rendering.

Some useful PlantUML architecture diagrams

Tue, 15 Apr 2025 10:00:00 +0100

PlantUML is very useful for creating architecture diagrams using text. However, the official documentation is so big (a 606 page PDF as of v1.2025.0) that it’s not always easy to find what I’m looking for. Yet, despite being so big, not everything is documented. For example, there are 45 “TODO"s, and norank is not mentioned anywhere (but is mentioned in the unofficial but extremely helpful The Hitchhiker’s Guide to PlantUML).

So, following is a small collection of useful example architecture diagrams:

5G: Frames, Subframes, Slots, Minislots, Resource Blocks & Resource Elements

Fri, 28 Mar 2025 10:00:00 +0000

Overview

Subcarriers

In order to understand frames (and their component parts), it is helpful to first have a refresher on subcarriers:

A carrier is formed of multiple smaller subcarriers:
- The subcarrier: the building blocks of a carrier; a smaller frequency channel within the larger carrier.
The subcarrier spacing (or SCS): the frequency separation between adjacent carriers.
- e.g. 15kHz, 30kHz, 60kHz, 120kHz.
The frequency band: defines a duplex mode (TDD/FDD), one frequency range (if TDD, or two if FDD), and a range of supported channel bandwidths.
- For example, band n78 (very common in 5G):
  - The uplink/downlink frequency is 3300MHz-3800MHz (n78 is TDD, so uplink and downlink share the same frequency range).
  - This provides a total available bandwidth of 500MHz…
  - …however, of this 500MHz, the supported channel bandwidths are 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100MHz.
The channel bandwidth: the total frequency range occupied by a carrier.
To calculate the approximate number of subcarriers (ignoring guard bands): num_subcarriers = bandwidth / subcarrier_spacing
- e.g. For a 5MHz bandwidth with 15kHz SCS: 5MHz / 15kHz = 5,000,000 / 15,000 = ~333 subcarriers
- Multiple subcarriers are sent/received in parallel using OFDM.

Frames (and their component parts)

Now we can dig into frames, their component parts (subframes, slots, resource blocks, resource elements) and how they relate to subcarriers:

C stack trace in Windows

Tue, 25 Mar 2025 18:15:00 +0000

TL;DR: It’s possible to generate a C stack trace in Windows with GCC, but it’s easier with MSVC or LLVM/Clang. Either way, you need to install the Windows SDK and then compile and link against DbgHelp. With GCC, you also need to convert the DWARF debug information to PDB, which can be done with a tool called cv2pdb, but that depends upon installing the Visual Studio build tools.

These notes cover how to print a stack trace from a C application built using Mingw-w64 (with GCC or LLVM/Clang)¹ in CMake on Windows 11.

Enable British English language in Firefox

Fri, 14 Mar 2025 17:00:00 +0000

TL;DR: To add a British English dictionary for spellchecking in Firefox, first install the British English Language Pack, then install a British English Dictionary such as this one. Finally, go to about:preferences and set the language to “English (GB)”; this sets both the default Language Pack and Dictionary.

Overview

By default, Firefox provides the English (United States) dictionary:

These notes cover how to add the English (United Kingdom) dictionary and make it the default:

Open5GS: Change WebUI IP and port

Fri, 28 Feb 2025 17:00:00 +0000

TL;DR: To change the IP and port used by Open5GS’s WebUI, modify the service file to set HOSTNAME=x.x.x.x and PORT=yyyy for the Environment field.

Situation

The Open5GS WebUI binds to localhost:9999 by default.
In general, the Open5GS documentation is excellent, however I could not find documentation on how to change the IP and port used by the WebUI.
I needed to change the bind IP from localhost to 0.0.0.0 so that the WebUI is accessible from outside of the core.

Solution

The Open5GS WebUI IP and port can be changed by modifying the open5gs-webgui.service file as follows¹:

Setting up a test 5G network

Fri, 28 Feb 2025 17:00:00 +0000

TL;DR: Beware that Android phones may silently ignore your configured MCC/MNC/slice! Monitor their network requests (e.g. via tcpdump or gNB/core logs) to verify what settings the UE is actually using.

Following are various notes from setting up a test 5G network.

Operating a private 5G network on cellular frequency bands may be tightly regulated in your jurisdiction.

Configuration

Here are some critical things to be aware of when configuring the core for a UE to attach.

The Important Files (part 11): Fixing Windows 11 Samba share

Sat, 11 Jan 2025 16:45:00 +0000

TL;DR: The Samba share permissions are determined by the TrueNAS user. By default, Windows will user the Windows user to try and authenticate, which may result in a read-only user if this does not match with the corresponding Samba user. A solution is to check “Connect using different credentials” when creating the share and use the correct user.

After upgrading from Windows 10 to Windows 11, the Samba share with my TrueNAS had become read-only. These notes document how I re-created the share to ensure it had the correct read and write permissions.

Configure Poetry to use a virtualenv

Tue, 17 Dec 2024 16:00:00 +0000

TL;DR: If you want to configure Poetry to use a local virtualenv (e.g. .venv/) instead of storing the virtualenv in the cache, first you must set virtualenvs.in-project = true, second you must then re-create the virtualenv with poetry env remove <...> and poetry install.

Valgrind 'Invalid read' with SDL2

Sat, 14 Dec 2024 13:30:00 +0000

TL;DR: Old versions of Valgrind detect false positive memory issues in SDL2 applications. This was fixed in Valgrind v3.20, but unfortunately v3.19 is the latest available version on Debian stable apt. A solution is to build & install Valgrind from source.

Pylint: Use source-roots instead of init-hook to fix E0401 'import-error'

Fri, 13 Dec 2024 13:30:00 +0000

TL;DR: Instead of using sys.path.append(<path>) in init-hook to configure Pylint, use source-roots = "<path>".

Convert a Python script into a Poetry package

Fri, 13 Dec 2024 09:00:00 +0000

Tl;DR: Run poetry init convert a Python script into a Poetry package, then edit the resulting pyproject.toml to define the package, entry points (scripts), dependencies and more.

A new Python project can easily begin as a single script.py, but there soon becomes a point where it’s worth packaging up the script. That way you can benefit from proper dependency management, entry points, installation and much more.

Here is how to go from a script.py to a Python package using Poetry.

CMake and coverage with gcov

Mon, 25 Nov 2024 17:30:00 +0000

TL;DR: To generate coverage data in CMake: enable CTest, enable debug, build and link all relevant targets with --coverage, then run ctest --test-dir build -T Coverage. Use gcov or lcov to generate a CLI or HTML coverage report respectively.

Setting up Unity test framework for CMake

Wed, 20 Nov 2024 10:00:00 +0000

Unity is a test framework written in C, which is commonly used for unit testing embedded software. It can be integrated with CMake, but official support is “a work in progress and not suitable for use yet”. However, as shown by this blog post from November 2022 and its corresponding example project, Unity can integrate well with CMake.

It’s been two years since that blog post was published, and both Unity and CMake have changed a little, so I decided to share the steps I took today in November 2024. The main differences between my steps and the aforementioned are:

Fixing a minor issue (that arose due to changes in Unity).
Updating the usage of CMake (due to changes in CMake).
Aligning some things with my personal preference.

Installing Docker Desktop using local admin

Tue, 12 Nov 2024 16:00:00 +0000

TL;DR: If you installed Docker Desktop with a local admin account, you must manually add your (non-admin) user account to the docker-users group. This is mentioned in the official docs, but is easy to miss.

Zig: First impressions

Mon, 04 Nov 2024 20:00:00 +0000

I have been working through the excellent Crafting Interpreters by Robert Nystrom. Specifically, “part III: A Bytecode Virtual Machine”. For this part the book uses C, but I decided to use Zig. I’ve never used Zig before. Following are some rough notes from my first experience of using Zig.

C memory layout

Mon, 04 Nov 2024 10:00:00 +0000

The memory used by a C program is laid out in various segments:

Switching from Netlify to Uberspace

Wed, 30 Oct 2024 20:00:00 +0100

TL;DR: Some rough notes on how I switched hosting this website from Netlify to Uberspace, and analytics from Plausible to Goatcounter.

Background

I have been hosting this website with Netlify since January 2018, which has in general been a great experience! It was easy, convenient and free to host a static website. While some features were quite useful (e.g. branches automatically creating website previews on hidden subdomains), in general Netlify was far more featureful than I needed (e.g. severless functions, team management).

Hugo: How to create a post series

Tue, 29 Oct 2024 17:30:00 +0000

TL;DR: Follow these steps to implement a post series in Hugo, just like the one at the top of this page!

A few years back I followed this blog post to implement support for post series in Hugo. However, when I upgraded to Hugo 0.136.2 recently, this broke, and the series was just blank. I found this more recent blog post helpful in coming up with the following solution.

FDM, OFDM & OFDMA

Tue, 29 Oct 2024 09:00:00 +0000

Following is an interactive overview of OFDMA, which is used many wireless standards, including 5G NR, to enable efficient use of bandwidth for multiple users.

Debian: Upgrading Dropbox

Mon, 21 Oct 2024 21:00:00 +0100

TL;DR: Dropbox on my Debian machine had stopped syncing. Turns out it had not been updated in 4 years. Fix was to update Dropbox.

Upgrading Node, Go and Hugo for this website

Sun, 20 Oct 2024 21:00:00 +0100

Time for some routine maintenance. Need to upgrade the dependencies for this website and fix any resulting issues.

Ubuntu: allow a port through the firewall

Tue, 08 Oct 2024 15:00:00 +0100

To allow TCP port 12345:

$ sudo ufw allow 12345/tcp

Setting up Zig on Windows for VS Code

Tue, 01 Oct 2024 09:00:00 +0100

TL;DR: On Windows, I recommend installing Zig via Scoop, then setting up the ZLS (Zig Language Server) with VS Code.

The Important Files (part 10): Fixing borg backup

Mon, 30 Sep 2024 18:00:00 +0100

TL;DR: Tried to run backup, it failed because the rsync.net username and hostname have changed. Updated the script and re-ran the backup, and it completed successfully after 3.5 hours. It took so long because this is the first time it has been run (successfully) in a few months.

Git: display current branch in Bash

Tue, 03 Sep 2024 20:00:00 +0100

To show the current Git branch in Bash, e.g.:

user@host:/path/to/directory (branch_name)$

Jupyter: plot multiple layers on a single map

Wed, 14 Aug 2024 21:00:00 +0100

TL;DR: Create a map (m) in a Jupyter Notebook by calling m = explore(...) on a geopandas.GeoDataFrame. Add multiple layers to the map by passing the m object back in for each layer, e.g. m = explore(m=m, ...).

Box2D v3 is released

Mon, 12 Aug 2024 12:00:00 +0100

I was excited today to see that Box2D v3.0 has been released.

Docker: understanding it

Fri, 09 Aug 2024 12:00:00 +0100

It’s common to see docker run -it or docker run -itd, but what do those arguments actually mean?

Setting up Rye on Windows

Fri, 09 Aug 2024 12:00:00 +0100

TL;DR: Do not run pipx install rye. Instead, first enable Developer Mode in Windows, and then use the official Rye installer.

Recursively update Git submodules

Fri, 02 Aug 2024 12:00:00 +0100

To recursively update Git submodules:

git submodule update --init --recursive

Python forward reference

Fri, 02 Aug 2024 10:00:00 +0100

Prior to Python 3.12, recursive type require from __future__ import annotations:

Example of a recursive type

from __future__ import annotations

class Something:
 def __init__(self, parent: Something | None) -> None:
 self.parent: Something | None = parent

Python argparse

Thu, 01 Aug 2024 15:30:00 +0100

In Python, there is no need to write arguments --like_this.

You can do arguments --like-this, and argparse will convert the - characters to _.

For example:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument("--like-this", type=int)
>>> args = parser.parse_args("--like-this=123".split())
>>> args
Namespace(like_this=123)
>>> args.like_this
123

What is O-RAN?

Wed, 31 Jul 2024 12:00:00 +0100

The terms Open RAN and OpenRAN are often used interchangeably, but can have different meanings depending upon the context¹:

Open RAN (with a space): Refers to the general concept of an open Radio Access Network (RAN) architecture, which can be disaggregated into component parts, with openly defined interfaces between them.
OpenRAN (without a space): Refers to the specific implementation of an “Open RAN” as defined by the O-RAN Alliance.

To avoid ambiguity, this note will use the term O-RAN to refer to the specific implementation of OpenRAN by the O-RAN Alliance.

Getting started with clang-format

Mon, 29 Jul 2024 12:00:00 +0100

Just as Black and isort can be used to format Python code, clang-format can be used to format C/C++/C# code (and more¹).

pytest: Selectively mark values as xfail in parametrize

Tue, 11 Jun 2024 20:00:00 +0100

In these notes, we use the pytest parametrize decorator to apply marks to some values for that parameter, rather than marking the whole test.

Xming: bind to IP

Thu, 06 Jun 2024 12:00:00 +0100

Xming is a server¹ that can be used to display X11 applications on a Windows machine. The last release of Xming was on 9th August 2016 (almost 8 years ago), but still works just fine on Windows 10. For comparison, the last release of X11 was on 6th June 2012² (12 years ago today!).

Directory specific environment variables with direnv

Thu, 30 May 2024 12:00:00 +0100

direnv is a very simple but useful tool. Create a .envrc file that defines some environment variables, e.g.:

export MY_VAR=123
export SOMETHING_ELSE=blah

When you cd into that directory, the environment variables are loaded. When you cd out, they are unloaded.

VirtualBox VM freezes at 'Loading essential drivers'

Wed, 08 May 2024 20:00:00 +0100

TL;DR: If a VM gets stuck during boot, simply pausing and resuming can get it unstuck.

Hugo: Global resources

Wed, 01 May 2024 19:30:00 +0100

In February 2018, I used a bit of a hack to create a global resources object by using Hugo v0.32 page resources, i.e. .Resources.Get.

In July 2018, Hugo v0.43 was released, which added resources.Get. This uses the global resources object rather than a page-specific .Resources.

Using resources.Get instead of .Resources.Get makes it trivial to use global resources. As the docs say:

This function [resources.Get] operates on global resources. A global resource is a file within the assets directory, or within any directory mounted to the assets directory.

These notes how to switch from my hacky solution to the built-in global resources.

Pure CSS dark mode support for code highlighting

Wed, 01 May 2024 08:00:00 +0100

These notes cover how to add support for code highlighting that automatically respects the user’s light☀️/dark🌑 mode configuration using pure CSS. No JavaScript!

Upgrade version of pkginfo used by Poetry

Tue, 30 Apr 2024 18:00:00 +0100

Some Poetry issues¹ are fixed by upgrading pkginfo. However, the method for upgrading the verison of pkginfo used by Poetry depends upon how you installed Poetry:

Interactive geodetic coordinates

Mon, 29 Apr 2024 18:10:00 +0100

For some applications of geodesy, the Earth is modelled as an ellipsoid, since the planet bulges at the equator.

Python: mock reading and writing files

Sat, 20 Apr 2024 16:45:00 +0100

Since Python 3.3, the standard library has included unitest.mock, which provides utilities for mocking out parts of the system for testing.

These notes cover how to use unittest.mock when reading from one or more files, or writing to a single file.

Pure CSS dark mode support

Thu, 11 Apr 2024 08:00:00 +0100

It is easy to support dark mode with pure CSS. No JavaScript required.

Hugo: Migrating from Pygments to Chroma

Wed, 10 Apr 2024 17:20:00 +0100

These notes cover how this website switched from Pygments to Chroma for code syntax highlighting.

The Important Files (part 9): Updating rsync.net account

Mon, 08 Apr 2024 10:00:00 +0100

TL;DR: Backup status was showing as “Idle” even though logs indicate backups had been succeeding. Decided to accept the rsync.net request to update my account to the new format in case that was (in part) responsible.

Trying out Coolify

Sun, 24 Mar 2024 20:00:00 +0000

TL;DR: I attempt installing Coolify on Raspberry Pi OS, but it fails because my OS turns out to be 32-bit, and Coolify requires 64-bit. In future, I would use an OS such as Ubuntu Server instead of Raspberry Pi OS.

I like the idea of self-hosting, but since 2018 I’ve been using Netlify. Some features of Netlify I find useful are:

Upon commit to main the website is automatically re-built and published.
Upon build failure the website remains at the previous working build.
Branches (with Merge Requests) automatically deploy build previews.

You cannot self-host Netlify, but you can self-host Coolify, which looks like it has the same key features.

In these notes, I plan to try out using Coolify to replace Netlify by self-hosting this website.

Replacing Tox with Poetry and pre-commit for CI linting

Fri, 22 Mar 2024 19:00:00 +0000

TL;DR: These notes meander a little. The initial problem was Tox being an inefficient way to do linting with my setup. I arrive at the solution of replacing Tox with pre-commit for running linting tools, both locally and in CI.

Useful symbols

Wed, 20 Mar 2024 08:30:00 +0000

Some useful symbols.

Degrees

°

Arrows

↑ ↓ → ←
↔ ↕ ↨
▲ ▼ ► ◄
« »

Maths

× ≈ ≠ ≥ ≤

Footnotes

Traditional order:

* † ‡ § ‖ ¶

Additional:

# Δ ◊ ↓ ☞

Setting up PowerShell

Fri, 08 Mar 2024 11:00:00 +0000

These steps improve PowerShell by:

Adding tabs.
Changing the tab completion to be more bash like.
Displaying the current Git branch (if applicable).

Git typo hamming distance

Thu, 07 Mar 2024 12:00:00 +0000

A typo of git is just one Hamming distance away from twelve other commands:

$ gti commit -m "FIXUP"
Command 'gti' not found, did you mean:
 command 'gtg' from snap getting-things-gnome (0.6)
 command 'gsi' from deb gambc (4.9.3-1.2)
 command 'gtf' from deb xserver-xorg-core (2:21.1.4-2ubuntu1.7~22.04.2)
 command 'git' from deb git (1:2.34.1-1ubuntu1.10)
 command 'ti' from deb ticgit (1.0.2.17-2.1)
 command 'gt' from deb genometools (1.6.2+ds-2)
 command 'gli' from deb ruby-gli (2.14.0-1.1)
 command 'ghi' from deb ghi (1.2.0-1.1)
 command 'bti' from deb bti (034-6)
 command 'gt5' from deb gt5 (1.5.0~20111220+bzr29-4)
 command 'gmi' from deb lieer (1.3-6)
 command 'gth' from deb genomethreader (1.7.3+dfsg-6)
See 'snap info <snapname>' for additional versions.

NiceGUI with async classes

Thu, 07 Mar 2024 12:00:00 +0000

These notes build upon the class based NiceGUI app written about here, and modify it to be async. It may help to read those notes first.

Python: how (not) to reset a dataclass

Wed, 21 Feb 2024 20:00:00 +0000

In Python, it can be useful to reset a dataclass back to its initial values, but this can be easy to get subtly wrong.

Using gprof with POSIX FreeRTOS

Thu, 15 Feb 2024 20:00:00 +0000

TL;DR: To profile a C program with gprof, build and link with -pg, and ensure that the program terminates cleanly (e.g. handles SIGINT and SIGTERM). Then it will generate a gmon.out file that you can analyze with gprof. If running the program under systemd, be sure to set GMON_OUT_PREFIX and WorkingDirectory=, so that it writes gmon.out.<pid> to a known location.

shm_open not working under systemd

Mon, 05 Feb 2024 19:00:00 +0000

TL;DR: shm_open, which opens shared memory, will fail with fd != -1 if it does not have permission to read/write to /dev/shm/<name>.

Sphinx: Link to HTML from MyST Markdown

Mon, 22 Jan 2024 19:00:00 +0000

TL;DR: If you try to create a link in Sphinx from MyST Markdown to an HTML file in the _static directory, Sphinx seems to only include the individual file you link to, and places it in the _downloads directory. If you want to include a group of HTML files, use {.external} as shown below.

New Year's Resolution

Fri, 19 Jan 2024 12:00:00 +0000

Tl;DR: After replacing the capacitors twice during 16+ years of regular use, I am finally getting a new monitor.

Append to .bashrc using Ansible

Wed, 17 Jan 2024 18:00:00 +0000

You can append to .bashrc using the Ansible blockinfile module.

Using Kerberos from an Ansible playbook

Mon, 15 Jan 2024 10:00:00 +0000

TL;DR: To run kinit under Ansible, use ktutil to create a ticket.keytab, and then Ansible can authenticate without prompting.

Poetry: install alpha builds

Tue, 09 Jan 2024 20:00:00 +0000

Tl;DR: Use allow-prereleases = true to tell Poetry to consider alpha/beta/dev builds.

Detect if Ansible is running under Vagrant

Sat, 06 Jan 2024 20:00:00 +0000

TL;DR: To tell if Ansible was invoked by Vagrant, define an is_vagrant flag as false in the playbook, but set it to true using extra_vars in the Vagrantfile.

Reflections on 10 years as a professional Software Engineer

Mon, 01 Jan 2024 20:00:00 +0000

As we begin 2024, I’ve now been working full time as a Software Engineer for over 10 years¹. These past few years in particular have been significant for several reasons, so I thought now would be a good time to take a moment to reflect. Thinking back, there’s a lot I could talk about: I’ve worked on many different projects, for a range clients, with numerous technologies, in various locations across the globe. However, I thought I’d try and focus on the general life lessons, and draw out some underlying points that will hopefully better stand the test of time. We shall see! Here are a few of the most important things I have learnt:

GitLab CI and poetry-dynamic-versioning

Wed, 20 Dec 2023 18:00:00 +0000

The poetry-dynamic-versioning plugin enables configuring Poetry to automatically generate the package version number from your VCS (e.g. Git), rather than from the version field in pyproject.toml.

For example, you can tag a commit in Git, and the release built from that commit will use the name of the tag.

How to set up Ansible and Vagrant in WSL

Fri, 15 Dec 2023 22:00:00 +0000

These notes walk step-by-step through setting up Ansible and Vagrant in WSL, so that you can run vagrant up from within WSL to bring up a VirtualBox VM, and use Ansible to provision that VM!

Why might you want to do this? Primarily because Ansible does not support Windows, but it does work under WSL.

There are lots of scattered pieces of information across the Internet about how to do all this, but support appears to have changed over the years, which should be expected given that this is not officially supported by Ansible or Vagrant.

Testing Vector and Clickhouse

Tue, 05 Dec 2023 10:00:00 +0000

I have a basic Python program which logs to journald.

Using a collector, such as OpenTelemetry Collector or Vector, I want to get these Python logs into ClickHouse.

Following are notes on trying this out with a VM.

Python log storage

Mon, 04 Dec 2023 17:00:00 +0000

OpenTelemetry is for collecting telemetry data (metrics, logs and traces), and forwarding them to storage and analysis tools. Here are more details on what it is.

Using structlog and journald

Thu, 30 Nov 2023 17:00:00 +0000

I was surprised to find practically nothing online regarding using structlog with journald. This seemed odd given how both are strong proponents of structured logging.

Update 2023-12-08: After writing this up, I did find this structlog issue, which seems to indicate there is interest in journald integration.

To skip straight to my example implementation, jump to the last section of these notes.

If you want to see how I got there, carry on reading. In the following notes I experiment with connecting structlog up to journald before converging onto a good solution. The end result is a Python application that uses structlog, and can be configured to send logs either to the console or journald. This allows the application to be run as a command line tool, or as a service under systemd.

Run Poetry command as systemd service

Wed, 29 Nov 2023 13:30:00 +0000

These notes cover how to run a Python Poetry command as a systemd service. I’m using Ubuntu 22.04, but the steps should be the same for systemd on other OSes.

Create an example Poetry Python package

Run poetry new example_service to create a new Poetry Python package named example_service.

Add systemd-python so that we can write to journald (see these notes for more details on installing systemd-python):

poetry add systemd-python

Add a basic main loop

Create an example_service/service.py file with a basic main loop, which just writes to journald every second:

Reading from and writing to journald in Python

Wed, 29 Nov 2023 13:00:00 +0000

These notes cover how to do the following with journald in Python:

Writing to journald
- Writing custom fields
Reading from journald
- Reading all recent entries
- Reading custom fields
- Filtering by the executable
- Filtering by the systemd service (or “unit”)

I’m using Ubuntu 22.04 running inside a Virtual Machine.

Overview

To demonstrate this, we’ll create a new Python package imaginatively named py_sysd_pkg (Python systemd package). Inside that we’ll add some functions for reading/writing journald entries, and hook them up to entry points using Poetry.

Installing systemd-python on Ubuntu

Wed, 29 Nov 2023 11:00:00 +0000

Following are steps for installing the official systemd Python package on Python using Poetry, on Ubuntu 22.04:

Steps

Install dependencies

Before installing the Python package, you must install the libsystemd-dev and pkg-config packages for Ubuntu:

apt-get install libsystemd-dev pkg-config

You’ll probably also need a compiler such as gcc.

Some documentation (such as the README.md) mentions needing to install libsystemd-journal-dev and various other libsystemd-* packages. However, as far as I can tell, these have all been combined into a single libsystemd-dev package. I can’t find a good reference for this, but the 2015 Debian mailing list mentions a bunch of these libsystemd-* packages as being either “deprecated” or a “transitional package”.

python-systemd != python-systemd

Fri, 24 Nov 2023 12:00:00 +0000

tl;dr: To install the official systemd Python package, use pip install systemd-python, even though python-systemd is the name of the repository on GitHub.

`systemd-python` ✔️

The systemd-python package on PyPI comes from the systemd/python-systemd repository on GitHub. To install this package:

pip install systemd-python # Correct.
pip install python-systemd # Wrong!! This will install the archived package described below.

This package is the official Python wrapper for systemd provided by freedesktop.org, and is currently maintained

PlantUML in Sphinx (using MyST Markdown) and GitLab

Wed, 22 Nov 2023 12:00:00 +0000

tl;dr: PlantUML in Sphinx cannot be written in a way that is compatible with both MyST Markdown and the GitLab UI preview. You need to either use reStructuredText instead of Markdown or stick with Markdown and forget about GitLab UI preview support).

Context

I have some documentation written in Markdown, and use Sphinx to convert it into HTML. By default, Sphinx uses reStructuredText, but I have used MyST to get Markdown support.

With this existing setup, my goal was to do the following:

Managing multiple remotes in Git

Fri, 17 Nov 2023 12:00:00 +0000

Following are some quick notes on how to manage using multiple remotes in Git.

What are remotes?

Remotes specify other repositories to which we can push or pull commits. Before pushing to or pulling from another repository, we must add it as a remote, identified by a name of our choice. By convention, origin is typically the name for the default remote. Multiple remotes can be defined.

Add a new remote

Create a new remote named some_org_repo which identifies the specified repo:

Renaming Git branches

Thu, 16 Nov 2023 12:00:00 +0000

Following are some quick notes on how to rename a branch in Git, both remotely and locally. In the following examples we are renaming the branch old-name to new-name by copying the branch and then deleting the original. These notes assume you have one remote called origin.

Poetry: Offline installation of packages

Tue, 14 Nov 2023 17:00:00 +0100

These notes are about how to perform an offline installation of a Python application using poetry install, not about how to install Poetry itself offline.

Configuring a Brocade FastIron switch

Wed, 01 Nov 2023 20:00:00 +0000

Following are some notes on configuring a Brocade FastIron FCX 624S network switch.

NiceGUI: Change threshold for binding propagation warning

Wed, 11 Oct 2023 12:00:00 +0100

In NiceGUI you may start to see a binding propagation warning logged repeatedly if you have lots of binding.

NiceGUI: Bind visibility to arbitrary value

Mon, 09 Oct 2023 09:00:00 +0100

NiceGUI gives lots of examples of using bind_visibility_from to show/hide an element based upon the value of some attribute.

NiceGUI: tkinter error when updating pyplot

Wed, 04 Oct 2023 13:00:00 +0100

TL;DR: if you get the error can't delete Tcl command when updating a pyplot within NiceGUI, add close=False to ui.pyplot().

The Important Files (part 8): Setting up NAS after shipping across the Atlantic

Tue, 03 Oct 2023 18:00:00 +0100

At the end of May I returned to the UK from Seattle with just few suitcases. Ten weeks later my shipping container turned up, which included my NAS, which has now completed a round-trip from the UK to Seattle and back. Two months has passed, and now it’s time to set up the NAS again. Let’s plug it in and get started.

An overview of this post is:

Powered on NAS (which was uneventful, fortunately - last time I had some errors to fix)
Updated network settings (network is now 192.168.1.x instead of 192.168.0.x)
Got backup running again
Updated jails
Re-added network drive

Upgrading transmission-jail

Got an alert saying that the transmission-jail plugin has an update available:

Django REST framework performance (part 2: techniques)

Fri, 29 Sep 2023 12:00:00 +0100

This is part 2 in a series of notes about how to improve the performance of Django REST framework.

NiceGUI with Click, Poetry, auto-reload and classes

Wed, 27 Sep 2023 12:00:00 +0100

It’s very easy to get up and running with NiceGUI, and the development cycle is fast because it has an auto-reload feature, where the web page automatically updates whenever you save a change to your project’s Python code.

However, almost all the official NiceGUI examples show UI code being written at the top-level, outside of a class, and outside of a main guard.

FastAPI: Pretty print JSON

Mon, 25 Sep 2023 15:00:00 +0100

FastAPI is a Python web framework for building APIs. The default response class is JSONResponse, but to improve performance you can install orjson and switch to ORJSONResponse.

The steps below cover how to enable pretty printing with ORJSONResponse in both FastAPI and NiceGUI.

Django REST framework performance (part 1: profiling)

Fri, 22 Sep 2023 12:00:00 +0100

This is part 1 in a series about how to improve the performance of Django REST framework.

NiceGUI: File upload and download

Thu, 21 Sep 2023 12:00:00 +0100

If your NiceGUI application has some sort of state or configuration, it can be useful to provide a way to save and load that state. We can do that through serializing the state into JSON, and providing a way to download and upload that JSON file. Following is a simple example.

NiceGUI: Show a confirmation popup

Tue, 19 Sep 2023 11:00:00 +0100

The NiceGUI dialog element can be used to create a confirmation popup or modal. With an awaitable dialog, it is easy to create an “Are you sure?” popup, to let a user confirm before doing an action. The official documentation provides a good example of a single dialog, but if you were to follow the same approach for multiple dialogs then the boilerplate code quickly multiplies.

To minimise the boilerplate code, you can re-use the same popup in multiple places.

NiceGUI: Always show main scrollbar

Fri, 15 Sep 2023 08:00:00 +0100

In a NiceGUI application, it can be annoying if the main vertical scrollbar appears and disappears depending upon whether there is more content than fits in the screen. This issue is of course not unique to NiceGUI, but just part of how HTML styling works by default.

Poetry: Fixing permission error when upgrading dulwich

Wed, 13 Sep 2023 09:00:00 +0100

When attempting to upgrade from Poetry v1.5.1 to v1.6.1 by running poetry self update I got the following error:

Python: Using isort with Black to improve your code base

Thu, 07 Sep 2023 12:00:00 +0100

isort is a Python package that sorts the order of your imports, and groups them into sections by type. So while a code formatter like Black will ensure the spacing and line length of your imports is correct, it will not modify their order.

So why bother ordering your imports?¹

Poetry: Running Black and isort with pre-commit hooks

Mon, 04 Sep 2023 14:00:00 +0100

Git Hooks provide “a way to fire off custom scripts when certain important actions occur”, and are typically stored in .git/hooks within your Git repository. For example, the Git pre-commit hook runs just before files are committed (e.g. when you run git commit), and so is a useful time to perform linting.

Poetry: Fix warning about sources

Thu, 24 Aug 2023 10:00:00 +0100

In Poetry, if you have defined custom sources but have not set a default source, you will get the following warning:

GitLab: Enable coverage reporting with pytest

Wed, 23 Aug 2023 21:59:00 +0700

GitLab supports coverage reporting. There are essential two types of coverage:

Total coverage, which is a single percentage displayed next to a job (or in Merge Requests, or badges).
Detailed coverage, which is a line-by-line coverage visualisation.

Following are the steps for configuring pytest to generate coverage statistics, and hook both of these coverage types up to GitLab.

Sphinx: Cross-referencing with MyST Markdown

Mon, 21 Aug 2023 11:30:00 +0100

In Sphinx, using the MyST extension enables using Markdown (instead of ReStructuredText). To create a cross-reference, first you must enable auto-generated header anchors by adding this line to your conf.py:¹

Poetry: Automatically generate package version from git commit

Wed, 09 Aug 2023 11:30:00 +0100

When building a Python package that is stored in git, it can be convenient to automatically generate the package version from the git commit and/or tag. That way there is no need to manually “bump” the version number in your Python code. Instead, each commit in git automatically generates a version such as:

Poetry: build.py example

Mon, 07 Aug 2023 12:00:00 +0100

Poetry contains an undocumented feature which allows running custom code when poetry build is executed. This can be particularly useful to, for example, compile a C extension, or perform some sort of pre-processing. This is done via a build.py and some configuration.

As can be seen by reading this thread, there has been a lot of churn over how to use this feature¹. Given that it’s undocumented, this comes with all the usual caveats regarding stability. But for now it appears to be the only way to call custom code during poetry build, which for some cases is a necessity.

So, without further delay, following is an example of using this feature, as it works today.

Poetry: Fixing dubious ownership error

Thu, 03 Aug 2023 12:00:00 +0100

On a Python project using poetry, when running poetry build I got the error:

Debugging missing UDP packets with Wireshark

Thu, 03 Aug 2023 07:00:00 +0100

I had a device connected via Ethernet to a Windows 10 PC. The device was sending UDP packets to the PC, where a Python application was listening for them. Python could see no packets arriving, however, if I ran Wireshark on the PC I could see the packets arriving.

GitLab CI: Using a private project's container in a Dockerfile

Sun, 23 Jul 2023 19:10:00 +0100

In GitLab, if you have a Container Registry set up for a private project (“Project A”), and you wish to use one of those containers in the FROM <image> field of a Dockerfile in another project (“Project B”) to create a new container, which you will then push to Project B’s Container Registry, e.g.:

GitLab CI: Using a private project's container as an image

Sun, 23 Jul 2023 19:00:00 +0100

In GitLab, if you have a Container Registry set up for a private project (“Project A”), and you wish to use one of those containers in the image: field of .gitlab-ci.yml in another project (“Project B”), e.g.:

Installing Cartopy on Ubuntu

Tue, 18 Jul 2023 09:00:00 +0100

The Python package Cartopy has some external dependencies, meaning installation is more complicated than just pip install cartopy. The official documentation is a great starting point for Ubuntu, but you may need to follow additional steps for the external dependencies.

Hugo: Setting up Plausible analytics

Tue, 04 Jul 2023 20:00:00 +0100

Add Plausible analytics to Hugo

Follow the official installation instructions for installing Plausible as a Hugo Module.

Convert website to a Hugo Module

Run hugo mod init. This creates go.mod and go.sum.

For more details see this blog.

Add Plausible as Hugo Module

Update config.toml with:

config.toml

[module]
[[module.imports]]
path = "github.com/divinerites/plausible-hugo"

Then run hugo mod get -u.

Add Plausible params section to `config.toml`

Add a new [params.plausible] section to config.toml:

Memory allocation error in WAV files created by arecord

Mon, 19 Jun 2023 16:00:00 +0000

The ALSA tool arecord provides a very easy way to record audio. You can run it in the terminal to start recording audio, which is progressively written to a WAV file. You then press Ctrl+C to stop recording. The resulting WAV file can then be played back or edited in a variety of programs.

Normalizing a Django model

Wed, 19 Apr 2023 20:00:00 +0700

As your Django database schema changes over time, inevitably one day you will need to normalize a model. Typically this consists of pulling out a column into its own table, and using foreign keys to store the relationship.

Applying this change involves writing a migration which does two main things: updates the schema, and migrates the data. In Django it’s quite easy to make the schema change, but it’s also quite easy to write a very inefficient data migration.

These notes focus on the problem of writing an efficient data migration.

Celery and systemd: how to avoid a restart loop

Fri, 03 Feb 2023 17:30:00 +0700

TL;DR: If using Celery as a systemd service with just one worker, I would suggest using Type=simple and celery worker, rather than Type=forking and celery multi, to avoid a potential race condition where systemd repeatedly restarts Celery.

Handle FreeRTOS tick rollover with 64-bit ticks

Thu, 26 Jan 2023 20:00:00 +0700

In FreeRTOS, with a 32-bit systick running at 1000Hz (i.e. configTICK_RATE_HZ = 1000), the maximum duration it can run before a rollover occurs is ~50 days:¹

Print a 64-bit integer using 32-bit integers

Thu, 26 Jan 2023 18:00:00 +0700

To print a 64-bit unsigned integer in a portable way, ideally you would do as follows:

VMware Workstation VM gets sluggish then locks up

Mon, 09 Jan 2023 16:13:00 +0700

TL;DR: Following Windows update 22H2, Ubuntu (and other VMs) running inside VMware Workstation 16 get progressively slower, over the course of about 30 minutes, before eventually locking up. This can be fixed by making a small config change.

Update 2023-12-12: Added Fix 2, and marked it as the preferred solution, since Fix 1 prevents Docker from working.

Hugo anchors next to headers

Sat, 17 Sep 2022 12:35:00 +0700

This note documents how to add anchors to your headings in hugo, which show up only upon hover.

Collecting Netlify Analytics data with Python

Mon, 16 May 2022 18:00:00 +0700

As you can see here, Netlify Analytics only spans the past 30 days:

It’s pretty quiet around here…

This could easily be solved if we could download the data and retain it ourselves. However, there is no way in the Netlify Analytics UI to download the data.

But what if we take a look under the hood?

Decimal error when upgrading from Django 3.1 to Django 3.2

Thu, 05 May 2022 18:00:00 +0700

Attempting to upgrade from Django 3.1 to Django 3.2 caused one of our Decimal handling unit tests to fail.

We run the unit tests against two backends: MySQL and SQLite. Interestingly, the failure only occurred against the MySQL backend.

Setting up CMocka with CMake for Linux and Windows

Tue, 26 Apr 2022 18:30:00 +0700

CMocka is a unit testing framework for C. These notes cover how to configure CMake for an existing C project to use CMocka, for both Linux and Windows, in the following steps:

Install CMocka.
Update your CMake modules to find CMocka.
Update your CMakeLists.txt to link against CMocka.
Update your project structure to add a tests directory.
Add a basic CMocka test to your project.
Run the test.
Add a more advanced test which hooks into your project.

In these notes I’m using Windows 10 and Debian 9.3 “stretch”. For a terminal, on Windows I’m using Git Bash.

Porting a simple SDL2 game to Emscripten

Thu, 21 Apr 2022 18:00:00 +0700

In these notes, we take a simple SDL2 game written in C, which builds on both Linux and Windows 10, and port it to Emscripten so that the game works in the browser. This assumes we have already set up the Emscripten build, which was covered earlier for Windows and Linux. This requires a few changes to the game’s source code, which are:

Fixing the compiler errors and warnings.
Adding some conditional #ifdef for Emscripten-specific code.
Add data (such as images) to the build, so that images display in the browser.
Update the main event loop to use the Emscripten emscripten_set_main_loop function, so that we do not cause the browser to freeze.
Increase the memory size to permit the game to run.

We’ll cover these points in more detail below.

Our first goal is simply to get a successful build, and our second goal is to ensure that build plays nicely in the browser.

Setting up Emscripten with CMake on Linux

Thu, 21 Apr 2022 09:00:00 +0700

Setting up Emscripten with CMake on Linux is largely the same as the earlier instructions for Setting up Emscripten with CMake in Git Bash on Windows 10. The steps about creating Python aliases can be skipped, since they only apply to Git Bash. The rest is then the same, since using Git Bash means we can use the Linux commands.

As a quick refresher, assuming you have a git repository with emsdk as a submodule, the steps for getting started on Linux are as follows:

Setting up Emscripten with CMake in Git Bash on Windows 10

Wed, 20 Apr 2022 09:00:00 +0700

These notes cover in detail how to set up Emscripten with CMake in Git Bash on Windows 10.

Network adapter disappears after reboot

Wed, 20 Apr 2022 07:50:00 +0700

At some point recently my Windows 10 computer developed an issue where the Ethernet network adapter would seemingly disappear after waking up from sleep. I would then have no network connection until restarting the machine.

A portable timegm alternative

Tue, 05 Apr 2022 09:28:11 -0700

Today I ran into an issue on an embedded software platform. The timegm function is nonstandard. My host unit tests pass, because timegm exists on the host, but it fails to build for the target platform because the function does not exist on that platform.

Fortunately, there is an easy solution. However, it was well hidden in my searches.

In the notes section of the timegm man page, there is a portable implementation:

Fixing apt-key deprecation on Debian 9 ("stretch")

Wed, 30 Mar 2022 14:13:00 +0700

Recently apt-key has been deprecated, which has meant that running apt update on my Debian 9 (“stretch”) machine started giving out warnings such as Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8). In these notes are documented how I fixed these warnings for:

Dropbox
Sublime Text
Enpass
NodeJS (Node Source)

Overview

The general process is, for each application:

Find the key (usually on the application’s website somewhere).
Install the key to /usr/share/keyrings/<application>-archive-keyring.gpg. * Depending upon whether the key is ASCII, non-ASCII or a keyserver affects how we install the key.
Update /etc/apt/sources.list.d/<application>.list to add [signed-by=/usr/share/keyrings/<application>-archive-keyring.gpg].
Run apt-key del to delete the key from /etc/apt/trusted.gpg.

The process for each application is detailed below, since there are some special steps.

The Important Files (part 7): Switching to TrueNAS after 2 years powered off

Fri, 13 Aug 2021 21:58:00 +0800

Three years ago, I first set up my NAS using FreeNAS (see part 1). A few months later it was powered off, placed in a shipping container along with the rest of my household contents, as I moved across the world from the UK to Seattle. Now that I’ve been in Seattle for almost two years, I’ve finally got around to setting up my NAS again(!). These notes cover the process of getting it up and running, recovering from a dying USB stick, and bringing everything up-to-date.

An overview of this post is:

Powered on NAS
Unable to boot due to error message
Determined error was due to USB stick used for FreeNAS OS dying
Discovered that FreeNAS has become TrueNAS
Used ddrescue to clone dying USB stick
Resurrected NAS using cloned USB stick
Decided to use SSD instead of USB stick to avoid same issue happening in future
Ordered parts to fit extra SSD in NAS enclosure
Installed FreeNAS on SSD
Exported NAS config from USB stick
Imported NAS config onto SSD
Upgraded OS on SSD from FreeNAS to TrueNAS
Verified backups still work

Notes on history of LTE and 4G

Thu, 07 Jan 2021 12:30:00 +0000

Overview

LTE was designed by 3GPP (Third Generation Partnership Project)
LTE is known in full as 3GPP Long-Term Evolution
LTE evolved from UMTS (Universal Mobile Telecommunications System), which in turn evolved into GSM (Global System for Mobile Communications)

Permission denied in Docker container upon COPY and RUN of a file from git

Fri, 16 Oct 2020 12:17:42 -0800

TL;DR: Using COPY to copy files into a Docker container retains the permissions from the host system. If host obtains the files from git, make sure that file attributes are set correctly in git to avoid getting a “Permission denied” error when using RUN to execute these files in the Docker container.

Docker build invalid reference format

Thu, 08 Oct 2020 17:17:42 -0800

TL;DR: If “Container Registry” is not enabled for your project, docker build will give a cryptic error such as: invalid argument ":master" for "-t, --tag" flag: invalid reference format

nmcli tips

Wed, 12 Aug 2020 10:41:42 -0700

nmcli is used to control NetworkManager from the command line. This can be very useful to troubleshoot and configure headless machines.

Workaround for installing Dropbox on Debian with libpango transitional package

Sun, 31 May 2020 12:24:00 +0700

tl;dr:

On Linux, Dropbox depends upon libpango1.0-0, which was renamed to libpango-1.0-0 over six years ago. In the meantime, libpango1.0-0 became a deprecated, “transitional package”, which depends upon libpango-1.0-0 as a means of easing the renaming transition. A month ago, Debian removed the transitional package. Until Dropbox update their package to depend upon libpango-1.0-0, it is no longer possible to do apt install dropbox.

The best workaround for now is to create your own transitional package:

apt install equivs
mkdir /tmp/pkg
cd /tmp/pkg
cat <<EOF >>libpango1.0-0
Package: libpango1.0-0
Version: 1.44.7-4
Depends: libpango-1.0-0
EOF
equivs-build libpango1.0-0
apt install ./libpango1.0-0_1.44.7-4_all.deb
apt-mark auto libpango1.0-0

Removing a third-party Django app and its tables

Thu, 21 May 2020 15:30:00 +0700

Removing an app from Django is fairly straight-forward, if its an internal app, i.e. one for which you control the source code. In a nutshell, you delete your models, then run makemigrations to generate a migration that deletes the tables from the database, and then you can delete the app.

But if it is an external (third-party) app, i.e. one for which you don’t control the source code, the same process isn’t possible. Since the migrations for deleting the tables cannot be added to the app that you don’t have control over.

I had assumed it would be a common use-case to want to remove an app from Django, and to also remove all its associated tables. However, either my search-fu has failed me, or it’s not as common as I had assumed.

Running inkscape_merge against Inkscape v1.0

Sun, 17 May 2020 19:00:00 +0700

Just upgraded Inkscape from v0.92 to v1.0. This caused my inkscape_merge tool to break, with the error:

$ inkscape_merge
bash: /usr/local/bin/inkscape_merge: /usr/bin/ruby2.5: bad interpreter: No such file or directory

Sonic Pi on Debian

Sat, 16 May 2020 18:30:00 +0700

Sonic Pi allows you to compose music with code, and edit it on the fly.

I heard about it through Hacker News.

mypy tips

Fri, 17 Apr 2020 19:30:00 +0700

Following are some tips from my experience of using mypy.

SSH tips

Thu, 09 Apr 2020 19:30:00 +0700

Following are some tips to help remind myself how to perform SSH tasks that I do less frequently.

mypy and verbose logging

Tue, 24 Mar 2020 19:30:00 +0700

tl;dr:

If you want to add a custom logging level to Python (e.g. verbose), and you want it to play nicely with mypy (e.g. to not report error: "Logger" has no attribute "verbose" every time you call logger.verbose()), there doesn’t seem to be a better way than appending # type: ignore to every call to logger.verbose():

init_verbose_logging()
logger = logging.getLogger(__name__)
logger.verbose("this is logged at verbose level") # type: ignore

I’d love to find a better way to do this, so please reach out if you know a solution. I’ve done quite a lot of digging into this, but I’m new to mypy, so perhaps I’ve missed something. Regardless, read on for the story on how I came to this conclusion, and what the worse alternative solution is.

Researching generating playing cards from an SVG template using Inkscape

Sun, 02 Feb 2020 20:02:02 +0100

I’m working on developing a board game, and would like to make some playing cards for it. My thinking is that it should be possible to have one (or a few) SVG templates (made in Inkscape), which are then populated with data from a CSV (or otherwise), somewhat akin to mail-merge. This would then make it easy to iterate on the design by updating the CSV file and then automatically generating a new set of playing cards, which could be printed off.

Migrating from Tastypie to Django REST Framework

Tue, 16 Jul 2019 09:00:00 +0100

Here are some tips from my experience of migrating an existing Django application from using Tastypie for its REST API to instead using Django REST Framework.

These notes assume the following versions:

django==2.2.3
django-tastypie==0.14.2
djangorestframework==3.9.4
django-filter==2.1.0

Python abi3 wheels

Wed, 03 Jul 2019 09:00:00 +0100

When updating the Python cryptography package from v1.x to v2.7 I could not find a manylinux1 wheel for Python 3.6. It turned out I was not the only person to notice this. However what I learnt is that past v2.1.4, Cryptography publish “abi3” wheels. (See PEP 425)

Configure Python package to install dependencies only for specific combinations of Python version and platform

Thu, 07 Mar 2019 11:30:00 +0000

PEP 508 introduced environment markers which let you specify conditions for whether a package should be installed. These can be used inside the install_requires or setup_requires argument in setup.py, or within the requirements.txt.

The most simple example is:

argparse; python_version < "2.7"

This would only install argparse if the Python version is lower than 2.7.

However PEP 508 details many more enivonment markers, and specifies the grammar for combining these. For example:

baz; python_version >= "3.6" and platform_system == "Linux"

This would only install baz if the Python version is greater than or equal to 3.6, and the system is Linux.

Python click: allow user to retry input upon validation failure

Thu, 07 Feb 2019 18:00:00 +0000

It can be a frustrating experience for a user if you naïvely reject their input without giving them another chance to correct their input, or know why it is wrong:

def validator(ctx, param, value):
 """Aborts if value is too long.
 """
 if len(value) > 32:
 raise click.UsageError("Too long!")
 return value

@click.command()
@click.option("--name", callback=validator)
def assign(name):
 click.echo(f"Assigning {name}")

Using the above assign command with a namer longer than 32 characters will cause the command to abort. This frustration compounds if the command takes multiple options, since the user may have already given input to several options, only to have to retry and enter it all over again.

Instead it is much better to let the user retry straightaway if their input fails:

def validator(ctx, param, value):
 """Lets user retry, with explanation, if value is too long.
 """
 while True:
 if len(value) > 32:
 # Actually tell the user why their input was rejected
 err = "must be 32 characters or fewer"
 else:
 return value
 
 # `get_error_hint` will return `"--name-"`
 hint = param.get_error_hint(ctx)
 
 # Use styling to make the error message prominent
 click.secho(f"Error: Invalid value for {hint}: {err}", fg="red", bold=True)
 
 # Prompt the user to try again
 value = click.prompt(param.prompt)

@click.command()
@click.option("--name", callback=validator)
def assign(name):
 click.echo(f"Assigning {name}")

The above implementation is better because:

The user can try again any number of times (they can CTRL-C to abort)
The user is told why their input was rejected
(As a bonus) the user can easily distinguish at a glance that it is an error because of the styling

Call Python script from pylint init-hook

Fri, 25 Jan 2019 12:51:00 +0000

This note is deprecated as of December 2024. I recommend using source-roots instead, which is available from Pylint v2.17.7. See this note.

The .pylintrc has a init-hook field, which is a one-liner of Python code that gets executed when pylint initialises. While you can use semicolons to write multiple lines within the init-hook, this does not scale well, and does not diff well. There seems to be no way to do multi-line Python code within the init-hook, so the best way I’ve found is to use the init-hook to import a separate file, in which you can write multi-line Python code.

First, create an import_hook.py in the same directory as your .pylintrc. Put in here whatever Python code you would like. Typically this might be adding stuff to the sys.path:

import_hook.py

import sys
sys.path.append("foo")
sys.path.append("bar")
# etc...

Then update the .pylintc to import the import_hook.py. We’re being lazy here and making use of import side-effects (i.e. there is no if __name__ == "__main__" in the import_hook.py, all code just executes upon import). That could easily be avoided but it would make the init-hook a little longer:

.pylintrc

init-hook="import imp, os; from pylint.config import find_pylintrc; imp.load_source('import_hook', os.path.join(os.path.dirname(find_pylintrc()), 'import_hook.py'))"

Python DEPRECATION warning and pip --no-cache-dir breakage

Wed, 23 Jan 2019 10:48:00 +0000

Today, Python 2.7’s pip started echoing the following:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Simultaneously (and perhaps related, though perhaps coincidentally) Python 2.7’s pip started giving the following error:

 File in "...\lib\site-packages\pip\_internal\cli\base_command.py", line 176, in main
 status = self.run(options, args)
 File in "...\lib\site-packages\pip\_internal\commands\install.py", line 346, in run
 session=session, autobuilding=True
 File in "...\lib\site-packages\pip\_internal\wheel.py", line 848, in build
 assert building_is_possible
AssertionError

libusb usb_open error -4

Thu, 03 Jan 2019 13:36:00 +0000

If you get the following error with pylibftdi:

[...]
 File "/opt/myvenv/lib/python2.7/site-packages/pylibftdi/device.py", line 156, in open
 raise FtdiError(msg)
FtdiError: usb_open() failed (-4)

Make sure to first have these udev rules in place:

/etc/udev/rules.d/11-ftdi.rules

# The MODE="0666" is necessary, else pylibftdi will get error -4 when it calls libusb

SUBSYSTEM=="usb", ATTR{idVendor}=="1234", ATTR{idProduct}=="5678", GROUP="plugdev", MODE="0666"

These are all the possible libusb error codes:

LIBUSB_SUCCESS = 0
LIBUSB_ERROR_IO = -1
LIBUSB_ERROR_INVALID_PARAM = -2
LIBUSB_ERROR_ACCESS = -3
LIBUSB_ERROR_NO_DEVICE = -4
LIBUSB_ERROR_NOT_FOUND = -5
LIBUSB_ERROR_BUSY = -6
LIBUSB_ERROR_TIMEOUT = -7
LIBUSB_ERROR_OVERFLOW = -8
LIBUSB_ERROR_PIPE = -9
LIBUSB_ERROR_INTERRUPTED = -10
LIBUSB_ERROR_NO_MEM = -11
LIBUSB_ERROR_NOT_SUPPORTED = -12
LIBUSB_ERROR_OTHER = -99

Troubleshooting wifi driver crashing (part 4)

Thu, 29 Nov 2018 20:16:00 +0000

The USB Wi-Fi adapter I ordered a month ago arrived today! It took so longer because the first order got lost in the post. Once I reached out to the Think Penguin support, they were very quick to help and sent my order again. It arrived 3 days later.

Now let’s try and get this USB Wi-Fi adapter working. (Skip to the end to see how to actually do it. There is a lot of meandering first…)

Making Visio 2016 Usable

Wed, 14 Nov 2018 10:44:00 +0000

The bane of my experience with Visio usually comes down to connectors; being unable to attach them where I want, being unable to adjust the lines to where I want, and then once I’ve got them in place, they start jumping around by themselves when something else is moved. Fortunately I found a few tricks to help ease the pain.

Use the “Connection Point” command to adjust where connectors connect to an object

If you’re like me, you’ll have never even noticed this button existed. Under the Home tab next to the Connector button, there is a blue “X” symbol called “Connection Point”. Select this, then select an object. You can move existing connection points by clicking on them and dragging. You can delete existing connection points by selecting them and pressing delete. You can create new connection points by holding down Ctrl and clicking.

Troubleshooting wifi driver crashing (part 3)

Wed, 24 Oct 2018 20:30:00 +0100

We’re going to try and downgrade iwlwifi from version -22 to -17, roughly following advice from here.

This person also has trouble with the -22 version:

Other people reported problems with the -21 version:

Okay, first stop apt from trying to upgrade this package:

sudo apt-mark hold firmware-iwlwifi

Then remove the -22 and -21 version. This should force the kernel to load the previous version which is -17 from what I can see in /lib/firmware/.

sudo mv /lib/firmware/iwlwifi-7265D-22.ucode .
sudo mv /lib/firmware/iwlwifi-7265D-21.ucode .

Time to reboot…

Troubleshooting wifi driver crashing (part 2)

Thu, 18 Oct 2018 20:06:00 +0100

Going to attempt upgrading to a more recent iwlwifi firmware version to see if that fixes my wifi issues. Using the Debian testing distribution instead of stable allows me to upgrade from iwlwifi -22 to -29:

Install Hugo testing distribution on Debian

Sun, 14 Oct 2018 21:01:00 +0100

On a default installation of Debian, trying to install hugo with apt install hugo will fetch the latest stable version of Hugo, which at time of writing is v0.18.1, and was released in December 2016! I would like to install the more recent version 0.47.1 which came out last month in August 2018. However it is on the testing distribution so by default cannot be installed. To enable installing the testing version of Hugo, follow these steps:

Debian center window keyboard shortcut

Sun, 14 Oct 2018 18:24:00 +0100

On Debian Stretch (9.5), wanting to add a keyboard shortcut which un-maximises the current window and centers it, making it slightly smaller than the screen resolution, so that some of the background can be seen.

Finds the Mozilla Firefox window and stops it from being maximised:

wmctrl -r "Mozilla Firefox" -b remove,maximized_vert,maximized_horz

Finds the Mozilla Firefox window and sets it to a specific size. Note that if the window is maximised, this will appear to do nothing:

wmctrl -r "Mozilla Firefox" -e 0,210,80,1500,900

Uses xdotool to get the active window rather than hard-coding “Mozilla Firefox”. Combines the above two commands into a one-liner:

wmctrl -ri "$(xdotool getactivewindow)" -b remove,maximized_vert,maximized_horz && wmctrl -ri "$(xdotool getactivewindow)" -e 0,210,80,1500,900

This one-liner can then be bound to a keyboard command.

In Debian, go to Settings → Keyboard, then scroll to the bottom and click the plus sign to add a new keyboard shortcut. I pasted the above command, named it “Center window” and bound it to Ctrl+Super+Down.

Troubleshooting wifi driver crashing (part 1)

Sat, 13 Oct 2018 12:20:00 +0100

Perhaps a couple of months ago (unfortunately I’m not sure exactly when) I started experiencing the following wifi problem on my laptop: the wifi connection would die, and nothing except rebooting the machine would fix the problem. Sometimes it would happen after the laptop had been on for hours, and sometimes it would happen within a minute of booting, but it was inevitable that at some point the wifi would die.

I had been hoping this issue would go away in a future update, but it persisted and became ever more irritating, since each time it struck I had to close everything and reboot. So at last I began to dig into this issue with hopes to find a resolution.

Bootstrapping pip

Fri, 31 Aug 2018 15:38:00 +0100

What doesn’t seem to be documented in the pip installation guide is that pip can bootstrap itself. You can install pip using pip without pip being already installed. No internet access is required, just an installation of Python and the pip wheel:

python pip-18.0-py2.py3-none-any.whl/pip install pip-18.0-py2.py3-none-any.whl

Invalid handle error from Coverage inside Tox

Fri, 31 Aug 2018 14:06:00 +0100

My tox py36 task was failing with a The handle is invalid error, and the traceback showed it was somewhere inside Coverage v4.2.2:

Run Django makemigrations as tox task

Thu, 30 Aug 2018 14:26:00 +0100

The following is an example Windows batch script which runs makemigrations inside a tox virtual environment, then copies the migrations out of the virtual environment into your src directory. This is very useful for making changes to your Django model and generating the migrations on your Windows development machine. It would be straight-forward to convert this to run on Linux too.

REM Ensure the py36 virtualenv present
tox -e py36 --notest

REM Activate the py36 environment
call .tox\py36\Scripts\activate.bat

REM Execute the makemigrations command
REM (Lots of quote escaping going on!)
python -c "from django.core.management import execute_from_command_line; execute_from_command_line([\"\", \"makemigrations\", \"--settings=my_django_app.settings\"])"

REM Copy the migrations from the py36 virtualenv into the src
copy /Y .tox\py36\Lib\site-packages\my_django_app\migrations\* src\my_django_app\migrations\

New dependency font(:lang-en) required by fontconfig-2.10.95-11

Tue, 28 Aug 2018 10:57:00 +0100

I upgraded fontconfig on a machine that does not have internet access, from fontconfig-2.10.95-10 to fontconfig-2.10.95-11, and got the following error:

error: Failed dependencies:
 font(:lang=en) is needed by fontconfig-2.10.95-11.el7.x86_64

This dependency upon font(lang=en) was added in #1403957 (which I do not have access to) and a related bug was raised in #1484094 (which I do have access to!).

Installing rpms offline using a local yum repository

Fri, 24 Aug 2018 18:00:00 +0100

If you have a CentOS 7 machine that does not have internet access, resolving dependencies when installing rpms can be tricky. Rather manually trying to resolve all dependencies (e.g. rpm -Uvh blah.rpm blah-dependency-1.rpm ...etc), it is much simpler and more maintainable to create a local yum repository. Then, with only a couple of extra command line options, you can use yum to install dependencies from your local repository and let it resolve all the dependencies for you (e.g. yum --disablerepo=\* --enablerepo=offline-myrepo install blah). This page documents how to achieve that.

The Important Files (part 6): Linux Samba share

Tue, 07 Aug 2018 21:15:00 +0100

Trying to fix operation not permitted when trying to write to NFS share.

Using SMB and NFS together is perhaps not safe:

https://forums.freenas.org/index.php?threads/cifs-and-nfs-together.10918/

Let’s get rid of the NFS share and just use SMB.

The Important Files (part 5): Setting up borg backup to sync.net

Tue, 31 Jul 2018 22:28:00 +0100

In these notes I configure borg on my FreeNAS jail (named fnbbu) to automatically backup up every night to both a local repository, and a remote repository at rsync.net.

rsync.net vs Wasabi

First, a quick comparison of rsync.net and Wasabi on price alone. Note that there are many aspects we could use to compare them (uptime, data center distribution, technical support, etc.) so this is not a well balanced comparison.

The Important Files (part 4): Setting up borg in a jail

Sun, 29 Jul 2018 23:00:00 +0100

In these notes I create a FreeNAS jail (using iocage rather than warden), install borg inside the jail, and use borg to communicate with a remote borg server hosted by rsync.net. Initially I try using Wasabi rather than rsync.net, but decide against Wasabi.

As usual, the notes were gathered as I ran through the process, and tidied up a little afterwards, so this is not a slick guide. I go back on myself and try alternative routes.

How to reload Ethernet udev rules without reboot

Thu, 19 Jul 2018 15:55:00 +0000

This note detail how to update udev rules for an Ethernet connection and have them take effect without rebooting the machine. The critical part is using rmmod and modprobe to remove and re-load the networking module for the Ethernet connection, which effectively “unplugs” and “replugs” the connection, causing the udev rules to take effect. Note that this will cause brief downtime for the specific network interface, but avoids rebooting the entire machine.

First lets run ip addr to show the interfaces we are starting with:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: ens33u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
 inet 192.168.0.12/24 brd 192.168.0.255 scope global dynamic ens33u1
 valid_lft 2460sec preferred_lft 2460sec
 inet6 fe80::20c:29ff:fe1a:4605/64 scope link
 valid_lft forever preferred_lft forever

Next we add our udev rule, which should give the Ethernet interface with the MAC address aa:bb:cc:dd:ee:ff the interface name ethfoo.

Modify bash history

Tue, 10 Jul 2018 16:21:00 +0000

Using bash history to try and find a previous command, I found it had vanished and there was a mysterious * present against the line, which I had never seen before:

 56 vi /etc/resolv.conf
 57 sudo systemctl restart dnsmasq
 58*
 59 curl foo.bar

Some searching found out this indicates the line has been modified.

It turns out modifying bash history is rather simple: press up until you find the line, modify it, then press down. I must have modified the history by accident.

Freeing up disk space on Arch Linux

Fri, 06 Jul 2018 20:55:00 +0100

Visualising what is using up disk space

To visualise what is taking up disk space I use qdirstat:

$ sudo pacman -S qdirstat
$ qdirstat

Launching it opens a GUI which can be used to see what is hogging disk space.

The pacman cache is taking up a significant chunk.

The Important Files (part 3): Create a dataset and shares

Sat, 23 Jun 2018 10:00:00 +0100

In these notes I create a dataset in FreeNAS, then set up the Windows (SMB) and Linux (NFS) shares.

To create the user and Windows shares, I was helped by this guide.

The Important Files (part 2): Install FreeNAS

Tue, 12 Jun 2018 10:00:00 +0100

In these notes I write the FreeNAS ISO to USB and use it to install FreeNAS onto an HP Proliant Microserver; the server has four 3.5" SATA HDDs which will be configured as RAID, and one USB stick which will house the FreeNAS operating system. (FreeNAS cannot be installed on its own storage volumes).

Calling iPerf3 from Python

Wed, 06 Jun 2018 16:12:00 +0100

Creating the Client with iperf3-python

If when calling this:

>>> import iperf3
>>> client = iperf3.Client()

You get this error:

Exception AttributeError: "'Client' object has no attribute '_stdout_fd'" in <bound method Client.__del__ of <iperf3.iperf3.Client object at 0x2505f90>> ignored
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/opt/myapp/virtualenv/lib/python2.7/site-packages/iperf3/iperf3.py", line 414, in __init__
 super(Client, self).__init__(role='c', *args, **kwargs)
 File "/opt/myapp/virtualenv/lib/python2.7/site-packages/iperf3/iperf3.py", line 110, in __init__
 lib_name
OSError: Couldn't find shared library libiperf.so.0, is iperf3 installed?

Try explicitly setting lib_name to the location of your libiperf.so.0. For example:

>>> import iperf3
>>> client = iperf3.Client(lib_name='/usr/local/lib/libiperf.so.0')

The Important Files (part 1): The goal

Sat, 02 Jun 2018 10:00:00 +0100

Our irreplacable family photos are stored scattered throughout the following diverse locations:

My phone
My old phone
My wife’s phone
My wife’s laptop
My DropBox
My old desktop PC
My old Mac
The SD card in our camera
The SD card in a box in the loft
The DVD of our wedding given to us by my Uncle

My old desktop PC also contains the source code for all video games I’ve developed from when I was a teenager (read: did not use version control).

I refer to all of the above as “The Important Files”. They are all files that we would like to hold on to, are irreplacable, and would feel saddened if it was lost.

In this note I begin the plan to design and assemble a solution.

Debugging a Python process not terminating due to active threads

Wed, 16 May 2018 11:41:00 +0100

Situation

A multi-threaded Python process is not terminating because a non-daemon thread is still alive. Given a running Python process, we want to identify the name of this thread.

Lint Bash on Windows

Tue, 15 May 2018 09:25:00 +0100

Install Scoop, a command line installer for Windows.
Using Scoop, install ShellCheck, a Bash linter.
Within Sublime, install SublimeLinter and SublimeLinter-shellcheck.

Convert Emacs org files to markdown

Thu, 26 Apr 2018 22:35:15 +0100

What is Orgmk?

Orgmk is a suite of bash Bash scripts for automating the conversion of Org mode documents to different formats (such as HTML or PDF)

Or in my case, to markdown!

The documentation for Orgmk is thorough and clear, so you could do little better than to read the official documentation. While it hasn’t been updated in 2 years, that is perhaps because it largely does what it needs to do. However towards the bottom of the readme it becomes apparent that the author intended to do more.

Emacs package managers

Thu, 26 Apr 2018 20:45:33 +0100

Starting with Emacs 24, package.el is included with Emacs which provides a built-in package manager.

You can list packages:

M-x package-list

And install packages:

M-x package-install [RET] org

Make temporary network changes permanent under CentOS 7

Wed, 25 Apr 2018 10:47:00 +0100

Say you make some temporary changes to your network configuration in CentOS 7. For example, assigning an IP address to an interface:

ip addr add 10.12.23.42/24 dev ens192u1

In order for the above change to become permanent it needs to be written to file at /etc/sysconfig/network-scripts/ifcfg-ens192u1.

Rather than creating this file manually, you can automatically generate it from the current network configuration (including your temporary changes).

Run nmtui (as root), select “Edit a connection”, select your connection, then without making any changes select “OK” and then quit. It will have generated the ifcfg-ens192u1 file for you if it did not exist already.

Lightweight Python SIP library

Fri, 20 Apr 2018 12:07:00 +0000

Situation

Need to create SIP requests and parse SIP responses as defined in RFC 3261. Do not need a “heavyweight” solution that runs a SIP server/proxy. Needs to support Python 2.7 and 3.6.

Removing unused COM ports from Windows

Thu, 19 Apr 2018 14:00:00 +0000

Through Device Manager in Windows you can view which COM port has been assigned to which device. Each new device gets assigned the next COM port number, starting from COM1. If over the years you have connected many different devices which are assigned COM ports, this is not forgotten:

Hidden devices hidden

It may be desirable to unassigned COM ports from devices you no longer use. This way, when you connect your new device it might get assigned COM2, for instant, rather than COM46.

Attach shell to running Python process

Mon, 16 Apr 2018 09:47:01 +0000

pyrasite-shell (part of pyrasite) can be used to open a shell in an already running Python process.

Installation

It can be installed on CentOS 7 with the following:

$ yum install gdb
$ pip install pyrasite==2.0

Hugo 0.37 does not support h6 markdown heading

Mon, 19 Mar 2018 20:55:00 +0000

Posts for this website are written in markdown, and the HTML you read is generated by hugo. Recently, when trying to use ###### blah for a markdown heading, which should generate a <h6> HTML tag, I found that hugo started generating a table of contents for the post. Remove the heading, or reduce it to ##### blah, and the table of contents goes away.

Normally a table of contents would only be shown for a post if the {{ .TableOfContents }} tag is used. However if an h6 markdown heading is used, then a table of contents is generated and there is no way to stop that happening.

Bind hugo to localhost for development

Sun, 18 Mar 2018 20:30:00 +0000

Situation: Want to run hugo on my laptop and access the website from my phone over wifi so that I can debug an issue that only occurs on my phone (issue does not occur in responsive mode using developer tools).

By default hugo binds to localhost which is only accessible internally (e.g. from on the same computer). Typically binding to 0.0.0.0 will allow a server to be accessed externally (e.g. from another computer on the same network). However, just doing:

hugo --bind=0.0.0.0

Does not completely work. Hugo will still use localhost to try and load resources (e.g. images, stylesheets).

Calling MPSSE over FTDI from Python on CentOS 7

Mon, 12 Mar 2018 10:00:00 +0000

If wanting to use MPSSE commands with an FTDI device, you cannot simply use Python pyserial’s serial.Serial(). Instead you must use the FTDI driver in order to switch the device into MPSSE mode.

Hugo tag and category pages

Sun, 11 Mar 2018 15:38:00 +0000

Taxonomies are ways of grouping content within Hugo. By default, Hugo defines two taxonomies: “tags” and “categories”. You can then, for example, write a post and define tags = ["test"] in the front matter. Hugo will then generate a taxonomy list page at https://mywebsite.com/tags/ and you can view all posts with the "test" tag at https://mywebsite.com/tags/test.

As far as Hugo is concerned, there is no difference between “tags” and “categories”. Each post can have as many tags and/or categories as you would like!

Conceptually though, I consider “categories” to be the broad topic, and “tags” to be more specific aspects. For example, for this post I might set:

categories = ["software", "web-dev"]
tags = ["Hugo", "taxonomies"]

Plural vs singular taxonomies

So far we have been using “tags” plural. For this website, I instead wanted the taxonomy list page to be https://mywebsite.com/tag singular. It turns out this is rather easy!

Calling C from Python

Fri, 09 Mar 2018 10:00:00 +0000

ctypes is Python library for calling C code from Python. Following is a very simple example of how to build a shared object (*.so file) from C code which can be called from Python using ctypes.

Building against `libusb`

Thu, 08 Mar 2018 15:40:00 +0000

Situation: given an existing small C program which was developed on Ubuntu 12.04 32-bit, build it on CentOS 7 64-bit with minimal, if any, modifications to the source. The C program depends upon libftdi and libusb, of which there are four “versions”.

Get errno from Python requests `ConnectionError`

Tue, 20 Feb 2018 11:05:00 +0000

Using the Python requests library, I wanted to make a PUT request, and ignore any ConnectionError if it was an error 104 “Connection reset by peer”. (104 is ECONNRESET defined in errno.h). If this error was raised, the traceback I received was as follows:

[...]
 File "/usr/bob/venv/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
 resp = self.send(prep, **send_kwargs)
 File "/usr/bob/venv/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
 r = adapter.send(request, **kwargs)
 File "/usr/bob/venv/lib/python2.7/site-packages/requests/adapters.py", line 488, in send
 raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

Python logging: %s vs format

Fri, 16 Feb 2018 11:05:00 +0000

Using format rather than %s syntax for formatting strings in Python has many advantages.

It handles tuples more intuitively.
It allows using kwargs to more clearly name arguments.
It allows using kwargs to reuse arguments in the format string.

However when it comes to logging, often the reverse is the case.

Escaping Hugo shortcodes within Hugo markdown

Sun, 11 Feb 2018 16:45:00 +0000

If you naïvely try and write a post in Hugo about shortcodes using markdown, including code examples of the shortcodes, Hugo will still try to parse and render the shortcode. For example, if your markdown is as follows…:

Automatic image thumbnails in Hugo from static directory

Sun, 11 Feb 2018 16:30:00 +0000

Hugo v0.32 introduced image processing, which can be used to resize, fit or fill images. This can be useful to, for example, automatically create a low resolution thumbnail from a high resolution image. Hugo v0.34 (latest at time of writing) modified the API for image processing.

Using the image processing in Hugo v0.34, my goal is this:

Store my full resolution “original” images in the static directory (/static/img/)
Use Hugo to automatically generate low resolution “thumbnail” images
Use a shortcode to include the thumbnail image in markdown as a link to the original image

Captured `SystemExit` with pytest and tox

Fri, 09 Feb 2018 15:58:00 +0000

I wanted to unit test some Python code that uses sys.exit(exit_code) to verify whether the exit_code was as expected. The first answer I came across suggests using the built in capsys fixture which is enabled with setting --capture=sys when running pytest. However for me this clashed with my tox.ini:

[testenv]
commands =
 pytest -vv {posargs}

I take advantage of the {posargs} feature by sometimes running tox with arguments for pytest, for example tox -e py27 -- --capture=no in order to stop pytest from capturing stdout. Using the --capture=sys method in my tox.ini as mentioned above did not seem to play well with optionally using --capture=no when invoking pytest through tox.

Tastypie "TypeError of type 'NoneType' is not iterable"

Fri, 09 Feb 2018 10:20:00 +0000

A unit test which performed a GET to a Tastypie endpoint always raised the error:

 def check_filtering(self, field_name, filter_type='exact', filter_bits=None):
 """
 Given a field name, a optional filter type and an optional list of
 additional relations, determine if a field can be filtered on.

 If a filter does not meet the needed conditions, it should raise an
 ``InvalidFilterError``.

 If the filter meets the conditions, a list of attribute names (not
 field names) will be returned.
 """
 if filter_bits is None:
 filter_bits = []

> if field_name not in self._meta.filtering:
E TypeError: argument of type 'NoneType' is not iterable

The fix was to define filtering to at least be an empty dictionary in the Tastypie resource. For example:

Using a non-pk as a foreign key in a Tastypie endpoint

Thu, 08 Feb 2018 16:00:00 +0000

Situation: using a non-pk for the detail URL in tastypie. Then performing a PATCH to update an item which is a foreign key to a non-pk detail URL gives the following error:

{
 "my_endpoint": {
 "my_field_to_a_detail_url": [
 "Select a valid choice. That choice is not one of the available choices."
 ]
 }
}

pytest and functools decorator

Tue, 06 Feb 2018 10:20:00 +0000

If using a decorator created using functools.wraps with pytest, it will give an error similar to the following if you attempt to use fixtures:

TypeError: test_func() takes exactly 1 argument (0 given).

The solution is to use the decorator package instead of functools to create the decorator.

A comment on the above stackoverflow answer (which is over 4 years old) states that pytest depends upon the decorator package. However as far as I can tell from looking at the current setup.py for pytest, it does not depend upon decorator. However using decorator still fixed the issue.

normalize-scss with hugo

Fri, 02 Feb 2018 17:30:00 +0000

Before beginning to develop the CSS for this website, I wanted to ensure I had a consistent base style across all browsers. Originally I was going to use reset.css but upon further research decided to use normalize.css because it normalizes everything to sensible defaults rather than zeroing everything out, among other benefits.

Automating hugo development with npm scripts

Fri, 02 Feb 2018 17:00:00 +0000

I wanted one command I could run that would automatically watch my SCSS files and HTML files, and build them upon change, and serve up the resulting website. In order to avoid complicating development by introducing a task runner as an additional dependency (such as grunt or gulp) I decided to try using npm scripts. It turned out to be very simple and easy!

RabbitMQ handshake timeout caused by hostname change

Mon, 29 Jan 2018 10:00:00 +0000

Changing the hostname of our VM running rabbitmq-server from old-hostname to new-hostname caused RabbitMQ to timeout all handshakes. See this excerpt from /var/log/messages where a worker attempts to connect but the handshake times out exactly 10 seconds later.

Run Django `manage.py makemigrations` without a database

Mon, 29 Jan 2018 10:00:00 +0000

The makemigrations command does not require a database to be present. As such, it is actually quite straightforward to run makemigrations without even having a database. This can be very useful for the development cycle of: modifying model, run tests, update tests; without having to have a full deployment.

Avoid a process terminating when logging off a remote SSH session

Thu, 25 Jan 2018 09:41:00 +0000

For any arbitrary command:

nohup <cmd> > cmd_log.txt &

The & forks <cmd> into a separate process, and nohup (“no hang up”) ensures the process does not terminate once you log out. This can be useful to, for example, SSH into a remote machine and launch a long running command, without forcing you to keep the session open.

Celery pinging

Thu, 25 Jan 2018 09:41:00 +0000

In Celery, you can ping all workers with either app.control.ping() or app.control.inspect().ping(). The former has an optional timeout keyword argument, while the latter does not. Under the hood, the latter simply calls the former, but returns a slightly different data structure.

Kombu "timed out" bug: connection hangs forever in Kombu 4.1.0

Thu, 25 Jan 2018 09:41:00 +0000

Kombu connects to RabbitMQ. However in Kombu 4.1.0 there is a bug where it waits forever if the broker is down rather than timing out. This affects Celery too. There is a fix in trunk but at time of writing there is no new release of Kombu which contains this fix. A workaround is to downgrade to Kombu 4.0.2.

Manually invoking Celery app control commands from a Python package

Wed, 24 Jan 2018 09:46:00 +0000

Given a Python package named my_package installed in the virtual environment /opt/venv/, and which contains a top level file celery_app.py which defines the Celery app object, this is how you can manually invoke Celery commands (e.g. to perform a ping):

$ source /opt/venv/bin/activate
(venv) $ python
>>> from my_package.celery_app import app
>>> app.control.ping()

Tastypie and pytest

Wed, 24 Jan 2018 09:46:00 +0000

Tastypie offers a ResourceTestCaseMixin which can be inherited in conjunction with the Django TestCase to allow unit testing Tastypie endpoints in a unittest style manner. However if wanting to use pytest instead of unittest in order to take advantages of its features such as fixtures, I found a different approach had to be taken.

Debugging RabbitMQ

Mon, 22 Jan 2018 09:58:00 +0000

Monitor rabbitmq-server’s status with:

$ rabbitmqctl status

Note that it must be run with root else you will get the rabbitmq help printed out with the appended message Only root or rabbitmq should run rabbitmqctl.

SSH tunnel via gateway

Fri, 19 Jan 2018 11:24:00 +0000

To set up an SSH tunnel from here to remote via gateway, as illustrated below:

+------+ +---------+ +--------+
| here | | gateway | | remote |
| 22:+-----+:22 | | |
|50000:----------------+-----+:22 |
| +-----+ | | |
| | | | | |
+------+ +---------+ +--------+

Show whitespace in vi

Thu, 18 Jan 2018 10:35:00 +0000

Opened a configuration file with vi:

vi /etc/iproute2/rt_table

Using a different port with Fabric

Tue, 16 Jan 2018 10:39:00 +0000

By default Fabric uses port 22 because it is over SSH. However you can easily specify a different port e.g. 123 in the typical format:

$ fab -H 192.168.202.100:123 my_command

CentOS 7 firewalld

Mon, 15 Jan 2018 14:20:00 +0000

General

The full manual for firewalld on CentOS 7 can be found in section 5 of the RHEL 7 security guide.