omniverse

Using gbp-buildpackage in a Docker way

2026-02-23T00:00:00+00:00

To build a package on Debian, I was always using my server called “atom” (more on it in later posts). With Debian stable installed, I can build packages for Debian Sid as intended. My favourite way was to use pbuilder for most packages and sbuild for heavier software. It is fast and reliable, without having to spin up a standalone isolated virtual machine.

Sometimes you are not able to use a server, or it is not convenient. It is easier to spin up a Docker container to build a package or introduce a small fix. That’s why I wrote a small automation around Docker and gbp.

It is available in my GitHub repo. The main idea is to build a prebaked image with all the required tools. The user launches a container. Inside it, package-specific build dependencies are installed, gbp buildpackage is executed with post-build checks, lints are run, the package is optionally signed with GPG, and then the container exits. So it allows you to boil down the whole process to a single line:

docker compose run --rm debian --workdir /workspace/asn1c

All the sources are in the host machine’s directory (so I can easily edit them with my favourite editor) and the build artefacts (.dsc, .build, and .changes files) are placed in the same /workspace directory. No copying, no rsyncing. Feel free to use!

Also, the modern Debian infrastructure in SalsaCI builds and tests packages for you via familiar GitLab CI. It’s a slightly longer feedback loop and is well-suited for a final check. I prefer iterating fast with local builds.

That said, I tested this suite while maintaining two new versions of my packages, which recently landed in Debian testing:

asn1c 0.9.28+dfsg-6. It includes fixes to man pages and fonts, new standards, and general cleanup
pstreams 1.0.4-2. It’s a new upstream version, with new Debian standards and build tweaks.

Meeting C++ 2025 Trip Report

2025-11-27T00:00:00+00:00

This November, I had the opportunity to attend the famous Meeting C++ conference. Located in Germany, it attracts many C++ developers not only from continental Europe, but also from the UK and the USA. I am grateful to the organisers, especially Jens Weller, for accepting my proposal to be part of such a great lineup of talks across five tracks during three days. So yes, it was huge.

Let me share my thoughts about a few talks and what I’ve found especially interesting.

Software and Safety keynote

Anthony Williams worked on the C++ concurrency standard, wrote a lot of articles and a book “C++ Concurrency in Action”, which is an excellent crash course into (non-)crashing multi-threaded programs. Now he explores the automotive industry, which has a special interest in concurrency and safety from undefined behaviour in C++.

I like the “Swiss Cheese” concept Anthony carried throughout his keynote talk, “Software and Safety”. It likens the system to a stack of Swiss cheese layers with randomly placed and sized holes in each slice, mitigating the overall risk. There is no silver bullet. You can only decrease the failure risk by having multiple techniques in place, like:

Design your system to minimise the potential for problems
Use static analysis
Test with potentially problematic input
Fuzz test
Use sanitisers and hardened libraries
Use contracts

It was a perfect start to the conference, providing the tone of safety for critical systems in C++. I’ve seen and spoken with many attendees working in highly regulated environments, such as automotive, aerospace, and healthcare. All of us would prefer these safety-critical systems to be written with these ideas in mind. For regulated financial institutions, it’s not that critical, but it’s also a key factor in ensuring a safe and secure language and foundational libraries.

From Acrobatics to Ergonomics - A Field Report on How to Make Libraries Helpful

Then I moved to an excellent and entertaining talk by Joel Falcou from INRIA on designing software libraries for scientists to use. It’s a different approach compared to writing libraries for hardcore C++ developers. You cannot just throw kilobytes of template errors at users without a reasonable explanation of the errors and, more importantly, actions to fix them from a user perspective. Joel explained how concepts, type systems, and proper API design can greatly simplify the developer experience. Joel believes that library developers should care more about users, and modern C++ makes it easier.

In Rust, developers have excellent diagnostics support from the compiler, and rich support for custom diagnostics is also provided to libraries via the form of annotate-snippets-rs crate and at the attribute-level with #[diagnostic::on_unimplemented] (docs). C++ compilers are becoming increasingly capable of reducing error clutter, but there is still a long way to go.

Also, a simple and intuitive design doesn’t have to be non-extensible. The speaker explained that intuitive means the incorrect usage is flagged clearly, and all components adhere to the Single Responsibility SOLID principles. Extensibility means that users should be able to combine pieces of the API and obtain sensible results. Power users should be able to customise their API usage to their own corner cases, and finally, developers should have a clear path to extend the API.

Lessons learned from the talk.

Use Concepts and static asserts to enforce API design
Use if constexpr and static asserts to simplify metaprogramming
Prevent problems earlier with proper diagnostics

Harnessing constexpr: A Path to Safer C++

Mikhail Svetkin spoke about memory safety. Undefined behaviour can be easily stopped by LLVM sanitisers, but it’s not the only way. Developers could start sprinkling constexpr into functions, allowing them to operate not only at runtime but also at compile-time. The main side effect of constexpr is the lack of undefined behaviour. So if your constexpr unit tests work on code, it’s free of undefined behaviour. It allows unit tests for such functions to run even before they are executed! Maxing this out requires some tricky techniques that Mikhail has shown, and the result is fascinating — unit test errors are displayed in your IDE before compilation.

The constexpr support has grown extensively during the last standards. I cannot resist mentioning this evolution list:

C++11: expression
C++14: variables, control flow, array
C++17: lambdas, string view, if constexpr
C++20: string, vector, virtual functions, try, new/delete
C++23: basic math, unique_ptr, optional, variant
C++26: sorting, floating-point math, atomic, hashes, exceptions
C++29: format

Monadic Operations in C++23

Robert Schimkowitsch delivered a great, well-structured and perfectly paced talk that brought functional concepts to engineers who used to develop in an imperative style.

The outcomes of this talk are to understand what functors and monads actually do, how to use monadic operations from the standard library and how to build a functional-style program based on this foundation.

It is a useful talk for whoever is not so familiar with monadic concepts. Users of Haskell and more practically oriented languages with functional concepts, like Rust or Swift, could also benefit from this knowledge to find the subtle differences in behaviour.

Asking Stoopid questions

The keynote, as it should be. Frances Buontempo, a seasoned math teacher and educator, explained how to teach people, how to learn, and how to tackle the learning process on one’s own. A philosophical keynote, with food for thought and references to my favourite cyberpunk writers, Neal Stephenson and William Gibson.

int != safe && int != ℤ

An insightful talk from Peter Sommerlad about another safety facet of C++ — undefined behaviour with integer overflow. Regulated environments like MISRA require careful handling of overflows and the resulting undefined behaviour.

Built-in types are limited in size (they are not math integers), we can do different things for, say, addition:

Wrap them (-1 after 0xFFFFFFFFF)
Saturate (0xFFFFFFFFF + 1 stays 0xFFFFFFFFU)
Panic or throw

It’s an inherent property of machine types. Safety-aware languages like Rust have a variety of operations on the same integer type:

let mut x: u32 = 100;
assert!(x+1 == 101);
assert!(x.checked_add(1) == Some(101));
assert!(x.strict_add(1) == 101); // panics on overflow
assert!(x.overflowing_add(1) == (101, false));
assert!(x.saturating_add(1) == 101);

x = 0xFFFFFFFF;
// assert!(x+1 == 0); // panics in Debug mode, overflows in Release
assert!(x.checked_add(1) == None);
// assert!(x.strict_add(1) == 0); // panics on overflow
assert!(x.overflowing_add(1) == (0, true));
assert!(x.saturating_add(1) == 0xFFFFFFFF);

To be fair, it’s not MISRA-compatible, since it provides a set of specific requirements, such as not mixing signed and unsigned types, restricting type promotions and limiting operations for signed/unsigned types

The idea Peter proposed is to use special wrapper types that simulate the ℤ algebraic group (signed integers) with arbitrary precision and the mentioned operation semantics without overflows. His set of libraries are header-only C++20, providing the following types:

#include <pssodin.h>
constexpr pssodin::cui32 x{10}, y{20}, z{x+y};
static_assert(z == pssodin::from_int(30));

It also works with primitive types:

int32_t x{10}, y{20}, z{0};
assert(!pssodin::add_overflow(x, y, &z) && z == 30);

A distinctive feature of this library is compile-time checks for type eligibility for the majority of operations. Hence, the library is almost entirely constexpr, greatly simplifying testing for undefined behaviour. The API design is based on lean enum classes, which are more performant than a class with an integer data member.

What about having some of these features in a C++ standard? Not so good as with external libraries and other languages. With C++26, some saturation helpers are available in the std library, but not overflow:

static_assert(std::add_sat(10, 20) == 30);

C++20 also improved the situation a bit with tiny helpers std::cmp_equal to avoid common pitfalls with signed/unsigned comparison:

static_assert(-1 > 1U);
static_assert(std::cmp_less(-1, 1U));
static_assert(-1 == 0xFFFFFFFFU);
static_assert(std::cmp_not_equal(-1, 0xFFFFFFFFU));

So, for developers dealing with integer arithmetic, MISRA requirements, and older standards, Peter’s library is a godsend. Impressive work.

The Missing Step: Making Data Oriented Design One Million Times Faster

A keynote-style talk from Andrew Drakeford. He explained how a methodological and systematic approach allows for solving complex problems in financial applications. He outlined ideas from George Polya’s book “How to Solve it”] (goes to my reading list).

It’s much better to listen to this talk and experience a problem-solving journey with the speaker.

Still, I’d like to mention key takeaways (credits to the author):

Design addresses a highly dimensional problem.
- Logical algorithm design
- Physical optimising spatio-temporal memory use patterns
- Idiosyncratic aspects of the actual problem itself.
Working through the problem space in a structured way
Always look to expand the context of your thinking around problem areas
Draw pictures and solve easier ancillary problem

Unlocking the Value of C++20 Features

A day wrapped with a good overview by Alex Dathskovsky from ScyllaDB of the new C++20 and C++23 features, spanning two talks. It’s handy to have an overview and a reminder to evaluate some new C++ features, given the vast scope of the recent standard. There are a few things I’ve found especially useful and interesting.

Generic lambdas and simplified metaprogramming:

auto gen_lambda = []<typename T>(T a, T b, auto c) {
  return a + b + c;
};

Constexpr’s are everywhere, even in virtual hierarchy — what a blasphemy:

struct Animal {
  constexpr ~Animal() noexcept {};
  constexpr virtual std::string say() const = 0;
};

struct Cat : Animal {
    constexpr virtual std::string say() const override { return "Meow"; }
};

int main() {
  static constexpr const Cat cat;
  constexpr const Animal& animal = cat;
  std::cout << animal.say();
}

Transparent comparison for containers. It’s a tricky technique that unlocks more performance while working with standard containers:

std::set<std::string, std::less<>> foo;

Since C++20, it is allowed to specify a transparent hasher for hashing containers, skipping key copy:

struct transparent_hash {
{
  using is_transparent = void; // a key ingredient

  size_t operator()(const char* s) {
    return std::hash<std::string_view>(s);
  }
  size_t operator()(std::string_view s) {
    return std::hash<std::string_view>(s);
  }
};
std::unordered_set<std::string, transparent_hash, std::equal_to<>> foo;

// no intermediate std::string constructed
foo.find("a C string");
foo.find(std::string_view{"a string view"});

Speed for free - current state of auto-vectorizing compilers

Who doesn’t want speed for free? Stefan Fuhrmann agrees and explains how the automatic vectorisation works in modern compilers. Of course, developers can call compiler intrinsics or provide compiler hints to insert SIMD instructions here and there. It turns out tha modern clang with -O2 and gcc with -O3 optimisation levels can vectorise simple scalar loops. They are still unable to produce masked instructions to care about data which doesn’t fill the whole SIMD lane. So your typical loop over data should be split into two: a main loop for evenly sized chunks and a tail loop. Compiler engineers do a fantastic job of optimisation. Figuring out whether you can rely on them and bring it to the audience — it’s a great job too!

Why managing C++ dependencies is hard (and what to do about it)

Kerstin Keller presented the journey of one software company from a bunch of prebuilt libraries committed to VCS to a modern Conan 2.0-based build system, with a special emphasis on Windows. It’s always great to hear about experience with Conan across different areas and for different platforms.

Sanitize for your Sanity: Sanitizers tools for Modern C++

Last but not least — my humble talk.

The slides are here on my website. The recordings would eventually be available, too.

Update: YouTube recording is here:

According to the Swiss Cheese concept, it is mandatory to use multiple levels of defence. Type system, linters, undefined behaviour check via constexpr sprinkles — it is only a part of the whole deal. At runtime, you can do much more with sanitizers (address sanitizers, thread, memory and undefined). I explained the best ways to work with them, especially how to tailor them to a complex mix of libraries, instrumented and non-instrumented, which is an extremely common but usually overlooked scenario. Also, I tackled a problem of multi-language integration of Rust and C++ code, when using sanitizers to track cross-language problems. So it’s not a choice between Rust and C++, it is a question of making them work better together. As usual, when I am anxious, I speak very fast, which is definitely a problem when delivering complex topics to the public, so there is room for improvement. Nevertheless, slides are always available online, and I am happy to help and give advice on memory safety and sanitizers usage.

Conclusion

This year was turbulent for the world, the industry, developers, and conference organisers worldwide. Why even attend conferences and read books when LLMs could provide all the knowledge of mankind (and mistakes, too)? Despite this, Meeting C++ was filled with great talks, enthusiastic speakers, and curious attendees. The most appealing to me was a shared sense of C++ moving forward — especially towards greater safety. I also shared my knowledge in the talk, and I hope it will help make C++ a better and safer place.

It was terrific to meet developers from many industries, with expertise across different languages and domains. It is energising and makes you feel more confident about the community, the ecosystem, and the people behind it.

See you all next time!

uv for fast wheels

2025-08-11T00:00:00+00:00

To build or not to build?

You have a Python package. It solves a technical or business problem. You have decided to share it with the open-source community. Okay, just set up GitHub Actions and publish it to the PyPI. Periodically, you carve a new version and publish it. Everybody is happy – developers, consumers, and the community.

Then you suddenly discover that Python is slow. It is perfectly fine for many use cases, but yours is doing CPU-intensive work. You think it’s worth shifting CPU-intensive work from the interpreted language to a native extension written in C, C++ or Rust. There are a few different approaches to do this – Cython, CFFI, pure low-level CPython extensions. CFFI is the most widely used. You need to write native code in C, build it somehow into a library, load the library from Python and call it while adhering to call conventions and ownership semantics.

The problem is how to build and package a binary distribution. Different projects go in different directions. Here lies my experience of trying and using them.

Poetry

Poetry prefers having explicit build scripts, and the Poetry build backend is aware of them.

[build-system]
requires = ["poetry-core", "setuptools"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.build]
script = "scripts/build.py"

The build script is a simple Python module invoking setuptools.command.build_ext and cffi or cython. The downsides of this approach – it is still unofficial and unstable, and the behaviour changed with Poetry 2.

Setuptools

With old, good setuptools, it is easy to use setuptools’ official and supported integrations with ext_modules using cythonize or ffibuilder using the following setup.py:

setup(
    setup_requires=["cffi>=1.0.0"],
    cffi_modules=["scripts/build.py:ffibuilder"],
    install_requires=["cffi>=1.0.0"],
)

Worked like a charm for decades, but now it’s outdated. Even with dependency control via pip freeze you will run into many well-known Python packaging problems.

uv

What if you want to combine a modern packaging tool like uv and build binary distributions?

uv uses pyproject.toml for project configuration. The main benefit of the pyproject format is the standardisation of storing metadata for a project, build backend and external tools. It’s a big step forward from the imperative setup.py. It allows not only storing metadata, but also simplifying tools and significantly improving performance, because dependency resolution no longer requires evaluating Python code in setup.py.

Of course, all the dependencies – build or runtime – can be specified in the same file. uv has a great build backend uv_build (used by default) – extremely fast (because, you know, it’s in Rust :zap:) and reliable. It fully supports the pyproject.toml standard, including the standard project tag.

Binary distributions and uv

Unfortunately, this uv_buildbackend only supports pure Python. For binary extensions, the easiest way to overcome this limitation is to use a setuptools build backend instead.

[build-system]
requires = ["setuptools>=61", "cffi"]
build-backend = "setuptools.build_meta"

setuptools itself perfectly supports multiple formats for configuration – TOML.

[project]
dependencies = [ "numpy", "h3" ]

or setup.py:

setup(install_requires=[ "numpy", "h3" ])

But that’s not for all tools. All the integrated tools should decide for themselves where to get data from. For example, CFFI doesn’t support declarative pyproject.toml as a data source for cffi_module, which is sad because you have to store a setup.py file, albeit simple:

from setuptools import setup
setup(cffi_modules=["scripts/build.py:ffibuilder"])

Okay, now we’re back to setup.py. Poetry 1.x was able to generate setup.py with build instructions under the hood, so this approach is more or less familiar.

The most challenging part is to configure package discovery and file packaging properly. Here’s how we did it in the open-source timezonefinder package with @jannikm. It has a flat layout. Having a flat layout can bring unnecessary files into the package compared to the src layout, and that can be improved with the where = ["src"] directive. If you have a flat-layout, but still follow standard naming conventions, then automatic package discovery could work without any additional tweaks. It excludes typical directories like tests, examples, etc. To include new files, like built binaries in a module timezonefinder/inside_poly_extension, you have to include them in the packages directive explicitly:

[tool.setuptools]
packages = ["tests", "timezonefinder", "timezonefinder.inside_poly_extension"]

[tool.setuptools.package-data]
timezonefinder = ["**/*.json", "**/*.c", "**/*.h", ]

Since automatic package discovery is used, the build.py script cannot be stored at the root of the repository (it is in the exclusion list). Placing the script in the timezonefinder package eliminates this problem. Also, in this example, the tests are packaged too, because we need them in the distribution. The build script itself is simple and compiles a dynamic library from sources.

Essentially, the whole process of building the package is either

uv build --sdist

uv build --wheel

uv and CI

uv also simplifies CI setup for many projects.

If a project is pure Python, nothing special is required. Run tests, run a linter, make an sdist and upload it to PyPI. uv can streamline commands and simplify tool installation.

Binary packaging complicates the CI build. Now the project has multiple binary targets. Possible dimensions are cp39,cp310, cp311 for Python versions; musllinux or manylinux for ABI targets. There are two approaches to this.

First, the cibuildwheel tool. It manages the whole process automatically with a single GitHub Actions step by creating Docker containers for all permutations of specified Python versions and architectures. Inside each container, it runs a specified command (setuptools by default) and exports the produced wheels.

- name: "Build wheels"
    uses: pypa/cibuildwheel@v3.1.3
    with:
      output-dir: dist
    env:
      CIBW_MANYLINUX_X86_64_IMAGE: quay.io/pypa/manylinux2014_x86_64:2025.03.09-1
      CIBW_MUSLLINUX_X86_64_IMAGE: quay.io/pypa/musllinux_1_1_x86_64:2024.10.26-1
      CIBW_BUILD: "cp39-* cp310-* cp311-* cp312-* cp313-*"
      CIBW_BUILD_FRONTEND: pip
      CIBW_BEFORE_ALL_LINUX_MANYLINUX2014: yum install -y libffi-devel clang make
      CIBW_BEFORE_ALL_LINUX_MUSLLINUX_1_1: apk add --no-cache libffi-dev clang make

The second approach is to utilise a matrix build. uv can manage Python versions by itself by downloading prebuilt CPython binaries and executing scripts in their context. So the GitHub Actions build becomes even simpler:

strategy:
  matrix:
    python-version:
      - "cp310"
      - "cp311"
      - "cp312"
steps:
  - uses: actions/checkout@v4

  - name: Make wheel
    run: uv build --python $ --wheel

Of course, if you want to achieve manylinux or abi3 compatibility, you’ll need to use cibuildwheel in combination with uv.

You could also use the GitHub action astral-sh/setup-uv@v6, but the example cited above is the shortest version to demonstrate the flow.

Conclusion

uv provides an excellent experience when working with modern Python packaging. Lightning-fast dependency resolution and tool ergonomics are the main benefits.

Binary distribution is not a first-class citizen in uv and requires a fallback to setuptools build backend, while still using uv as a frontend and keeping its ergonomics.

The problem of standardised binary distribution building is not yet solved by any tool. Approaching this could be the next big step for the Python ecosystem, especially when Python is increasingly accelerated with native code.

How Jenkins Age Becomes Tech Debt

2025-07-15T00:00:00+00:00

Jenkins is a very popular and highly customisable CI system. It is so customisable that it could even become a problem, especially if you use plugins.

I am maintaining a really old and weird CI system. The master worked on 2.289, which dates back to 2021. What is even worse is that agent machines were running the operating systems openSUSE 12 and Debian 7 from 2011 and 2013. Those were glorious times — the Higgs boson was discovered, athletes ran at the London Olympics, and Snowden leaked tons of classified data. Anyway, I didn’t think it was such an ancient system – master was only five years ago. How naive was I…

The Plan

The usual upgrade plan for Jenkins is:

Upgrade plugins
Restart
Upgrade Jenkins WAR to the next version
Restart
Upgrade plugins again because Jenkins is newer now
Enjoy

And it usually works – but not this time.

Plugins disaster

Image by Jenkins Project CC BY-SA 3.0

Of course, I didn’t upgrade to the version past 2.479 on Java 17, because Debian 10 only supports OpenJDK 11 out of the box. So it was the easy part.

Then an innocent plugin upgrade led to unforeseen circumstances. It turns out the plugin centre doesn’t provide data for plugins older than one year. It is documented with an obscure wording: “Do not use versions no longer supported by the update centre”. I think it’s crucial information and providing factually incorrect information is just wrong. In practice, it means that you could be provided with an ill-defined set of versions without a proper baseline Jenkins version check.

That’s exactly what happened in my case. I discovered Jenkins in a state where almost all plugins complained about incompatibility with the Jenkins version. I started downgrading them one by one. It’s quite difficult since the Jenkins update centre doesn’t provide a way to downgrade a plugin version, not to mention downgrading to a specific version.

It means that for each plugin, you must navigate to the plugins website, perform version bisecting for a plugin version targeting Jenkins 2.479 at most, download the HPI from releases, upload it to the Jenkins Plugin Manager, and restart. Of course, the dependency graph for failed plugins must be constructed by yourself! The easiest way is to find the most used plugin, whose node has the highest out-degree in the graph, and try to upgrade it. Then move to dependencies. Hopefully, you won’t encounter plugin conflicts with each other (I did).

After a few hours, the plugin hell was finally frozen, and all plugins and Jenkins were in a consistent state with plugins.

In search of OpenJDK

Jenkins agents must run the same version of JDK as the master, or newer. This fact made life a little more complicated, since Debian 7 only supported Java 7, and openSUSE was stuck at 6. Previously, I put OpenJDK 8 in /opt/jdk8 and specified the path to the JDK binary for Jenkins agent in “Launch Method -> Advanced -> JavaPath” without exposing this Java to the system or applications via the environment. It worked well.

Unfortunately, JDK 11 builds are not widely available for these operating systems. Ten years ago, we were able to navigate to the ~~Sun Microsystems~~ Oracle website and download JDK builds for a variety of Unix systems. Sometimes, after an easy registration wall, but it was possible at least, although the options were limited.

Nowadays, Oracle doesn’t provide free JDK builds. The stewardship of Java went to OpenJDK, which is the official reference Java implementation. It produces source code, but not distributions. Surely, you can compile it on your own, but testing, tracking security issues, and ensuring correct platform support are up to you. So, plenty of JDK distributions had emerged (alphabetically):

AdoptOpenJDK
Azul Zulu
Amazon Corretto
Eclipse Temurin
Liberica OpenJDK
Microsoft OpenJDK
Oracle Java
Red Hat OpenJDK

To be precise, Oracle provides support only for the latest JDK version, so there is no such thing as Oracle LTS.

So the plan is to go and grab any distribution for OpenJDK 11, right? That’s true only if it supports an old enough libc. If somebody builds a distribution on a newer build machine, binaries will use the latest system glibc; hence, it cannot be run on an older machine. So you should either have a dedicated old build machine, which sets a baseline glibc version, or bump glibc requirements.

I found a few distributions which were built a long time ago and are still available at a vendor’s website. For example, for Liberica jdk11.0.27+9:

% nm /opt/jdk11/bin/java | grep '@GLIBC'
  U getenv@@GLIBC_2.2.5
% /lib/x86_64-linux-gnu/libc.so.6
  GNU C Library (Debian EGLIBC 2.13-38+deb7u11) stable release version 2.13, by Roland McGrath et al.

It works because the system glibc version is newer than any symbol referenced from the Java ELF binary. Of course, it is even easier just to launch it and see whether it immediately emits any errors.

Sometimes vendors are kind enough to list system requirements for versions in terms of operating system versions (like Debian 9, RedHat 7 and so on). Only a few of them specify the kernel and glibc version, and in this case one can consult distrowatch or check the glibc version on a real system. Finding a matching and working version was a trial-and-error process.

Things became even more complicated for non-Intel build farms. If you’re thinking about a modern AArch64 ARM – no way, it is a 32-bit ARM7 hard-float. So you have an even smaller chance of finding a binary distribution for this old obscure architecture. The only distribution providing 32-bit ARM7 HF is Azul Zulu. However, all of their recent binaries are for glibc 2.15. And there is only one old build 11.1.8 targeting older glibc 2.13, which is able to run on that machine:

The fun doesn’t stop with Linux. Turns out, an old but stable OS X 10.7 machine cannot run any builds from OpenJDK, Azul, or Corretto. It is not only a matter of using a new operating system API, but also of using a newer kernel. All these builds crashed with SIGSEGV upon launch. Suddenly, I discovered a custom OpenJDK build from the open-source enthusiast Jazzzny, who had kindly built JDK 11 for OS X. The binaries are available at GitHub releases page, and binary jdk-11-snowleopard-r1.pkg (sha256 7c51f13993a7f38575d82aa2a7691aace6170c59074847e779a3d9f903f044fb) built for Snow Leopard (10.7) worked perfectly for me. Minecraft gamers used these builds to deal with older Java versions, which proves again that gaming is the driving force of the computer industry.

Outcome

Ten hours into the upgrade, I’d successfully fixed core Jenkins, all the plugins, and agents, and I was able to run a few test builds.

What went well? The diversity of OpenJDK distributions and the openness of the community greatly enhance your chances of finding a proper distribution suited to your needs.

What could go better? The Jenkins plugin management system. The older your system is, the less likely you are able to upgrade. The cut-off of the plugin compatibility information after a year or two, lack of built-in downgrade functionality and clear machine-readable requirement annotation drags you into the manual dependency resolution nightmare. Making a copy of the Jenkins installation for upgrading and switching it later could be a good solution.

What had I learned? The age of your CI infrastructure is the tech debt. Contrary to popular opinion, please do upgrade your Jenkins at the LTS track and the plugins, at least yearly. It enables the possibility of a much smoother upgrade.

Talk at Rust Nation UK 2025

2025-02-28T00:00:00+00:00

Last week, I had the chance to give a talk at Rust Nation UK 2025, one of the biggest Rust conferences.

The YouTube talk is already up, thanks to the conference team. If you prefer slides, here they are.

As always, I love talking about performance, optimisation, and Rust. So my topic was both useful and engaging (with a pun) - “Parallel Programming in Rust: Techniques for Blazing Speed”.

We covered a lot: high-level parallelism with Rayon and thread pools, the parallelism theory, concurrency primitives, low-level primitives (atomics, lock-free algorithms, SIMD, cache efficiency, profiling) and even GPUs - diving deeper and deeper to the hardware. The talk was probably a bit dense, but I hope it provides a solid overview of modern parallel computing approaches and gives ideas on where to go next.

I was impressed by the conference content. The keynotes from Mark Russinovich and Celina Val were inspiring and reassuring about integrating Rust into existing codebases. There were in-depth technical talks about libraries, language and kernel bindings, the evolution of both the ecosystem and the language. The workshops were also useful and practical about async and C++ interop.

And, of course, one of the best part of the event was the vibrant, friendly, and passionate community, always ready for a great discussion. Looking forward to attending the next one!

Scrobbling to Last.fm offline

2024-08-27T00:00:00+00:00

As a passionate stereo and vinyl enthusiast, I keep track of all my favourite tracks using Lastfm. But how about scrobbling the tracks spinning on my turntable? Here is how I made offline scrobbling happen using blazingly fast 🦀™ offline scrobbler!

Image by M. Johnson under CC BY 2.0

Imagine: you are sitting in a comfy chair near your stereo system, smelling the scent of a vinyl record spinning on your turntable, looking at the artwork, and listening to music. Is there something you might have forgotten to do?

It is not a problem at all since Last.fm allows you to scrobble using its API. That’s how your scrobblers for desktops and mobile devices work. So, I wrote a utility called offline-scrobbler to scrobble music to Last.fm for a given artist and album.

How to set it up

Install the utility. The simplest way to do it, if you have Cargo installed, is to run:

cargo install offline-scrobbler

GitHub releases contain Linux binaries. They are automatically built for Linux using musl to avoid problems with glibc linkage. Building macOS and Windows executables is a little more challenging. If anyone wants them, please ping me.

You need to create a Last.fm application account. Since this scrobbler is not a public service and it just runs on your machine, you need to create an account for yourself. Head to Last.fm, and log in as usual. Then go to Create API account page. Here, you need to enter your “Contact e-mail”, an arbitrary name to “Application name” field, and nothing more.

On the next screen, you will be presented with unique account keys for this application. Write down the “API Key” and “Shared secret”.

Provide the keys to the utility with the following call:

offline-scrobbler auth --api-key API_KEY --secret-key SHARED_SECRET

You are all set!

Scrobbling music

That’s why we are here, right?

I supported three typical scrobbling scenarios.

First of all, scrobble one song. It’s pretty easy:

offline-scrobbler scrobble --artist=Hooverphonic --track=Eden

Then, you can scrobble the whole album:

offline-scrobbler scrobble --artist=Hooverphonic --album="A New Stereophonic Sound Spectacular"

Finally, if you have a Last.fm album page open, just grab the URL and feed it to the scrobbler:

offline-scrobbler scrobble-url --url "https://www.last.fm/music/Hooverphonic/Blue+Wonder+Power+Milk"

Internals

Did I mention that the scrobbler is blazingly fast 🚀 because it is written in Rust 🦀 ? :)

The scrobbler relies on Last.fm artist and album names. So, it’s advisable to first run scrobbler in preview mode (--dryrun argument) to verify you’re scrobbling precisely the tracks you want. If there are any scrobble problems or tracks are ignored, they will be reported to the console.

I was surprised that Rust is missing a good scrobbler library. Anyway, it is a common problem. I wrote mine in Objective C while developing a desktop scrobbler for OS X a decade ago. The reqwest and serde simplify writing API clients a lot. Unfortunately, last.fm / audioscrobbler API is XML-based, so you have to write some boilerplate.

The core types for scrobbling logic are pretty simple:

 #[derive(Debug)]
 pub struct Track {
     pub title: String,
     pub duration: i64,
 }

 #[derive(Debug)]
 pub struct Album {
     pub title: String,
     pub tracks: Vec<Track>,
     pub url: Option<String>,
 }

All the error handling logic is nicely supported by anyhow (for propagating errors in CLI application) and thiserror (to model library error enum).

Writing CLI applications in Rust is a breeze. Combining clap, serde and env_logger is one of the finest ways to make a complicated CLI with parameter groups, validation, colour handling and other nice things. The only unfortunate thing I’ve noticed is that authors of clap v4 recently decided to unconditionally drop colour support in favour of more generic ANSI underlining. Let’s check the state of this issue.

Date and time

One curious thing I’ve discovered is that std library doesn’t provide formatters for std::time::Duration. It has Debug trait implementation, but the formatting is excluded explicitly because: “there are a variety of ways to format periods for human readability”. So, one should use an implementation from chrono or time. The time crate has a totally different Duration type apart from std, which is not so convenient. Finally, there are ongoing debates which external datetime library to use. Maintainers of chrono returned to support the crate, and one more library called jiff just arrived from BurntSushi, so the choice became even harder. The main downside of a well-supported and lightweight time crate for me is the lack of strftime-style format. It is so ubiquitous to use %H-%M-%S instead of

time.format(format_description!("[hour]:[minute]:[second]"))

To mimic normal human behaviour, I added pauses between tracks in the duration of each track. What’s more, you can scrobble into the past. Remember, you had been playing that album 6 hours ago? That is no problem - just specify an offset in human-readable form.

offline-scrobbler scrobble --artist=Hooverphonic --album="A New Stereophonic Sound Spectacular" --start=6h

Kudos to the great humantime library for this functionality.

Conclusion

My life has many entertaining activities, and music-related ones make up a significant part. I’ve worked on different tools to ease managing music libraries, encoding files and synchronizing with services. While building such tools, always remember that it could also be fun and rewarding.

Testing PostgreSQL Extensions

2023-05-20T00:00:00+00:00

You have written a useful PostgreSQL extension. It works perfectly, includes some tests and has excellent documentation. But can it be tested properly and automatically for multiple PostgreSQL versions?

Single-machine build

First, the best way to build an extension on a single machine is to use the standard PGXS build system. It allows writing a simple declarative Makefile that includes a lot of PGXS machinery:

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

Only a few Makefile variables with extension metadata must be added, and PGXS does the rest. Just type make, and the platform-specific binary and SQL scripts for extension are built. Target make install install the extension to the current (pointed by pg_config) PostgreSQL installation.

PGXS also provides regression tests with a tool pg_regress and makefile wrappers. It’s enough to put SQL inputs to test/sql/test1.sql and expected text output to test/expected/test1.out. Invoking make installcheck runs all regression tests and provides a report and a status code.

It’s an easy and powerful system.

Testing at scale

But what about testing against multiple PostgreSQL versions? Typical package managers like homebrew and apt don’t allow the installation of a large selection of PostgreSQL binaries because older versions are quickly replaced with newer ones. For a specific Debian installation, only one PostgreSQL version is supported at any moment, which the Debian community chooses. Homebrew allows to have multiple servers installed side-by-side. To select a specific version, pass a PG_CONFIG variable before calling make:

PG_CONFIG=/usr/local/opt/postgresql@14/bin/pg_config make

Another way is to build multiple PostgreSQL versions from the source and install them side-by-side. Usually, it works well because the build system works fine on a range of operating systems versions.

But we don’t want to build everything from scratch. Okay, let’s use official Docker images:

docker run --rm -it postgres:14-bullseye find / -name postgres.h

Oops, no headers. The image is not suitable for building extensions.

How can we avoid building code by ourselves?

It turned out that there are official APT repositories provided not by Debian but by the Postgres community itself. These APT and APT FAQ describe them in details. You can install postgresql-14 and postgresql-server-dev-14 (any version here) on any Debian distribution. Packages provide all needed headers and development files.

Test harness for images

Unfortunately, there aren’t any prebuild Docker images with these packages. With a prebuilt server binary, it’s a much simpler task.

All CI-related files for my parray_gin extension lay inside this directory.

First of all, the image contains everything needed to run actual tests.

Here we specify the build argument with a version.

FROM debian:bullseye

# PostgreSQL version (like 15)
ARG PGVERSION
ARG PGCHANNEL=bullseye-pgdg

We use the same compiler (gcc10 in bullseye) for all different PostgreSQL versions:

ENV DEBIAN_FRONTEND=noninteractive

# Install core packages
RUN apt-get update && \
    apt-get install --no-install-recommends -y sudo curl gnupg tzdata \
    locales lsb-release ca-certificates \
    make gcc libssl-dev libkrb5-dev libicu-dev libdpkg-perl

The APT repository for PostgreSQL must be configured in a modern way. Do not use apt-key anymore because it is deprecated. Remember to raise the priority of new packages because Debian packages could be pulled in instead of the community’s.

# Configure repository
RUN mkdir -p /etc/apt/keyrings && \
    curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
    gpg --dearmor -o /etc/apt/keyrings/postgres.gpg && \
    echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/postgres.gpg] http://apt.postgresql.org/pub/repos/apt/ ${PGCHANNEL} main ${PGVERSION}" | \
    tee -a /etc/apt/sources.list.d/postgres.list

# Prefer packages from the Postgres repository
RUN echo "Package: *\nPin: release o=apt.postgresql.org\nPin-Priority: 900" | \
    tee -a /etc/apt/preferences.d/pgdg.pref

# Install PostgreSQL
RUN apt-get update && \
    apt-get install --no-install-recommends -y postgresql-${PGVERSION?} \
      postgresql-server-dev-${PGVERSION?} && \
    rm -rf /var/lib/apt/lists/*

Finally, bootstrap the cluster and prepare the environment for building extension:

# Create cluster
ENV PGBIN=/usr/lib/postgresql/${PGVERSION}/bin
ENV PGDATA="/var/lib/postgresql/${PGVERSION}/test"
ENV PATH="${PATH}:${PGBIN}"

RUN echo "postgres ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

COPY docker/entrypoint.sh /entrypoint.sh
COPY docker/test.sh /test.sh
RUN chmod a+x /entrypoint.sh /test.sh

RUN mkdir /src && chown postgres /src

USER postgres

RUN initdb

WORKDIR /src

ENTRYPOINT [ "/entrypoint.sh" ]

The entrypoint launches the cluster with pg_ctl start in the background so the command in a container will be executed with an environment with an already running cluster:

#!/bin/sh
set -e
pg_ctl start
exec "$@"

Images could be prebuilt in advance:

docker build -t pgx:14 -f docker/Dockerfile --build-arg PGVERSION=14 .

Container builds and runs tests for an extension whose source code is in a given directory. The directory is mounted as a read-only volume. Everything from this directory is copied to the intermediate directory and built there.

docker run --rm -v `pwd`:/workspace:ro pgx:14 /test.sh

There is one tricky part. Sometimes at slow CI machines cluster has yet to start when the test is begun. We need to wait until Postgres becomes ready. For newer versions pg_isready command can be used, but for older versions (9.x), the fallback with a blind timeout is used.

if command pg_isready ; then
  timeout 600 bash -c 'until pg_isready; do sleep 10; done'
else
  sleep 30
fi

So the typical local workflow for testing against arbitrary versions is a one-liner:

export PGVERSION=14 && \
  docker build -t pgx:$PGVERSION -f docker/Dockerfile --build-arg PGVERSION=$PGVERSION . && \
  docker run --rm -v `pwd`:/workspace:ro pgx:$PGVERSION /test.sh

Testing on nightly

If you are going to test an extension against unreleased PostgreSQL, there are packages with development nightly snapshots. You don’t need to build a database from the trunk!

My scripts allow specifying a development channel PGCHANNEL=bullseye-pgdg-snapshot instead of standard PGCHANNEL=bullseye-pgdg for the image so the nightly package version will be used for tests.

Wrapping up in GitHub Actions

Since all tests are parametrised with a database version and channel, it’s easy to wrap them in any CI that supports matrix builds.

Here we go with GitHub Actions matrix build:

name: test

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        pg-version: [ "9.1", "9.5", "9.6", "10", "11", "12", "13", "14", "15" ]

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Build the Docker image
        run: docker build . --file docker/Dockerfile --tag "pgxtest:$" --build-arg "PGVERSION=$"

      - name: Test
        run: docker run --rm -v `pwd`:/workspace "pgxtest:$" /test.sh

Snapshot workflow is more straightforward and includes the only job for the snapshot.

Summary

The build and testing system for PostgreSQL extensions is stable and useful for local development. More efforts are needed when you need to test it against multiple PostgreSQL versions automatically. Code and approaches introduced in this post could greatly simplify the development experience with local development and public CIs.

Why the Riak matters

2023-04-23T00:00:00+00:00

Riak is one of the first and most prominent key-value databases implementing the famous Amazon Dynamo paper. The paper was unveiled in 2007. Sixteen years ago, it could even get a passport!

The more widespread database that follows the paper is Apache Cassandra. Funny enough that AWS DynamoDB had walked away from original ideas (leaderless replication), and all of these systems have different storage engines.

Riak differs by following the original ideas from Dynamo. So if you’d like to explore those principles and design a system for database, or maybe even enhance a database, Riak is a very good candidate, with a tiny peculiarity. It is written in Erlang and OTP which raises a barrier to entry.

In this post, I’ll describe the architecture of Riak, map-reduce pipeline development in Erlang and the Bitcask storage engine.

Dynamo foundations

Attempts to speak with the Riak Ring in Erlang as shown in the Arrival movie:

Maybe Riak’s glory days are passed. After Basho’s departure and giving away projects to the community, it lacks support. For example, you cannot download binary releases for macOS since the bucket with artifacts has some permission errors. So it’s required to build it from the source. And it’s impossible to build Riak with the latest Erlang 25, I only managed to do it with Erlang 22. A lot of dead links in the documentation. So you were warned.

Riak has a nice CLI to manage a cluster. Each node is accessible by HTTP and protobuf protocols. You can even fire an Erlang shell and work with a node directly because it is an OTP application. It really shines with distributed deployments since it is fully decentralised and uses leaderless replication. Membership and state are replicated across the cluster with gossip protocols. Riak is leaderless, therefore a client can connect to any node and perform operations. Depending on request types, a node can interact with other nodes to execute the request. Regarding CAP, Riak is called an AP system, but I’ll talk more about consistency guarantees later.

Data is partitioned between nodes with consistent hashing at the ring. The ring of ordered hashes represents the partitioning of keyspace to multiple continuous partitions identified by start and end keys. Each partition on the ring is handled by a vnode. Each Riak node is mapped to multiple vnodes. To fetch or store an object in the cluster, Riak first hashes the key, finds a small sequence (depending on availability requirements) of hashes on the ring, finds the vnodes, maps them onto physical nodes, perform reads/writes and waits for a read/write quorum until returning an answer to the client. The idea of consistent hashing lies in a faster rehashing when a node joins and leaves a cluster, becaise it affects only a small part of the ring instead of the whole ring. If a node leaves the cluster temporarily (i.e., network problems), rehashing is still unnecessary because this situation can be handled by hinted handoff. If a failure is permament, differences in keys between nodes can be found using anti-entropy procedures.

So many concepts from Dynamo are directly implemented in Riak without major changes. There is a good page in documentation about how closely these concepts are mapped to Riak’s.

Client interfaces

Riak is widely known for its good interface. It had been thoroughly following the REST principles. According to Richardson Maturity Model, which describes the adoption of REST concepts at Level 0 (HTTP transport), Level 1 (resources), Level 2 (HTTP verbs), Riak stays at the highest Level 3 (Hypermedia Control). It provides walking across linked keys using standard Link headers. Multiple values are served with the standard HTTP multipart/mixed responses. Different types of media are handled with the MIME types.

I’ve seen such powerful HTTP API in Hashicorp products too, for example, in Consul. The ability to have a full-featured HTTP API to perform KV operations, resolve conflicts and even perform a leader election without needing to write a lot of Java clients (I look at you, ZooKeeper) is fascinating.

Another available protocol is Protobuf. Curiously, it is not used over gRPC transport but instead with a very simple TCP transport.

Of course, many client libraries and drivers are written for Riak to interact with the store. For example, here is some Ruby code to create and put a value:

client = Riak::Client.new(:pb_port => 10017)
bucket = client.bucket("test")

obj = bucket.new('mykey')
obj.data = {'answer': 42}
obj.store()

val = bucket.get('mykey')
fail unless val.data['answer'] == 42

And an excerpt of quering the key by HTTP:

http http://localhost:10018/buckets/test/keys/mykey

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Link: </buckets/test>; rel="up"
X-Riak-Vclock: a85hYGBgzGDKBVI8R4M2cgds+X+VgUFdMIMpkTGPlSFaZ8tdviwA
{
    "answer": 42
}

Libraries lack maintenance too and show their age. For example, it’s not possible to use Python newer than 3.7.

Tunable consistency

It’s well known that Dynamo databases are eventually-consistent. Usually, they are put to “AP” class, but Kleppman disagrees and instists on “P”. Let’s check why.

To handle conflicts, Riak provides a variety of options. I will describe four of them.

The simplest conflict resolution strategy is “last write wins” (Cassandra uses it too) based on the latest timestamp. Client does not need to address conflicts.

The recommended option is to use multi-version values. For this option, Riak uses dotted version vectors. This structure is an improvement of vector clocks which can handle many clients with conflicting writes. Two conflicting writes create siblings – different values identified by their vector clock values. So you need to handle siblings at the client application.

Let’s create a bucket type with version vectors.

riak-admin bucket-type create vv '{"props":{"allow_mult":true}}'

Create conflict:

http PUT http://localhost:10018/types/vv/buckets/cats/keys/favourite name=Puffy
http PUT http://localhost:10018/types/vv/buckets/cats/keys/favourite name=Pirate

When we try to get value, Riak returns both siblings

http http://localhost:10018/types/vv/buckets/cats/keys/favourite
HTTP/1.1 300 Multiple Choices
...
Siblings:
4IgpOUc3rR7wPOKbgJTGfJ
5UX8mBVbx3Y9Vno6ncicuO

So to resolve the conflict, the next update must include a context (X-Riak-Vclock value from PUT responses) and therefore resolve a conflict. If no context is given, Riak cannot reconstruct lineage.

http http://localhost:10018/types/vv/buckets/cats/keys/favourite \
    name=Pirate X-Riak-Vclock:"hexademical-clock-value-from-response"

Libraries encapsulate sibling handling with a fetch-modify-write loop. For example, the equivalent code in Ruby:

bucket = client.bucket('cats')
# read from Riak
obj = bucket.get_or_new('favourite', type: 'vv')
# deal with siblings
fail if obj.siblings
# update object locally
obj.content_type = 'application/json'
obj.data = {"name"=>"Pirate"}
# write to Riak
obj.store

So to write a key, you always need to read a key before and therefore create a causal context which will be included in write operation.

The third option is a strong consistency mode. Strong consistency is specified at a per-bucket level. There Riak guarantees that no siblings will be returned during read operations. Since the causal context is used for write operations, no siblings will be produced too. Client code is free from handling conflicts (a line fail if obj.siblings can be dropped).

The fourth option appeared in 2014 in version 2.0 with the introduction of CRDT types (Conflict-free replicated data types) like flags, registers, counters, sets, and maps. It is a rare and complex feature for databases. CRDT types allow the developer to ignore all eventual consistency quirks, vector clocks, and handling sibling versions because the conflict resolution is fused into data types, not operations. Of course, data modelling using CRDTs is more challenging than just dropping blobs into the KV store. Full-fledged sequence CRDTs were out of scope too.

With this broad choice of consistency modes and options, it’s hard to call Riak a “simply AP” system.

MapReduce

As with every good SQL and NoSQL database, Riak provides server-side computational primitives.

In past years, we were able to run MapReduce queries with JavaScript functions which are registered to the Riak via HTTP API:

echo 'function(v) {
var parsed_data = JSON.parse(v.values[0].data); var data = {};
data[parsed_data.style] = parsed_data.capacity; return [data];
}' | http PUT http://localhost:10018/riak/my_functions/map_capacity

And then call the MapReduce with some inputs:

echo '{
  "inputs":[["rooms","101"],["rooms","102"],["rooms","103"] ],
  "query":[
    {"map":{
      "language":"javascript",
      "bucket":"my_functions",
      "key":"map_capacity"
		}}
	]
}' | http POST http://localhost:10018/mapred

The function can be inlined in this mapred call. Unfortunately, the JavaScript interface is deprecated and doesn’t work reliably anymore.

You should write your functions in Erlang, and it could be a little tricky. The official documentation on MapReduce is very good but still lacks a complete example. The general approach is similar to Hadoop and HBase ecosystem techniques. You build JARs, push them to the cluster, register with runtime and then execute. Let me show how to do it.

Colouring all the cats

Let’s analyse cat’s colours by their names:

White Stripes
Black Ash
Grey Hound
Small White Dot
Orange Pirate
Snow White

As we know from the internet, the number of cats is enormous. We should do it in a massively parallel fashion!

First, fire up the cluster. Official documentation described the process well enough. Let nodes live locally at $CLUSTERDIR/{dev1,dev2,dev3}. And the Riak libraries needed to build our extension are located at $SDKDIR.

To check cluster, run riak-admin member-status for the cluster membership and riak-admin ring-status for the state of the ring.

riak-admin member-status

================================= Membership ========
Status     Ring    Pending    Node
-----------------------------------------------------
valid      34.4%      --      dev1@127.0.0.1
valid      32.8%      --      dev2@127.0.0.1
valid      32.8%      --      dev3@127.0.0.1
-----------------------------------------------------
Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
ok

The interesting thing is that each VNode is implemented as an Erlang process, specifically a finite state machine with gen_fsm behaviour. For example, my three-node cluster has 64 partitions and 64 vnodes. Each physical node will get 21 or 22 vnodes (approximately 33% of all partitions). More than that, a lot of operations for node (storage maintenance, handoff) are also implemented as processes. Since they are really lightweight, running such an amount of processes is not a problem.

% riak-admin ring-status

================================== Claimant ========
Claimant:  'dev1@127.0.0.1'
Status:     up
Ring Ready: true

============================== Ownership Handoff ===
No pending changes.

============================== Unreachable Nodes ===
All nodes are up and reachable

ok

Write some code in colorer.erl for loading cats to the database (function load/1), for map phase (colormap/3) and for reduce phase (coloreduce/2).

-module(colorer).

-export([load/1]).
-export([colormap/3]).
-export([coloreduce/2]).
-export([get_color/1]).

%% Determine a cat color from its name
get_color(Name) ->
    NameTokens = string:tokens(Name, " "),
    case lists:member("White", NameTokens) of
        true ->
            "White";
        false ->
            case lists:member("Black", NameTokens) of
                true ->
                    "Black";
                false ->
                    "Mixed"
            end
    end.

%% map phase
colormap(Value, _Keydata, _Arg) ->
    ObjKey = riak_object:key(Value),
    KeyStr = binary_to_list(ObjKey),
    Color = get_color(KeyStr),
    error_logger:info_msg("colorer mapped key ~p to ~p", [KeyStr, Color]),
    [{Color, 1}].

%% map reduce
coloreduce(List, _Arg) ->
    error_logger:info_msg("colorer reduce list ~p", [List]),
    ListOfDicts = [dict:from_list([I]) || I <- List],
    Merged =
        lists:foldl(fun(Item, Acc) -> dict:merge(fun(_, X, Y) -> X + Y end, Item, Acc) end,
                    dict:new(),
                    ListOfDicts),
    % convert dict to list
    dict:to_list(Merged).

%% End of list
insert(_Client, []) ->
    ok;
%% Put a cat from head of list to the Riak
insert(Client, [Head | Tail]) ->
    % All cats are adorable, so it goes to the Riak value as a JSON
    Body = list_to_binary(mochijson2:encode(#{<<"adorable">> => true})),
    % Riak key is a cat name
    Key = Head,
    RawObj = riakc_obj:new(<<"cats">>, Key, Body),
    Obj = riakc_obj:update_content_type(RawObj, "application/json"),
    case riakc_pb_socket:put(Client, Obj) of
        ok ->
            io:format("Inserted key ~p~n", [Head]);
        {error, Error} ->
            io:format("Error inserting key ~p", [Error])
    end,

    insert(Client, Tail).

%% Load cats from text file
load(Filename) ->
    {ok, Client} = riakc_pb_socket:start_link("127.0.0.1", 10017),
    {ok, Data} = file:read_file(Filename),
    Lines = re:split(Data, "\r?\n", [{return, binary}, trim]),
    insert(Client, Lines).

Put cats in a bucket

First things first, load all our cats into Riak.

Compile the file with erlc colorer.erl. It produces a BEAM file colorer.beam in the current directory. Then run the loader:

erl -pa $SDKDIR/lib/riakc/ebin \
  $SDKDIR/lib/riak_pb/ebin $SDKDIR/lib/mochiweb/ebin \
  -noshell -eval 'colorer:load("cats.txt"), halt().'

Since we connect to the Riak cluster from the load/1 function by Protobuf protocol, we can start the standalone virtual machine, not the one which Riak uses.

Now check the bucket cats:

http http://localhost:10018/buckets/cats/keys keys==true

HTTP/1.1 200 OK
Content-Type: application/json
Server: MochiWeb/3.0.0 WebMachine/1.11.1 (greased slide to failure)
Vary: Accept-Encoding

{
    "keys": [
        "Snow White",
        "White Stripes",
        "Grey Hound",
        "Small White Dot",
        "Black Ash",
        "Orange Pirate"
    ]
}

Check a random cat:

http http://localhost:10018/buckets/cats/keys/Snow%20White

HTTP/1.1 200 OK
Link: </buckets/cats>; rel="up"
Server: MochiWeb/3.0.0 WebMachine/1.11.1 (greased slide to failure)
Vary: Accept-Encoding
X-Riak-Vclock: a85hYGBgzGDKBVI8R4M2ckekvudgYIyfmMGUyJnHyjDl29a7fFkA

{
    "adorable": true
}

Great, all our cats are in a bucket.

Running MapReduce job

All Riak nodes must be able to load the code colorer.beam. Add the path to this directory to riak/etc/advanced.config for each node and restart the cluster:

[
 {riak_kv,
 [
   {add_paths, ["YOURDIR"]}
 ]},

 {riak_core,
 ...
 }.

Each time you recompile the file, the cluster must reload the BEAM file. Since Erlang uses hot reloading, it could be achieved without restarting the cluster!

erlc colorer.erl && for N in 1 2 3 ; do ${CLUSTERDIR}/dev$N/riak/bin/riak-admin erl-reload ; done

Now we can ask Riak to launch a job like it was in the JavaScript example.

echo '{"inputs":"cats", "query":[
  {"map":{"language":"erlang","module":"colorer","function":"colormap"}},
  {"reduce":{"language":"erlang","module":"colorer","function":"coloreduce"}}]}' | http POST localhost:10018/mapred

And we got the answer

{
    "Black": 1,
    "Mixed": 2,
    "White": 3
}

So we just run a distributed colorer job on our cluster!

Let’s explore what is going on inside. Actually, you can test map and reduce functions from REPL with manually specified inputs. But for actual remote invocations let’s log inputs

error_logger:info_msg("colorer mapped key ~p to ~p", [KeyStr, Color]).

Output can be seen in $CLUSTER_DIR/dev*/riak/log/console.log: colorer mapped key "Black Ash" to "Black". So the mapper produces a list of tuples with the cat’s colour. We produce a string with a colour name (actually it is an Erlang binary) for each cat and initial count as one. Value is not analysed for simplicity but can be easily extracted with riak_object:get_value(Value) and converted to dict using mochijson2.

Then all the outputs of the map phase are passed to reduce phase (remember, we specified the function and module name in HTTP invocation). The coloreduce/2 gets the list of all tuples: colorer reduce list [{"Black",1},{"Mixed",1},{"White",1},{"White",1},{"White",1},{"Mixed",1}]. Then it converts them to dicts and folds in a resulting dict:

[{"Black",1},{"Mixed",2},{"White",3}]

So the function returns a list of dicts. Riak produces the MapReduce output as JSON. Outputs and inputs between stages are still binary-serialized Erlang objects.

It isn’t difficult at all.

Bitcask

Riak has pluggable backend architecture. The default storage engine is Bitcask. Another widely-used backend is a customised version of LevelDB which is an LSM-based (log-structured merge tree) key-value store. The main reason to prefer LevelDB to Bitcask is having a lot of keys that could not fit into the node’s memory because Bitcask requires the entire keyspace to be in memory.

A lot of literature is dedicated to LSM engines. I would like to describe Bitcask architecture a bit (pun intended).

This famous storage engine is described in a really short paper that is worth reading. The Bitcask is a key-value store. The original design covers on-disk format, data flow and access protocols and the Erlang application. Some parts could remind the LSM but it is a much simpler engine.

A database node handles a directory of data files exclusively. The directory has multiple datafiles, and the most recent one is called an active file. A node can only append to the active file. All other files are read-only. So all the writes are sequential and therefore blazing fast. When the file has grown over the limit, it is closed and another one is created as an active file.

The structure of key/value entry is simple. It is a variable-sized record with sizes of key and value specified in a header. The whole record is checksummed.

0           4           8            16           24
+-----------+-----------+------------+------------+--------+--------+
|   CRC     | TIMESTAMP | KEY_SIZE   | VALUE_SIZE |  KEY   | VALUE  |
+-----------+-----------+------------+------------+--------+--------+

Bitcask requires maintaining an in-memory structure called “keydir”. It stores a mapping from each key to a location information: file path, offset in a file to the corresponding entry, and the most recent timestamp.

Reads are simple. Node checks keydir in a constant time. Then it opens a file (if not opened yet) where the record is stored and seeks to the specified offset. Since size information is stored in an entry header, the node reads value_size bytes containing a value. The total cost of a read operation is a single random disk seek.

To perform a write, node just appends a new entry to the active file and updates information in keydir. If the key is already present, its previous apperance in data files became orphaned (no links to them). The cost of a write operation is a sequential write to disk without any seeks.

To prune orphaned records, Bitcask uses merging. It is much simpler than LSM merges. Bitcask knows all the keys because they are stored in keydir. It needs to iterate over all directory files and get only the latest entry for each key. Then a new file is written with only the latest entries for each key. Old files can be removed. Merge is completed.

The major advantage and both disadvantage of Bitcask is the keydir. It must store all the keys. If the keyspace is larger than available memory, it makes no sense to store keys in a hybrid manner (in memory and disk), and an LSM-based engine should be preferred. Also, when a node is started, it must populate a keydir with all keys from all files (not only an active file). It can be optimised by using hint files.

The interesting thing in Bitcask is the lack of a commit log or WAL. Data files are the log itself. Concurrency is not described in the paper and is addressed only in the implementation.

My implementation

Speaking of implementation, I stumbled upon a really fun project CaskDB. It is a skeleton for designing your own Bitcask store in Python. A lot of tests are already written.

I have written an extremely simple storage engine. Concurrency and network protocol are out of scope. Surprisingly, it was a really fun project to code Bitcask in modern Python for a few evenings. If by any chance you are interested, it is on my GitHub. I made a few additions compared to the original task, primarily in tests. I have used property-based testing with Hypothesis which helped to pinpoint hard-to-find bugs. I think it is an essential technique for data engines. Other useful techniques that should be used are fuzzing and secure coding with invariants. Of course, a whole class of problems can be avoided by using more safe languages with static typing like Rust, but it will be another project. Stay tuned!

Outcome

Implementing Bitcask storage engine helps to understand why the engine works, where it shines and what the downsides are. Definitely worth your Netflix weekend:)

The Riak is a little outdated, and 15 years seems like an eternity in the database world. But it is a pure elixir (pun intended) of design approaches for leaderless replication, tunable consistency, causal contexts, CRDTs, effective storage engines and another important concept. Also, it is a beautiful example of a well-architected OTP application. Long live Riak!

Learning by outdated books

2023-02-22T00:00:00+00:00

Learning new things is hard, and finding the right way to do that is even more complicated. Since academia, I prefer books to get a systematic introduction to a subject. The first chapters of a book, as well as per-chapter and book conclusions, are the most valuable parts to skim through. It usually works well with more theoretical material. For example, I dug into books about distributed systems and storage architectures. Usually, they are peer-reviewed, thick and stand a couple of editions. The theoretical foundations for those systems rarely change.

(not my actual bookshelf, but close enough, CC0)

There is a different story for more practical books. If you are going to read a book about a specific technology, you have a great chance to stumble upon an older book written a few years ago. The pace of technology is very high nowadays, and that book could be outdated soon after being published. I assume an attentive reader should follow the book and try doing the described topics by oneself, even if exercises and follow-ups are made-up. The first idea is to drop it and start learning from more up-to-date documentation. But wait! Learning from outdated books is a perfect chance to master technology. You would stop taking written as granted and begin to find out why the subsystems or approaches were deprecated or replaced with others. It helps to understand the rationale behind the outdated system design, its advantages and defects, and why the new system is superior to the former. You will dig into documentation more often rather than copy-pasting the code from the book.

And to be honest, everyone has a bookshelf with books that were bought a long time ago just to read them someday. So outdated books are helpful. Use them to your advantage.

Crossposted from my LinkedIn

Sad story about modems and routers

2021-09-13T00:00:00+00:00

Need for broadband

We are so used to have broadband and mobile internet that we take it for granted. This year we escaped to the summer house from a city, heat and pandemic. One of the most important things for us is a good and fast connection.

We have several internet usage scenarios:

casual browsing
watching streaming services
listening to the home music library

I was surprised how easily first the two scenarios can be accomplished even with a crappy internet connection. Browser caching and progressive loading work very nice. YouTube and streaming services can degrade video quality smoothly and buffer minutes of video without hiccups. But when you require a consistent and steady connection, problems arise.

Our house is equipped with a Wi-Fi router connected via a LoRa radio link to the base station which is connected to the ISP by an optic or copper channel. The problem here is a very limited bandwidth of the ISP, the download rate rarely exceeds 3 Mbps.

Okay, there are a lot of 3G/LTE USB dongles that can be plugged into a router and provide a steady LTE connection. Without hesitation, I got one from my mobile ISP with a prepaid plan. While it flawlessly works with a Windows laptop after plugging into a USB port, making it work with a Wi-Fi router took a while.

ZTE modem

The dongle appeared to be a branded ZTE 8810FT device that was sold by MTS ISP (VID 0x19d2, PID 0x1225). It can work as an usb-to-ethernet device or as a modem device. Internally this dongle works on an Android OS and has special drivers for radio and USB networking. Also, it has a simple web interface for configuring basic settings and working with SMS and USSD calls.

What is the difference between those modes? In an ethernet mode, a modem device should work without drivers with any system supporting a generic RNDIS or CDC device. On Linux, you will have a CDC ethernet device (cdc_ether), on Windows most likely an RNDIS device. In a modem mode, a dongle provides a modem device via a COM port that can be used to send AT commands for dialling to the mobile network. Configuring a modem connection is performed by NCM, QMI or MBIM protocols. You will have a /dev/ttyUSB0 or /dev/cdc-wdm0 on successful configuration.

Besides Windows, you are out of luck. On Mac, you need a special driver even for an ethernet mode. Because providing a driver for a USB device requires another USB device (it increases a package cost) or internet access (to download drivers for a modem), the dongle uses a trick. A dongle announces itself as a composite USB device including mass storage. So the first time the host system does not have network drivers for a dongle. A USB disk device appears. A user runs an installer from this disk and installs a driver for usb-to-ethernet. Next time a plugged-in device is detected as an Ethernet device and you get a DHCP-provided IP address. Unfortunately, this trick works extremely bad with headless hosts like a router or a server. It is extremely unreliable with Mac or Windows after touching any part of this fragile system (like removing an Ethernet adapter from the system).

My experiments with ZTE dongle and Linux systems show that the dongle just does not work on Linux. I had tried a laptop with a Debian Buster as well as a TP-LINK TL-WR842N v5 router with an OpenWRT 17 and 21. The dongle can enter an Ethernet mode (ethernet interface is created) but it does not provide a connection to a host system. Entering a modem mode is much more complicated. The dongle should create COM ports only after switching to serial mode (i.e. debug mode) which is not persisted across device reboots. But even before reboot COM ports are non-functioning and do not accept any AT commands on any available OS. A special factory mode that exists for other dongles is not available for this device. So both modes do not work without a special driver that does not exist for Linux.

Amusingly, switching between modes can be performed by nvram tweaks as well as by requesting a web server for a special URL. Hello, CSRF!

Unfortunately, my experiments with ZTE were ended early because this device just bricked without any special efforts from me. Host systems cannot see the USB device. Looks like a read/write filesystem on the device can be damaged due to mode switching, or it is just a faulty device. ISP employee told me I am the third person in a day returning this modem. I doubt all customers are tinkering with firmware so much. So if you can, avoid ZTE 8800FT, especially if you need non-Windows support.

Huawei modem

I purchased another well-known modem Huawei E3372h-320 (VID 0x12d1, PID 0x1f01). It has a usb-to-ethernet mode known as HiLink-mode after the firmware name. There was a more versatile 153 model that has two modes (modem and usb-to-ethernet) but it is out of production.

This new modem was recognized in macOS and Windows after installing vendor-provided drivers. A stock router firmware did not recognize the modem. OpenWRT 21.02 works fine after installing drivers:

   opkg install kmod-usb-net-cdc-ether kmod-usb-net-huawei-cdc-ncm

Then I just configured a bunch of OpenWRT settings and got a new ethernet device with a DHCP-provided IP address.

It would be a happy ending to the story but connections were extremely unstable. A connection to the router via Wi-Fi and Ethernet so the LAN is ok. Let’s check the WAN. Sadly, I can use only the web interface to the router which can only display basic instant metrics. It is awkward. So I decided to monitor metrics from the modem continuously.

Building a monitoring system on InfluxDB

To get metrics programmatically I found a few HTTP endpoints on the modem’s web server that provide data for displaying in a HiLink web interface. It was trivial to get a session cookie because Hilink does not require any authorization. Then I can fetch all data, normalize and store them into a time-series database. Because I gather data periodically, I need to push metrics into a database.

A push-based approach dismissed Prometheus and Grafana which are well-known instruments for visualizing metrics. So I tried InfluxDB. A version 2.0 provides a nice dashboard (ex-Telegraph, I think) which is more than enough to draw plots. Versatile queries allow me to calculate statistics over sliding windows.

I prefer SQL-based query languages like InfluxQL in older versions of InfluxDB. The second version provides a new javascript-based language Flux. It is easy to start with and fun to use but sometimes plain old SQL could be enough.

What I dislike in InfluxDB 2.0 is a very glitchy web interface for editing queries which sometimes lost updates from the query editor. I gave up, fired a Vim and started sending queries via API:

curl -X POST 'http://localhost:8086/api/v2/query?org=hilinkmon-org' \
  --header "Authorization: Token $(sed '/INFLUXDB/!d;s/.*TOKEN=//' .env)" \
  --header 'Accept: application/csv' \
  --header 'Content-type: application/vnd.flux' \
  --data "$(cat test.flux)"

One more disturbing issue is the inequality of table and scalar values in UI. One cannot simply display a scalar value in UI without wrapping it into table. Hope it will be addressed in future versions of UI but for now you are out for using API calls.

The monitoring application hilinkmon I wrote is available on GitHub. It is easy to set up via provided Docker compose. Hope it helps somebody with debugging LTE issues.

The repository includes a predefined dashboard for monitoring with key metrics for LTE connections: RSSI, RSRP, SINR and CQI.

Analyzing metrics

Okay, I can gather a lot of metrics now. Let’s describe what those acronyms stay for:

RSSI (Received Signal Strength Indicator) is signal strength in dBm. It is used in a bar indicator.
RSRP (Reference Signal Received Power) has a close meaning to RSSI.
SINR (Signal Interference Noise Ratio) is a signal to noise ratio. Simple and easy to understand.
CQI (Channel Quality Indication) is a discrete class of quality that specify a modulation scheme and code rate.

Different network generations like 3G and LTE use those metrics but physical interpretation differs. All metrics listed above are better with higher values. Consult Internets for proper ranges of each metric. What is more important is to check metric dynamics as you tune your system.

Why do you need to look for all those metrics? Maximal RSSI does not always mean the best connection. You will get a bad connection if the modem is too optimistic and prefer to pick a higher channel (large CQI is chosen by user equipment by checking SINR) while the signal (SINR and RSSI) is unstable. Looks like Huawei modems do exactly this. By moving the modem around I tend to maximize RSSI while keeping SINR stable to avoid rapid channel jumps.

With a help of hilinkmon, I figured out that my router makes a lot of radio noise to the plugged-in modem. So I wrapped a Wi-Fi router with a foil to make a shield between an LTE modem and router. Only Wi-Fi antennas and vent holes stay open. Things got much better. And hot because it was July. So I switched from a foil shield to an air shield.

A simple USB cable could not work because my modem has a weak PSU and a cable is long so the signal fades. So I built a weird construction with an additional power brick and USB cable splitter. The connection was the best when a dongle was at a precise position hold by rubber bands. No photo of a foil shield and the optimized position, sorry. Check a photo of a generic setup.

When I noticed a frequent switching between 3G and LTE on a graph, I quickly disabled 3G in the modem settings. Then metrics stabilized.

Problems with a router

After all those tweaks I noticed frequent freezes in the Internet connection. Connection from the router to the Internet was ok, connection to the router was ok but the Internet connection from the Wi-Fi client interrupted up to a minute. Nothing in logs, temperature sensors or metrics.

It can be reproduced on OpenWRT 21.02 (last release candidate) by a permanent load like copying files over a VPN connection. No multiple connections, additional Wi-Fi clients, even a simple client can trigger a freeze.

When I almost lost hope, I flashed a beta firmware for TL-WR842N which added support for a Huawei E3372h modem. And it helps!

Of course, I lost a lot of cool OpenWRT features but all freezes disappeared at all. A lot of threads about freezes can be found on OpenWRT forums without proper solutions. Looks like there is a problem in the firmware itself. Maybe this problem is somehow caused by a weak CPU. I did not expect that a super-stable OpenWRT can freeze in a strange unpredictable way. Based on my experience, it is much more likely to see it in DD-WRT firmware betas.

Regarding the stock firmware, I don’t know why adding support for a simple ethernet-based modem from 2014 took so much time for TP-Link (remember, it is still beta). As I understand from OpenWRT, there is no need to use a modem mode. Just add a few lines with USB PIDs/VIDs so an USB device is properly detected.

Outcomes

It was a long journey to a stable and fast connection, and I achieved it only at the end of my vacation. It is a sad story but what did I learn from it?

Never buy hardware that is not confirmed to work on your operating system. Confirmed means a reported success story in a forum or an explicit statement from a vendor.
Sometimes even a super-stable open-source firmware like OpenWRT can experience a strange bug.
Observability is a key engineering principle. Always measure system output before you change something.
Turn problems into a chance to learn something new. Like digging into USB mode-switching and drivers, reverse-engineering a strange android box, learning about cellular networks and trying a new time-series database. At least make it fun.

Talk about Conan and Python at Russian Python Week 2020

2020-09-11T00:00:00+00:00

I will be giving a talk about Conan next Tuesday at Russian Python Week 2020 conference.

We will discuss C++, Conan and the role of Python in Conan package manager development and open-source infrastructure.

This year the conference goes online so the talk format is a cosy Zoom room. Please join us if you are interested in those topics!

The talk will be in Russian.

UPDATED: Slides pdf

Fix apt-get networking problem on MIPS

2016-08-25T00:00:00+00:00

If you are building a chroot with Debian MIPS (sid, Aug 2016) or other non-primary architectures you can get a strange message during a chroot creating:

Err:1 http://httpredir.debian.org/debian sid InRelease
  Temporary failure resolving 'httpredir.debian.org'
Reading package lists... Done

It happens because IPv6 works bad or does not work on MIPS so just add a hook /var/cache/pbuilder/hooks/G90forceIPv4 with the following content:

#!/bin/sh
echo 'Acquire::ForceIPv4 "True";' > /etc/apt/apt.conf.d/80forceIPv4

Optimize Local Guides

2016-03-16T00:00:00+00:00

What is Local Guides?

Usually I contribute to openstreetmap.org with non-commercial objects, transportation and fixing vector maps. They use a wonderful new iD Editor that allows you to quickly edit Maps online without offline editor such as Josm. I tried to do something useful with Google Maps because they were broken at Moscow at many aspects. Recently Google provided an excellent instrument to boost contributing to Google Maps called Local Guides. Gamification and awarding contributors seems to be a vital idea.

There are points given to you for adding a review, photo, rating to existing place, editing missing information or adding a missing place. There are five levels with a few benefits given to contributor at each level. The most attractive one is a one terabyte of Drive storage for two years (4th level). An award for the 5th level is not so useful for non-US citizens. So why not?

Automate everything

I decided to automate tedious tasks of finding places with incomplete information. So I wrote a little Python script using Google Maps API Web Services python library and published it to the GitHub - google-incomplete-places. The script uses API for server applications. Provide a server key to specific location, replace a query type in source code and launch it. Free API is limited for several thousands of requests per day. Do not expect to run your app all day long. I had became tired long before a daily API quota had exceeded.

A good workflow is to open two web browsers and a console at two monitors, click for the next place and search for a place while the edit page is loading (it is not very fast).

Hope it helps you to accelerate a search for incomplete places and gain new rewards. This approach helped to find almost two thousands of broken places such as vet clinics, bike stores, banks, drugstores and hospitals. Now the problem is what to do with one terabyte of data!

How the script works?

The script searches a given area for a specific place type using Places Search. Search requires a place type and a query string (I used a place type here). Then we get a list of Place ids. Unfortunately they are not fully populated with details so it is needed to query Place Details for each found place. Usually a website or opening hours are missing and the script looks for them. Then it composes a handy URL that leads you directly to the edit page. Places API supports paging but limits number of pages so total amount of results is capped at 60. Be creative to formulate a query to get distinct sets of places!

Moving parts in the script are query text and place type. Script can be adapted for your place by changing location and language variables.

Spring clean all your git repos

2016-03-03T00:00:00+00:00

Often we have a lot of git working copies at the machine. They can be old and with a lot unneeded git objects. They could be mass-cleaned to save inodes, optimize speed and lighten backups.

To ease cleaning let’s write a simple script with gc, pack and prune. Remember, gc --aggressive is misleading.

Let’s count how much objects are in all our git repositories:

(find . -name .git -type d -not -path '*exclude*' | xargs -L1 find ) | wc -l

Then use the script to mass-clean all repositories:

find . -name .git -not -path '*exclude*' | xargs -L1 -t gitcleanup.sh

And finally count again with the first one-liner.

For me this simple spring clean reduces git objects from 38631 to 7050.

How to make a network quarantine with firewalld

2015-12-17T00:00:00+00:00

Surely it is not about reinventing a wheel but a short how-to about commands I cannot remember every time.

Sometimes it is needed to make a fully quarantined machine without incoming and outgoing network access, just with SSH connection from the local network. It can be achieved with a few firewalld / iptables commands.

RHEL 7 and CentOS 7 switched from a lot of well-known command line tools such as sysvinit, netstat, ipconfig to newer technologies (literally - systemctl, journalctl, ss, ip). Using iptables directly is not recommended too due to introducing the firewalld. Best tutorial I found is found at Digital Ocean. Native firewalld zones do no allow outgoing traffic filtering so it is needed to add “direct” rules which are clearly iptables rules. There are also Rich rules but I have not tried them yet.

So let’s assume you are in network 192.168.1.0/255. To filter limit outgoing traffic only to IPv4 SSH connections to local network run as root:

firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -m tcp --sport=22 -s 192.168.1.0/16 -d 192.168.1.0/16 -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 1 -p tcp -m tcp --dport=22 -s 192.168.1.0/16 -d 192.168.1.0/16 -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv6 filter OUTPUT 98 -j DROP
firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 99 -j DROP

Then reload the firewalld:

firewall-cmd --reload

Rules are saved permanently at /etc/firewalld/direct.xml. Check the list of loaded rules:

firewall-cmd --direct --get-all-rules

To remove all direct rules run:

firewall-cmd --direct --remove-rules ipv4 filter OUTPUT
firewall-cmd --direct --remove-rules ipv6 filter OUTPUT
firewall-cmd --reload

UNIX way to live preview AsciiDoc

2015-06-21T00:00:00+00:00

AsciiDoc is a nice markup language and I am slowly migrating my documents from various flavours of Markdown to the AsciiDoc.

For converting I prefer AsciiDoctor. Ruby AsciiDoctor implementation seems more robust and modern than original Perl implementation. I edit AsciiDoc documents in TextMate with a AsciiDoc-TextMate-2 plugin. It is a fork of great plugin by mattneub and it is slightly modified for my demands (syntax, AsciiDoctor support).

There are three ways to get a HTML webpage from the document:

Launch a command to get a notes.html file:
```
 asciidoctor notes.adoc
```
Use Ctrl+Shift+H shortcut to regenerate html from the current document.
Use a live reload preview feature.

I love an idea of a live preview and I am using it at my LaTeX workflow.

Official documentation suggests several ways to track changes in editing document and reload a browser. They seems a little complex and insecure for me (browser extensions) so I used a simple shell command:

fswatch -o notes.adoc | xargs -L1 sh -c "asciidoctor notes.adoc && chrome-cli reload"

Utility fswatch is a cross-platform wrapper for file notifications. Each file change is propagated to the asciidoctor command and then to the chrome-cli utility (AppleScript chrome wrapper) that reloads a current tab emits characters for each file change. That is it. You launch a command in a shell only when you need it and then terminate with Ctrl+C.

The best way may be an active browser reloading from TextMate plugin using chrome-cli command but it is not very portable between browsers. Unfortunately built-in open command (xdg-open analogue) does not know about browser internals and cannot reload a tab or even open document in a current tab.

The tale of automating BibDesk

2015-05-31T00:00:00+00:00

For organising scientific publications I use a standard LaTeX tool bibtex.

bibtex

It is wise to switch later to the biblatex which handles UTF-8 better but it is a different story. Bibtex specifies a file format for publications with a lot of standard and custom fields where each field actually is a text key-value pair. For example (wikipedia):

@Book{hicks2001,
 author    = "von Hicks, III, Michael",
 title     = "Design of a Carbon Fiber Composite Grid Structure for the GLAST
              Spacecraft Using a Novel Manufacturing Technique",
 publisher = "Stanford Press",
 year      =  2001,
 address   = "Palo Alto",
 edition   = "1st",
}

You can edit bibtex file by hands but there are some good programs to present and edit publications such as JabRef, Mendeley and BibDesk. I use BibDesk because of its good user interface and integration to OS X.

I am using a lot of specific bibtex tasks for my researches. For example, I often need to find bibtex publication id or title by filename and vice versa, grep citations, set PDF title and author fields from a publication and publishing missing publications to my Kindle via Calibre - a lot of small tasks that require reasonable amount of time and need to be automated. And I prefer a command line utility for these tasks. It accepts a command verb and optional argument and provides a list of strings as an output. Unix way rocks.

Scripting BibDesk

A few years ago I found an exhausted AppleScript support for BibDesk. I could write a script for each of these tasks. Anyone who ever wrote an apple script could understand a complexity of writing a complex data processing applescript. I wanted to wrote a script in a more friendly language, effectively any other language, preferable Ruby or Python or plain C or Java.

Scripting Bridge version

First version of script was written in “Scripting Bridge”. But it stopped working when required MacRuby died. MacRuby was needed because Scripting Bridge is based on Cocoa and the only good way to use Cocoa from Ruby is using MacRuby. MacRuby development halted in 2012.

Appscript version

Second version was rewritten in (ruby appscript)[http://appscript.sourceforge.net/rb-appscript/]. Script was nice except the part where each domain object needs to be manually extracted from the scripting object using .get call. Here is the example for task providing files by citation string:

def files_for_cite cite_str
    app("BibDesk").documents.first.publications.get
        .find { |pub| pub.cite_key.get == cite_str }.linked_files.get
        .map { |f| f.url.get }.compact
end

It uses a typical pseudo-functional Ruby chained call that can filter, map and zip things. Then in 2014 I realised that script does not work in Mavericks and Yosemite. Appscript simply does not builds here due to missing symbols. Official page says it is dead too.

Swift version

A week ago I made a third version of the script. Currently Apple Script supports binding to Objective C, JavaScript using Scripting Bridge. I do not know JavaScript enough to master a script and do not like it at all. Writing a script in Objective C is possible but very verbose. So I got a new Apple language called Swift. Apple positioned Swift as a replacement for Objective C that could work with existing codebase and improve a lot syntax and safety of Objective C. It is good for my purpose!

First of all it’s needed to generate a binding from Scripting Bridge to Swift using an experimental project SwiftScripting. Objective C bindings are supported out of box. Then you need to fix bindings by hands because (see below) it is experimental. Then just write a Swift script.

The same script in Swift looks like:

func files_for_cite(cite: String) -> [String] {
    let app: BibDeskApplication =
        SBApplication(bundleIdentifier: "edu.ucsd.cs.mmccrack.bibdesk")
    let pubs = (app.documents!().get()[0] as! BibDeskDocument).publications!()
        .get() as! [BibDeskPublication]
    let pub = pubs.filter({ pub in pub.citeKey! == cite }).first!
    return (pub.linkedFiles!() as [AnyObject])
        .map({ (f: AnyObject) in f as! BibDeskLinkedFile })
        .filter { x in x != nil }.map { x in x! }
}

Huh! I even cannot make a single chained call because amount of braces became astronomical. Technically Swift specifically encouraged chained calls but they seems very cumbersome because of static type system of Swift where we need to cast proxy chain objects to specific types. Swift functional capabilities are limited to the very weak Foundation and Cocoa library support while Ruby has a lot of useful functions in Enumerable and Array. Sometimes I just wrote a matching replacement for Ruby function for more direct porting.

Swift impressions

It took one or two hours for reading Swift manual and Stack Overflow questions and a few hours to rewrite and debug a dozen of tasks to Swift. Major problems in porting were unwrapping values and type casting.

Optional types are pretty good and could protect you from raw pointers usage and NPEs. It is a little similar to Rust optional enums but with added syntax sugar (question and exclamation marks).

Swift could automatically deduce type from right-hand expressions so variable declaration does not need a type. If type-deduction is not possible you need to manually cast type using as operator. Casting became a nightmare because Scripting Bridge provides only untyped pointers that required casting from/to AnyObject. And it looks like sad programming in Java 1.4 with non-generic containers.

So Swift is a pretty language that is objectively better than Objective C :) It has nice features that simplify existing code and improve its safety and readability. Programming Scripting Bridge in Swift is not very comfortable but entirely possible. Seems like it is the only sane way to script OS X applications without dealing with Apple Script or Objective C syntax.

omnifiles

2015-04-24T00:00:00+00:00

Recently I wrote a small web application for storing temporary files and providing short links to them. It is like a shortener service integrated with file storage. I had found a few simple services for screenshots but it was needed to store pdfs and archives sometimes. Another requirement was to allow easy access by curl both for accessing (downloading) and for storing (uploading). And of course I wanted to store sensitive files at my own server.

This application called omnifiles is open-source and can be found at https://github.com/theirix/omnifiles. omnifiles is a hobby project and a playground to tighten my web skills. I am using it for my own needs but you are welcome to provide feedback or patches!

Let’s talk a little about it’s architecture. It is a simple web app built with Ruby Sinatra framework, HAML for markup, MongoDB for metadata storage and filesystem for file storage.

Storage

Files are stored in filesystem with their unique shortened names. MongoDB contains documents for each shortened link containing shortened link itself (acts as a key), original filename and MIME type, access statistics. Initially metadata was stored in sqlite. Certainly there is no need in scaling, sharding and other CAP stuff but hey, it is 2015! It is reasonable to use NoSQL if there is no strict need in structured data.

I used MongoDB 2.6 and stable ruby driver. Ruby driver was recently rewritten to version 2.0 to support Mongo 3.0 but for now a stable version is sufficient and easy to use.

API

omnifiles should be simple and easily accessible. Requests to store files are sent using POST requests. POST request emits a shortened URL as a response body. I use a curl utility for command-line posting.

There are two ways to store a file (omitting auth and url).

Send a POST form with a single file field:
```
 % curl -F "file=@file.jpg" ...
```

Send a file using POST binary stream:

 % curl -H "Content-Type: application/octet-stream" --data-binary "@file.jpg" ...

Both variants are not very concise. I do not like an artificial form field at the first variant. Second variant just streams a file as a request body. Another vote against first variant is about intermediate form file saving to the temporary directory by the Rack middleware. You can be more efficient with a stream.

File is downloaded by GET request with a shortened link:

    % curl http://localhost/sge36a

Omnifiles provides an original filename as an additional header if a client wants to rename a downloaded file after downloading. Another nice feature is to return a saved MIME type to the response so browser can show images or pdfs directly inside browser window.

I added control panel to omnifiles to view a single file statistics (http://localhost/stat/sge36a), a whole storage statistics (http://localhost/stat) or to delete unneeded files.

Web

Web frontend and backend are written in Sinatra. In omnifiles API and presentation are not clearly separated for the sake of simplicity. For example, routes are not classic REST because it was necessary for me to minimize possible URLs and to group certain URLs for auth in web server. So it is not a proper way to build a REST API service.

I consider Sinatra as a good solution for simple REST services and simple apps without unneeded Rails complexity. There a lot of API, auth, model plugins and Rack middleware to Sinatra so you could build your app from the ground. Sinatra uses Rack middleware to work with requests/responses so with some Rack magic we could distinguish stream against form (see two POST approaches) and store a file in Mongo and filesystem.

Omnifiles separates logic into different apps - public (for GET requests) and protected (POST and control panel). Protected app is using digest auth using Rack::Auth. Routing between two apps is performed by Rack::URLMap. It is definitely less flexible than Rails routes and unfortunately cannot route by HTTP methods.

For presentation I am using HAML. HAML is an another markup language above HTML and it is a lot more concise than ERB. It’s required to learn it for a while it because it seems strange and awkward at first. A common problem with HAML is strict indent and string policy. Resulting markup is compact and beautiful and I like it.

App can be served as a thin app (there is a launcher script bin/omnifiles) or as a Rack app (using config.ru). Command line usage of thin or launcher can be cumbersome as you can see in README.md. Sometimes it is simpler to run Rack app inside Passenger.

pstack for OS X

2015-04-07T00:00:00+00:00

Unfortunately OS X does not have a wonderful Linux-world utility pstack that helps to inspect a running process with a shell one-liner:

pstack `pgrep myprogram`

So I wrote a replacement based on system gdb. Please use it if you find it useful.

I discovered a few interesting things while writing this utility. Homebrew gdb sometimes leaves a process in a sleep state after attaching and detaching. Okay, let’s use system gdb. But it is very old (version 6), non-standard and compiled without python support so we cannot use nice python scripting (initially I found a python script to run inside gdb). Okay, there is a nice native `gdb command

thread apply all bt

that does everything we need. The next problem with gdb was a lack of --ex command-line argument that allows to specify a gdb script inline. So we need to create a temporary file with a gdb script. And finally OS X lacks /proc filesystem so to determine a process binary it’s needed to consult with ps command.

The moral of this story. OS X and Linux are POSIX-compatible but differences environments, different flavors of developer tools and well-known bugs became nightmare to port even so simple utilities.

Amazon AWS SDK v2

2015-02-24T00:00:00+00:00

Recently I noticed that my hobby project usbunfreeze uses AWS SDK version 1. It was fresh vesion when I wrote it at New Year holidays. But a few weeks later the Version 2 became available. It was a challenge to explore a new API without any good up-to-date examples and tutorials. Older SDK contains a lot of official working examples inside. All Ruby service mappings were standard classes with well-defined methods without metaprogramming magic. Then Amazon moved to description of their services via JSON manifests (i.e. EC2.api.json). It is definitely more generic approach but there is no way to look inside the code and check an implementation. Only rubydocs. It looks like YARD docs are also automatically generated from JSON manifests…

All things became clearer when you understand the logic behind the mappings for each service. API is more flexible than v1, allows pagination, advanced async mechanics, provides more structured responses with automatic deserialization. The best reference to hack the new API is an official AWS SDK Ruby blog with a few examples and service descriptions in SDK. Very few projects moved to the new API so GitHub is not the best helper. Hope my project will be usefuly for someone who hacking new SDK right now :)

Tests and color

2015-02-13T00:00:00+00:00

It is a hard choice to pick a good test frameworks for C++. Generally I’m using Google Test and Boost Test. Both frameworks are mature, support fixtures, complex test cases, expects and asserts (Google Test suited me a little better because of non-fatal assertion support)

But there is only one feature that stops me from using Boost Test. It can’t produce colored output. Seriously, it’s 2015 and you must squeeze into raw output to find which test had failed. Do they remember the concept of green bar for unit tests?

It is especially sad that Boost Test has a wonderful extendable output generator (supports both xml and human-readable output) and there is no sign of any activity for adding colored output. People on boost mailing list advised to write generator on your own.

So I have chosen my testing framework by the coloring support…

Welcome

2015-01-05T00:00:00+00:00

It is just a welcome page.