I’ve also published this article in a video form, if you’re more inclined to that format:

Context: What’s testtrim, again?

testtrim is an experimental project to optimize execution of software tests.

It works by:

• running your tests once, and
• analyzing test dependencies (via code coverage + syscall tracing).

Then, when any future software changes occur, it can use that analysis to select which tests need to be run to cover the software change.

I went into more detail on the code coverage side in an earlier post, but today it’s all about syscall tracing.

syscall Tracing

There are two types of dependencies that I want testtrim to find that require syscall tracing:

1. reading files that are part of the repository, and
2. accessing resources over the network.

If a test does either of those things, it requires some special handling to determine when the test needs to be run again. For example, if a file referenced by a test (eg. test_data/Fibonacci_sequence.txt) changes in the future, it would make a lot of sense to run the referencing test (eg. test_fibonacci_sequence).

The tool that I always reach for when I need to do syscall tracing is called strace. It’s a superpower to be able to use strace effectively, but it’s super easy to start with. You take a command that you want to run, and you throw strace at the beginning, and you get a record of all the syscalls that the command made:

$ echo "This is some text."
This is some text.

$ strace echo "This is some text."
... snip output ...
write(1, "This is some text.\n", 19)    = 19
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

It took a couple weeks to develop the first draft of syscall tracing in testtrim, using strace. It was pretty straightforward engineering:

When you run testtrim from the command line:

1. Discover Tests: testtrim finds all the tests in your project (Discover Tests)
2. For Each Test, in Parallel…:
   1. It runs each test under strace, with options including an output file (--output), and including all child processes (--follow-forks).
   2. Open fixture.txt: When the test process (eg. pid 1234) performs a syscall like opening the file “fixture.txt”…
   3. strace writes a record to the output file.
   4. Read Trace Data: After the test is completed, testtrim reads the data file and parses it,
   5. … from which it can interpret the syscalls into dependencies that are part of that test - like in this case, that “test-case” requires “fixture.txt”.

If a file referenced by a test is part of the project’s repo, then it will become a dependency – and in the future, if it’s changed, then “test-case” needs to be rerun.

But, Nested syscall Tracing

But, here’s the catch: I want to use testtrim to test testtrim. I’ve always been a fan of “dogfooding” – although there are cases where it doesn’t make sense. But as a software developer working on a software development tool, it’s a pretty natural fit!

testtrim has integration tests which use miniature test projects to verify that testtrim works. Once syscall tracing was introduced, these tests prevented testtrim from running on its own project because you can’t use strace on a process that is already being traced. (It would work if you weren’t using --follow-forks, but then the trace would be arbitrarily incomplete.)

Honestly, understanding the roadblock this posed was a really disappointing moment for me. syscall tracing worked - my miniature test projects showed it - and there was apparently nothing blocking me from using it on other projects. But it was always a goal of mine for testtrim to work on itself. In addition to be emotionally disappointing, I think it was also important to the project to be able to use testtrim on itself:

• That would allow me to work out kinks and bugs involved in real-world usage.
• It would generate the dataset that would prove (or disprove 🤷) its own value in test target reduction.
• And I didn’t want to be a hypocrite trying to promote something that I didn’t even use myself.

It had to be solved.

Nested Tracing v1: What could go wrong?

After a bit of disappointment, I had a lightbulb moment: I control both of these programs. Can I just make them cooperate? The “tracer” knows everything that all of these processes are doing - it’s just a simple matter of getting the data from testtrim over to the integration test.

So, first pass… ah, I was so naive…

1. Inject an environment variable when running a test child that contains the path to the strace output file.
2. In the test, recognize the environment variable and:
   1. Do not use strace.
   2. Read the strace output file contained in the environment variable.
   3. Filter out the file to just the process IDs that are relevant to the completed test.

This does somewhat work - both trace points are able to identify the knowledge that they need.

But, if you can see the problem with this approach, then you are very, very clever. I, on the other hand, did not.

You see, every time the child goes to read data from the strace output file, it has to perform a read syscall… which goes back to the parent’s strace… which gets written to the output file… which then the child has to read…

sigh

It’s not quite an infinite loop. (Well, it was when I first ran the code, of course.) But there is an exit condition that can be added in: the integration test can stop reading the trace file once it finds the process exit record for its test cases. For example, in the diagram above the test was process 4321, and so process 1234 can stop when it hits the exit condition for 4321. So not infinite, but incredibly magnified.

The performance of this solution was very bad - the more tests that needed to be run in a subprocess, the more times it was re-reading the syscall data, which included every syscall involved in the last re-read of the data. When combined with multiple parallel tests running, there’s a geometric growth in strace output that needs to be read.

So, I came up with an alternative that seemed far more promising… for a while.

Nested Tracing v2: This Will Work For Sure!

strace has an option called --output-separately. This changes the output location to a directory rather than a file, and outputs each process that was traced into its own file.

This would work with testtrim wonderfully:

1. Use --output-separately to write one file per process.
2. Pass the location of the files to the child process.
3. When the integration tests’s test process completes, it can read just the child PID’s file – which is a terminated subprocess, avoiding the syscall amplification.

There is, of course, a complication. It seems small, and then it becomes really messy.

We can’t just read the file with the child’s PID, but we also need to read any subprocesses of that process, and subprocesses of those processes.

Like I said, seems simple, right? Read the child PID’s file, and since it is a record of syscalls, simply note any processes that it spawned and read those files as well. Repeat as necessary.

This seemed great! For a bit.

But the subtle problem is that processes aren’t independent. What software developers would usually call a “process” is a program execution which is operating independently; and we’d use the term “thread” to refer to multiple concurrent execution within a process. But in the Linux kernel, and in strace’s output, a thread is a process. strace outputs each thread to a separate output file… but they have shared state that is relevant to our syscall tracing.

To keep things simple, we’ll use the current working directory as an example. Let’s say pid 4321 and pid 4322 are threads, and our strace output shows:

$ cat test-case.strace.4321:
execve("test-executable", ["test-case"]) = ?
clone(... CLONE_THREAD, ...) = 4322
...snip...
openat(AT_CWD, "fixture.txt", ...) = ...
...snip...
+++ exited with code 0 +++

$ cat test-case.strace.4322
...snip...
chdir("../test_data")
...snip...
+++ exited with code 0 +++

We know:

• PID 4321 opened the relative file path fixture.txt
• PID 4322 changed the current working directory to ../test_data

But we don’t know what order those two events happened in. The order of them completely changes which file was accessed by the test.

After I started sending strace data to separate files, I no longer know the order of the events in them. And if they can affect shared state, like the current working directory of the process, or access shared file descriptors between the threads, the effective events of the process are lost.

I took a brief look at using strace’s timing outputs in order to provide ordering, but first it seemed sketchy in terms of accuracy, and second the performance of strace gets much, much worse. It needs to constantly access the system clock, so that performance drop makes sense.

Nested Tracing v3: syscall Tracing Sans syscalls?

Getting to a solution that met all the criteria required a little backtracking, one simple tweak, and one complicated tweak.

First, I reverted to the single output stream, rather than --output-separately - other than timestamps, this was the only way I was going to get an ordered view of the syscalls.

The simple tweak was to start to stream the data from strace into the parent process through a FIFO pipe, which allowed the parent process to parse and understand the asynchronously while waiting for the subprocess to finish. Now that data was live in the parent process, and I just needed to get access to that event stream from the child… without using syscalls. (Well, syscalls to get things “started” were fine, but there had to be no syscalls involved in reading data.)

Simplifying a few details, the architecture ends up looking like this:

1. The testtrim process opens a UNIX socket, and passes the address down to subprocesses through an environment variable.
2. When an integration test wants to start a test-case with nested tracing, it connects to the testtrim process, and sends a request to subscribe to the PID of the child process.
3. testtrim starts a shared-memory buffer and begins to store any syscalls from the PID of the child. This is where the live connection through the pipe to strace matters - we’re able to instantly connect into that data stream and copy it into the buffer.
4. testtrim sends back to the test process an address for the shared memory buffer.
5. On an ongoing basis, testtrim writes syscall data to the shared buffer, and the test process reads it out. A ring buffer is used so that we can have a fixed memory allocation which supports the continuous stream between the two processes.

There are are few tricky parts of the implementation:

• When a child process starts a test, we can’t miss syscalls - so we need to synchronize starting up the child process, subscribing to the events, receiving acknowledgement that we subscribed, and then allowing the child process to actually start.
• We can’t guarantee the order of syscalls coming out of strace: a child process’s clone or fork or whatever might begin in the syscall stream, and then syscalls from that child process would appear, and then the finish of the clone/fork/etc could be emitted. So testtrim needs to occasionally “lightly reorder” the syscalls during process startup, just to ensure that subprocesses are recognized correctly.
• When a test process terminates, we need to make sure that all the syscalls are delivered to the child process, and none are still in-memory. To accomplish this, the testtrim process over here keeps an eye out for process exit messages and sends an EOF marker into the shared memory buffer.

Conclusion

Does it work?

Yes.

It took more than a few passes to get the synchronization logic correct, and with this complexity there might still be a surprise or two to learn. But, it functions!

With this design, nested strace data collection is possible, and the order of syscalls is preserved such that we can recognize dependencies between threads like changing the working directory on one thread, and opening a file on another.

This has been the final piece in the puzzle to getting testtrim to be able to run on its own project, in its own CI. It’s a shame that the nested logic will never be useful for anything else, but it’s pretty cool.

There’s one more neat thing that testtrim is doing with syscall tracing which is more generally applicable to other test projects - but that seems like a topic for another post. Until then, hope you’ve enjoyed this deep-dive!

Table of contents:

• Current Project Goals
• New Features Since Last Update
  • syscall Tracing
  • External Dependency Tracing
  • Go & .NET Platforms
  • Remote API Server & Client
  • History Simulation
  • Minor Features
• testtrim on testtrim (“Self Hosting”)
  • Real-world Network Tracing
  • Recursive strace Failure
• What’s the future?
• Follow along, share your thoughts!

Current Project Goals

The core goal of the project is: determine whether this will work for large-scale projects.

In short, the concept is:

1. Just like you would use to report test coverage (eg. “75% of our code is tested!”), run tests with a coverage tool. But rather than running the entire test suite and reporting generalized coverage, run each individual test to get the coverage for each test.
2. Invert the data; change “test case touches code” into “code was touched by test case”, and then store it into a database.
3. Look at a source control diff since the last time you did #2 to find out what changes occurred to the code, then look them up in the database to see what test cases need to be run.

It has been trivial to prove that this will work on microscopic test projects, but real-world projects come with a lot of complexity which muddy the issue up:

• Tests which read data files, for example for test fixture data
• Tests which access network resources, which you might want to run always, or you might want to run never, or somewhere in-between
• Programming environments which allow you to embed data files into code (eg. in Rust, include!() and similar; in Go //go:embed; etc.)
• Upgrades of third-party dependencies

These are just some of the most obvious problems. In order to uncover more problems, and develop solutions to these problems, testtrim has two medium-term goals:

1. Run testtrim’s own test automation suite, in it’s own CI system, under testtrim.
2. Incorporate testtrim into an Open Source product’s CI system.

As these goals are tackled I expect to get some clarity on that core determination, and find out whether this concept will work, won’t work, or most likely, that it will work but has some limited applicability which I can begin to define.

New Features Since Last Update

syscall Tracing

On Linux, testtrim is capable of tracing a test with strace in order to identify all the system calls (syscalls) that are being made by a test. After analyzing the output, testtrim identifies two things that add onto the code-coverage logic and could cause additional reasons for a test to run again in the future:

• Opening local files within the repo
  • If test_a reads a local file tests/fixture/data.txt when the test is run, then in the future if the git data shows that tests/fixture/data.txt is modified, testtrim will rerun test_a.
• Network access
  • If test_a opens a network port to 10.1.1.1:5432 when the test is run, then in the future testtrim will always rerun test_a with the assumption that the network access might “be different now”.

However, the network access logic of “rerun every network test” is extremely conservative. So, testtrim now has the capability to have network configuration in order to customize the behavior for network access; it is now possible to:

• Ignore test network access – good for something like an internal test server.
• Rerun tests that access the network, only when other files change – good for something like “rerun all database tests whenever the database schema in these files change”.

External Dependency Tracing

When you upgrade an external dependency, testtrim will identify which tests used that dependency and rerun those tests. The modification to Cargo.lock or go.mod will be used to identify the changed module, and coverage-tracking being enabled across the dependencies will be used to find the relevant tests.

Go & .NET Platforms

testtrim is a project written in Rust, and so it targeted Rust as a first platform to operate on. I added some abstractions early in the development to support multiple platforms in the future, but I didn’t have any idea whether those abstractions would allow other platforms to actually be implemented.

The Go platform is basically feature-for-feature comparable with Rust. There is one major exception: Go doesn’t perform coverage tracing on test files (*_tests.go). testtrim tries to cover this up with some file heuristics and regexes, but it does leave a gap where functions in *_tests.go that are reused in other tests won’t trigger their execution correctly.

The .NET platform ran into some very large limitations with external dependency tracking and has been frozen half-implemented for the time being. Further developing it doesn’t get me any closer to the main “determine whether this will work for large-scale projects” goal, but I think it will be possible to improve it in the future at the right time.

Remote API Server & Client

In order to use testtrim in a CI environment, it will be necessary to have a persistent database that is accessible between CI invocations. To support this, testtrim has an API server and client that exposes its coverage database.

The server is run with the testtrim run-server subcommand.

The client is used by setting the TESTTRIM_DATABASE_URL to access the http or https address of the server.

History Simulation

I’ve been exploring how effective testtrim is by using Open Source projects, and running testtrim against their recent commits. In other words, going back 100 commits, and running testtrim on each commit afterwards to see how many tests will be run.

The testtrim simulate-history subcommand automates this process and outputs a CSV file with statistical information about the simulation.

For example, as part of developing the Go platform I completed a simulation of an Open Source project called zap. The simulation results are inline with other Rust projects, suggesting about 90% of the tests are skipped on average.

Minor Features

• Embedded files:
  • When files are embedded in the source code (Rust: include!(), include_str!(), include_bytes!(), Go: //go:embed), any modifications to those files will be treated as a modification to the embedding file. This means that related tests should be rerun automatically.
• Test coverage tags:
  • When running tests, the --tags command-line option can be used to differentiate coverage, allowing the same project to be tested in different configurations. For example, you could run tests with a tag database=postgresql, and later tests with a tag database=mysql, and the two tags would have coverage maps tracked separately. This would allow a change that only affects one codepath to trigger only tests that are related to that codepath.
  • An automatic platform tag is enabled by default, eg. x86_64-unknown-linux-gnu. If a project is multiplatform but uses conditional compilation, this will prevent corrupted coverage data from leaking between the two platforms.
• Transparency into test targeting:
  • When get-test-identifiers is run, testtrim outputs each test that it plans to run for the current repository, as well as why it will run. It might run because: there’s no coverage map to calculate results on, the test is new, a file has changed, a network dependency is present, an external dependency of the project has changed, etc.
• Parallel test execution:
  • For the Go and Rust projects, test execution is parallelized to improve performance.
• Release tooling:
  • As part of running testrim in a CI environment, and hosting a testtrim API server, I need to have a released version of testtrim. testrim now has a fully automated release process with automatic changelogs, and publishes an OCI container for running the API server.

testtrim on testtrim (“Self Hosting”)

A lot of issues have been resolved on the pathway to getting testtrim to run on itself, but a couple complex issues remain:

Real-world Network Tracing

testtrim runs its tests under strace and captures the network access that the tests performs. It’s very common for an automated test to reach out to a database, or a locally hosted version of the application, for example.

flowchart LR
    localhost:5432[(PostgreSQL on localhost:5432)]
    localhost:8443[(API on localhost:8443)]
    testtrim-->test_coverage_db
    test_coverage_db-->localhost:8443
    test_coverage_db-->localhost:5432

testtrim has new capabilities to match against these network patterns and apply rules to them. FIXME: add a link here to the network policy

When running testtrim-on-testtrim, most of the tests had network access patterns that were easy to match with network patterns and apply the right logic to the tests. But one end-to-end integration test started to expose some complications. This is a trimmed output from the get-test-identifiers command, which in testtrim shows you which test will be run and why; all the easy to deal with problems have been removed:

RustTestIdentifier { test_src_path: "tests/linearcommits_filecoverage_tests.rs", test_name: "linearcommits_filecoverage::dotnet_test::dotnet_linearcommits_filecoverage" }
  ...
	NetworkDependency(Inet([::ffff:152.199.4.184]:443))
	NetworkDependency(Inet([::ffff:192.229.211.108]:80))
	NetworkDependency(Inet([::ffff:23.217.131.226]:80))
	NetworkDependency(Inet([2001:67c:1401:20f0::1]:443))
	NetworkDependency(Inet(10.89.0.1:53))
	NetworkDependency(Inet(217.197.91.145:443))
  ...

The remaining network dependencies fall into three categories:

• Downloading from nuget.org during this .NET build process. This specific test is running a .NET build with an external dependency that needs to be downloaded – but even though there’s network access on every test, it’s downloading a file that doesn’t change between test runs.
• Downloading from codeberg.org a test repository (dotnet-test-specimen) to run the .NET project. This is somewhat the “subject under test” and might be kinda fair – but again, I don’t want this test to rerun just because it does this behavior.
• Accessing a CRL/CA check to inspect whether any https access is subject to a certificate revocation list. There is extremely little reason to rerun tests because of this dependency – but need to identify how to ignore it accurately.

Boiling down these examples a little gives two problems:

• Sometimes a test might download “static content”. You don’t want the test to rerun just because it does this download; you want it to rerun if the content changes.
  • nixpkgs has an approach to external dependencies that might be inspiration for a solution here: download the dependencies outside of the test, hash them, use the hashes as an input to the test that is traced by testtrim, and then update the tests to use the downloaded local copies. It’s a great outcome for test quality – they get the right external content, they rerun when it changes, the test accomplishes the same test goal – but has the downside of drastically changing how a test is run. Maybe with tooling that does the heavy lifting it would be feasible?
• When a test accesses a network resource, the syscall tracing doesn’t capture the DNS resolution involved, just the raw socket connect syscalls with their IP addresses. As a result it’s hard to create useful long-term network policies.
  • I’d love to be able to intercept the DNS requests a process makes and match the network policies on DNS names. With modern Linux systems that use caching daemons (nscd) I’m not sure if I can capture these requests through an strace…

Recursive strace Failure

When testtrim runs on a project, it runs each test independently to collect its coverage data. Each of those tests is also run under strace in order to capture the network and filesystem access that the test performs.

Imagine that you’re running testtrim on a project called “Test Project”, which has a test case “ABC”. After discovering all the tests in the project and evaluating what needs to be run, testtrim will eventually run the test “ABC”, wrapped in strace:

sequenceDiagram
    testtrim->>strace: Run Test ABC
    strace->>Test Project: Run Test ABC
    Test Project->>strace: Test Results w/ Coverage
    strace->>testtrim: Test Results w/ Coverage & syscalls

However, when I’m attempting to run testtrim on testtrim, it has a small number of test cases which validate the strace functionality. That means that after discovering all the tests in the target project (testtrim), testtrim will try to run a subprocess to execute that test under strace. (testtrim(cli) here indicates the testtrim command-line that is being executed, and testtrim(sut) indicates testtrim in the context of being a subject-under-test):

sequenceDiagram
    testtrim(cli)->>strace(1): Run Test 'e2e-test'
    strace(1)->>testtrim(sut): Run Test 'e2e-test'
    testtrim(sut)->>strace(2): Test uses `strace`...
    strace(2)--xtesttrim(sut): PTRACE_TRACEME:
Operation not permitted

This failure is because strace(1) is running the test process with the --follow-forks option to capture all subprocesses’s information. The subprocess is already being traced, and another invocation of strace will fail.

This is an outstanding issue that prevents testtrim-on-testtrim, but I think it’s a pretty narrow problem that doesn’t generalize very well to other projects. That said, I still want to get it resolved. There are a few approaches that I have in mind:

• It would be possible to mark some tests as exempt from syscall tracing. The affected tests won’t be subject to reexecution when they should be, but, it’s an option.
• It might be possible for the subprocess to recognize that it is being traced already by testtrim, and then work with the tracing data generated by it. This would look something like testtrim(cli) setting an environment variable TESTTRIM_STRACE_OUTPUT, and then when testtrim(sut) starts its test it won’t start a new strace, but instead will read the data file at TESTTRIM_STRACE_OUTPUT for its syscalls. It’s a pretty ugly solution.
• testtrim could have a new capability added to skip tests. Although this doesn’t truly solve this problem, it might be a reasonable workaround – I want testtrim to run on it’s own CI, but I also think I can’t trust testtrim to test testtrim, and it might make sense for testtrim to rerun its test suite unconditionally.
• testtrim could use some compile-time flags to remove the tests, rather than skip tests, during the CI.

An interesting problem that I’m a little stumped on, other than “don’t run these tests”.

What’s the future?

Short term:

• Finish having testtrim integrated into it’s CI.
  • Discover new problems from this. Fix them.
• Choose a target Open Source project to integrate testtrim into.
  • Discover new problems from this. Fix them.

Medium-term:

• testtrim should be usable on developer workstations to reduce test cycle time. But there are some obvious blockers:
  • syscall tracing only supported on Linux.
  • Other capabilities that might be platform-dependent (eg. file pathing related work) haven’t been tested extensively outside of Linux.
• Evaluate alternatives to having testtrim be a test executor.
  • Every platform has basic or advanced tools for test execution; for example, Rust has a great cargo test capability, but also an advanced alternative cargo-nextest with great value-add. I want testtrim to be able to be focused on the challenging problem of fast, accurate, test impact analysis and test selection. I don’t want to have to rebuild all the other capabilities of a test executor. Reconciling these two paths will be important to making a tool that can be value-add to an ecosystem.

Longer-term:

• Distributed testing – eg. web application testing – I have a theory on how to do this with Open Telemetry tracing, but no design, no implementation today.

Follow along, share your thoughts!

testtrim is under heavy development, and work will continue in 2025… until we see whether it will really be practical. It’s an Open Source project, and I’d love to get feedback from people on the concept, and share in the development as it goes.

• Source Repo
• YouTube Channel for Updates
• Mastodon for Updates / Feedback
• Bluesky for Updates / Feedback
• Blog RSS for Updates

If you’re more inclined to follow this kind of content in a video format, I’ve also published an introductory video (in two formats):

Short Introduction Video, 10 minutes

Deep-dive Introduction Video, 37 minutes

Problem

The longer a good software project lives, the longer its automated testing solution will take to run. This is especially true if the engineers responsible aren’t able to dedicate a lot of time to carefully managing their test suite, pruning it like a Japanese garden into a state of zen minimalism. In my experience, nobody has time for that.

Long automated testing cycles can be managed pretty effectively; they can be parallelized, they can be run on the fastest hardware, changes can be batched together, they can be moved around to different stages of the release cycle to get early likely-failure feedback, and of course, they can be [Ignored] (or .skip or #[ignore] or whatever.)

But can they be made more efficient by only running the things that are relevant to test the change being made?

testtrim’s Equation

1. Just like you would use to report test coverage (eg. “75% of our code is tested!”), run tests with a coverage tool. But rather than running the entire test suite and reporting generalized coverage, run each individual test to get the coverage for each test.

2. Invert the data; change “test case touches code” into “code was touched by test case”, and then store it into a database.

3. Look at a source control diff since the last time you did #2 to find out what changes occurred to the code, then look them up in the database to see what test cases need to be run.

This is the core concept behind testtrim.

How well does it work?

It’s early days for testtrim.

To evaluate how well it worked, I took an Open Source project (alacritty), and I ran the last 100 commits through testtrim.

For each commit, testtrim identified that an average of only 8.4% of tests needed to be executed to fully test the code change that was being made.

I could list a dozen reasons why this analysis isn’t generalizable… and so I will:

• This analysis didn’t include changes to Cargo.lock files (eg. references to external dependencies). I’ve added that capability to testtrim, but I haven’t redone this analysis yet.
• alacritty is a pretty mature project that isn’t undergoing rapid development.
• This specific measurement didn’t take into account new tests being added to the repo during this period.
• alacritty has some tests that work through reading static files; those commits were removed from the analysis, possibly lowering the numbers.
• There’s no evidence that a test on a single project will generalize to lots of other projects.
• The only guarantee of correctness in this analysis is my own eyeballing of the changes and proposed tests.

But on the other hand, this was just based upon the simple heuristic of “this test touched a file, and that file changed, therefore rerun this test.” I think that could become a lot more sophisticated with more work in the future as well.

I think it’s promising, but not promised.

What’s the future?

• Small scale technical items – these prevent testtrim from working well, even in the scope of Rust local apps which are presumably going to be its jam since that’s the only thing it supports today:
  • Test execution performance / parallelism – testtrim doesn’t run tests in parallel, so it’s slower than using cargo test even when it runs fewer tests
  • Doesn’t work on all Rust codebases – possibly indicative of environmental differences between cargo test and testtrim, but could be other issues
• Larger requirements – things that would really unlock value
  • Multiple platforms – eg. JavaScript, Java, C#, etc.
  • Distributed testing – eg. web application testing – I have a theory on how to do this with Open Telemetry tracing, but no implementation today
• Project risks – aside from technical work
  • How effective can it really be? – a few data points aren’t quite enough to be confident
  • How effective does it have to be for people to want to pick it up and use it? (Cost/Benefit)
    • Even as an Open Source project (which is the plan, today!), it still has costs to implement: complexity, engineering time, operational costs, and maintenance costs

Follow along, share your thoughts!

testtrim is under heavy development, and I expect it to move quickly through the end of 2024 and early 2025 into… well, we’ll see. It’s an Open Source project, and I’d love to get feedback from people on the concept, and share in the development as it goes.

• Source Repo
• YouTube Channel for Updates
• Mastodon for Updates / Feedback
• Bluesky for Updates / Feedback
• Blog RSS for Updates

pixelperfectpi

The first target was my LED matrix clock. There were some things that needed tweaking since I wasn’t in Calgary. The weather forecast was read from Environment Canada’s XML feeds – no good in Paris. So, I rebuilt weather based upon Home Assistant weather components, sent to an MQTT broker, and the clock reads that data. There are plenty of HA weather components for different services; easy to adapt this to any new location. Also added a UV index display.

https://github.com/mfenniak/pixelperfectpi/compare/40383af796900b3fcac19b6a15185a6da5752fd6…d05d6e39217fa17b9aa11ce8bb2a3d6e10256713

Along the way, I ripped out the usage of the dependency_injector library since it’s no longer supported upstream. I like having a DI-based design… but I will admit that for a Python app, and a small Python app at that, it’s probably not super useful.

https://github.com/mfenniak/pixelperfectpi/commit/07e3c04d06a59e8d840598ce0b6a25a555a1473a

Now that I had a better source of weather data, I really wanted to display an hourly forecast. The clock is very small so I was limited in the display options… but also all the layout to-date had been shitty pixel hard-coding. So the next improvement was reimplementing the layout on the clock to use a flexbox calculator stretchable, which required touching every component and changing how they handled layout and drawing.

https://github.com/mfenniak/pixelperfectpi/compare/f7924871c88e9c5ec2c172594cee71ef9c8d1389…e51e97c1ca4b09c0f00cafbed3d4cecbd9f48878

The hourly weather output isn’t super amazing… but it’s useful enough. One last tweak to the clock was adding a display of the time at home. https://github.com/mfenniak/pixelperfectpi/commit/fbfabe331141d247c6021068989cf4607a089fd1

Crowd Control Video Game

Having my clock all settled, I started getting interested in the idea of a video game with the Bevy engine. I had built up a Godot tutorial game earlier in the year, and I rewrote that game with Bevy to learn the basics. Not really too exciting, but it set the foundation to learn how to code with Bevy’s ECS model.

As we traveled around France and started to attend events during the Paris 2024 Olympics, I was exposed to good, bad, and complicated crowd control mechanisms. Barriers, volunteers, police, signage & lack of signage, security, etc. All of these contributed to holding safe events where crowds were efficiently moved from place to place with limited time & risk. And it’s all thrown into chaos if someone rides in a bike in the wrong place.

Well, that sounds like it could be a video game to me.

Many of you reading this will immediately think about the complex part of building this – pathfinding. Video game pathfinding is difficult, especially if you consider that every actor in this system would have a different goal, a different level of patience, a different level of comfort getting close to others, shoving others, moving a barrier, ignoring a sign…

Anyway, I hacked away at a proof of concept with Bevy. Just enough to see, if I have a “crowd generator” on one side of a map, and a variety of “targets” on the other side of the map, can I impact the efficiency of moving people just by moving barriers? The answer is yes, but making the pathfinding efficient is hard. 😝 So, it’s a fun idea that I might revisit in the future, but I put it aside.

hyprland

At this point, I decided to sabotage my own productivity for a bit. It’s completely rational though. You see, I’ve been using a Plasma KDE 5 desktop for a few years, and I’ve become very accustom to the bismuth window tiling plugin. But, it isn’t supported in Plasma 6, and while there is an autotiler polonium for Plasma 6, I don’t find it to be nearly the same experience.

So I thought this was a good opportunity to give a solid try to hyprland, a tiling window compositor. Well… let’s say it’s a throwback to my early Linux desktop days. Other than composing windows, everything else you might want in a desktop environment is something to piece together. So, a week later, and I’ve configured hyprland, bemoji, wl-clipboard, wtype, cliphist, rofi, grimblast, waybar, dunst, hyprlock, hypridle… and I feel like I have a usable system.

What did I gain out of this? I’ve made the move to Wayland, and I’m not blocked on that Plasma major upgrade.

But I also love the multi-monitor support of hyprland. I have a set of workspaces dedicated to each monitor. When I disconnect the 2nd monitor, the workspaces automatically collapse back to the 1st monitor, and when I reconnect it, they pop back.

I’m not sold on this for the long term, but I’m pretty happy with it for a little over a month. We’ll see what the maintenance looks like!

Zenbook Duo Fixes

Speaking of the dual-screen laptop, the Asus Zenbook Duo UX8406MA, one of my goals during the vacation was to make some Linux compatibility improvements for it. This sorta happened; I fixed two things, kinda.

Secondary Display

The first was that myself and others experienced a regression in June where the secondary display stopped working, which I reported upstream to the i915 team. In July an engineer provided a patch which fixed the problem.

However in August, the secondary display stopped working again. I started to bisect the kernel to find the issue, but made a grave error - I assumed that the current kernel was bad and my last kernel was good. This seemed to make sense - after all, it was working and then broken, right?

Each bisect required having my desktop at home build the kernel remotely, and then download it. After dozens of iterations I eventually found that the “broken commit” didn’t do anything when reverted. Whoops.

So I changed my approach to bisecting nixpkgs, and eventually found that the linux-firmware-20240811 was the true cause of the secondary display breaking. Reverting this upgrade restored the secondary display again. The fault was reverted out of linux-firmware, and the 20240909 is back to working.

So, I didn’t really fix anything here, but I did help identify some problems.

rfkill

Another problem with the UX8406MA is that connecting the keyboard to the laptop caused the wifi to shut off. I think the cause is that when the keyboard is turning off its Bluetooth on USB connect, it happens to send a wireless disable keypress? But that’s a bit guessy.

Regardless of the cause, I was able to create a kernel patch that detected and discarded these keypresses from this specific keyboard, which otherwise doesn’t have a wireless disable button.

This is the first time I’ve ever sent a Linux kernel patch upstream, and I’m very pleased that the platform maintainer integrated it in just a few days with one quick review cycle. https://github.com/torvalds/linux/commit/9286dfd5735b9cceb6a14bdf15e13400ccb60fe7 I’m a Linux kernel contributor!

This summer was great. One more stretch of hacking left…

testtrim

As September started to tick on, which I think is summer but my wife says is fall (and she’s technically correct, the best kind of correct), I started to experiment on a project that I’m calling testtrim.

The goal of testtrim is: run only the automated tests that are required to test the changes being made for software. The way it works is by running each test in an individually instrumented environment, and then using the code coverage map that generates to figure out what code a test runs.

I’m going to share a lot more about this when I get home and spend some more time working on it. Right now though, it looks very promising. I think it has quite a complex future in front of it… but a lot of potential value.

The original thermostat was a low-frills device. It allowed for two schedules to be configured, one for weekdays, and one for weekends, with the floor heat being set to four different temperatures at four different times. The temperatures were not measured in degrees, but rather, in unlabeled numbers from 1 to 10. The thermostat’s control interface was… better to not use it. And it lost time constantly, requiring manual adjustment every few months.

On the “backend”, the thermostat was a simple device. It had a 120V AC input for power, and a 120V AC output for the resistive heating of the floor. A thermistor input was used to measure the temperature of the floor; a thermistor is a resistor whose resistance changes with temperature.

So, things that needed to be done:

• Figure out what I was going to replace this with that could both control the 120V AC output and measure the thermistor input.
• Rip out the old thermostat and install and wire the new device.
• Integrate it with Home Assistant for remote control.
• Test that I could control the 120V AC output; at this point it was pretty much a win because a good scheduling configuration would be fine.
• Bonus points: figure out how to measure the thermistor input and integrate that into Home Assistant.

Hardware Hunt

My usual go-to for such projects would be an ESP32, flashed with ESPHome for its versatility and ease of use. Everything on my TODO list would be pretty straightforward to implement with it… but, usually when I work with an ESP32 I’m working with low-voltage DC, not high-voltage AC. I wanted something that was designed to be installed in a wall gang box, and that could handle the high-voltage AC safely. The resistive heating of the floor is about a 5A, 600W load; it would be easy to create a fire hazard if I didn’t do this right.

After some research, I opted for the Shelly Plus 1, a compact Wi-Fi & Bluetooth smart switch that’s designed precisely for such scenarios. I was originally thinking of going with a switch that I would flash with ESPHome… while I think that’s possible still with the Shelly Plus 1, I decided it might make more sense to just try it out-of-the-box first.

Additionally, to manage the temperature sensing with the thermistor, the Shelly Plus Add-on seemed like the perfect addition to the Plus 1. It’s obviously designed for this exact usage; it has a voltage divider circuit that’s perfect for reading the thermistor, and it snaps right onto the Plus 1.

Physical Installation

With the Shelly Plus 1 and Add-on in hand, the next step was installation.

The AC power wiring was quite straightforward; my input AC power was split between the “L” input to power the Shelly Plus 1, and the “I” input as an input for the relay. The output AC to the floor heating was wired to the “O” output of the relay. The neutrals from the input, the floor, and the Shelly Plus 1 were all connected together.

The thermistor was a bit more confusing. Shelly’s documentation for the Plus Add-on had a perfectly correct diagram, but, I screwed it up on the first pass due to my lack of familiarity with the voltage divider circuit. Once I figured it out, it was easy to fix:

• “VREF + R1 Out” was connected into a wire connector.
• One wire from the wire connector was connected to the “ANALOG IN”.
• The other wire from the wire connector was connected to the thermistor.
• The other end of the thermistor was connected to the “GND”.

Software Set-up

It’s hardly worth talking about the initial connection of the Shelly to Home Assistant; at least in my experience it was virtually automatic. I connected to the Shelly via its builtin Wi-Fi, and configured its connection to my home Wi-Fi. Home Assistant picked it up immediately, and I was able to control the relay output from Home Assistant.

At that point, I was able to write automations for Home Assistant to turn the floor heat on and off. With some timing settings, this probably would have been perfectly fine… but I was going for those bonus points.

Getting the thermistor read wasn’t exactly challenging… it was just undocumented. Which is really the primary reason why I thought I’d put this blog post up; because otherwise this was just a matter of reading the documentation and following the instructions.

So, here were the steps I took to get the thermistor read:

1. I configured the Shelly Plus Add-on as a voltmeter. This was done via the Shelly Web UI, which I accessed through my local Wi-Fi address, but it’s also possible to do it through the Shelly App. There aren’t a lot of settings involved here, so it’s not exactly complicated… but I wasn’t sure whether I should be using the “Analog Input” (since I’m wired to the “ANALOG IN”, you know), or the Voltmeter, and eventually figured out that the Voltmeter setting was what I needed for the next steps.

2. After 5-10 minutes, the Voltmeter began to appear in Home Assistant.

3. I configured a template sensor in Home Assistant to convert the voltage into a temperature. This was a bit of a challenge, because I didn’t have the thermistor’s specifications. What I did know was: the reference resistor in the Shelly Plus Add-on’s voltage divider circuit is 10k Ohm; the reference voltage was 10 V. Theoretically I either needed a B-calibration value and a reference temperature of the thermistor, or, to measure a few temperatures and calculate the appropriate calibration values. Well, I couldn’t find the right documentation, and changing the temperature of a floor by a huge amount and measuring it consistently is pretty impractical, so I chose reasonable approximate values for those, an extracted the NTC resistance to temperature calculations from ESPHome.

   The values ref_voltage, ref_resistor are used to calculate the resistence in the thermistor, and these values refer to the voltage on the circuit and the pulldown resistor in the Shelly Plus Add-On.

   The values b_constant, ref_temp, and ref_resistence are used to calculate the temperature from the resistence, and refer to the B-calibration value of the thermistor, the reference temperature and reference resistance that the B-calibration value was measured at.

    template:
    - sensor:
        - name: "Bathroom Floor Temperature"
            unit_of_measurement: "°C"
            state: >
            {% set ref_voltage = 10 %}
            {% set ref_resistor = 10000 %}
            {% set b_constant = 3950 %}
            {% set ref_temp = 25 %}
            {% set ref_resistence = 10000 %}
            {% set resistence = (
                (states.sensor.shellyplus1_b8d61a8aaa94_voltmeter.state|float) * ref_resistor)
                /
                (ref_voltage - (states.sensor.shellyplus1_b8d61a8aaa94_voltmeter.state|float))
            %}
            {% set t0 = ref_temp + 273.15 %}
            {% set a = (1/t0)-(1/b_constant)*log(ref_resistence) %}
            {% set b = (1/b_constant) %}
            {% set c = 0 %}
            {% set lr = log(resistence) %}
            {% set v = a + (b * lr) + (c * lr * lr * lr) %}
            {% set temp = (1 / v) - 273.15 %}
            { temp|round(2) }
   

4. Then it was possible to configure a thermostat in Home Assistant based upon the temperature calculation, which turns the Shelly switch on and off based upon the temperature.

    climate:
    - platform: generic_thermostat
        unique_id: bathroom_floor
        name: Bathroom Floor
        heater: switch.shellyplus1_b8d61a8aaa94_switch_0
        ac_mode: false
        target_sensor: sensor.bathroom_floor_temperature
   

5. And finally I’ve reached nirvana when I can add Home Assistant automations to turn the thermostat heating on and off at the right times.

Now, the manual time adjustments are a thing of the past. My thermostat knows what time it is, all year round… and actually technically it doesn’t care because Home Assistant knows the same thing and is remotely controlling it.

After being annoyed for a bit, I realized this was a great time to learn how to bisect the linux kernel, find the problem, and either report it upstream, or, patch it out of my kernel! I thought this would be useful to someone else in the future, so here we are.

Step #1: Clone the Kernel; I grabbed Linus’ tree from https://github.com/torvalds/linux with git clone git@github.com:torvalds/linux.git

Step #2: Start a bisect.

If you’re not familiar with a bisect, it’s a process by which you tell git, “this commit was fine”, and “this commit was broken”, and it will help you test the commits in-between to find the one that introduced the problem.

You start this by running git bisect start, and then you provide a tag or commit ID for the good and the bad kernel with git bisect good ... and git bisect bad ....

I knew my issue didn’t occur on the 5.15 kernel series, but did start with my NixOS upgrade to 6.1. But I didn’t know precisely where, so I aimed a little broader… I figured an extra test or two would be better than missing the problem. 😬

git bisect start
git bisect good v5.15
git bisect bad master 

Step #3: Replace your kernel with that version

In an ideal world, I would have been able to test this in a VM. But it was a graphics problem with my video card and connected monitors, so I went straight for testing this on my desktop to ensure it was easy to reproduce and accurate.

Testing a mid-release kernel with NixOS is pretty easy! All you have to do is override your kernel package, and NixOS will handle building it for you… here’s an example from my bisect:

boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux_6_2.override { # (#4)
  argsOverride = rec {
    src = pkgs.fetchFromGitHub {
      owner = "torvalds";
      repo = "linux";
      # (#1) -> put the bisect revision here
      rev = "7484a5bc153e81a1740c06ce037fd55b7638335c";
      # (#2) -> clear the sha; run a build, get the sha, populate the sha
      sha256 = "sha256-nr7CbJO6kQiJHJIh7vypDjmUJ5LA9v9VDz6ayzBh7nI=";
    };
    dontStrip = true;
    # (#3) `head Makefile` from the kernel and put the right version numbers here
    version = "6.2.0";
    modDirVersion = "6.2.0-rc2";
  };
});

Getting this defined requires a couple intermediate steps…

Step #3.1 – put the version that git bisect asked me to test in (#1)

Step #3.2 – clear out sha256

Step #3.3 – run a nixos-rebuild boot

Step #3.4 – grab the sha256 and put it into the sha256 field (#2)

Step #3.5 – make sure the major version matches at (#3) and (#4)

Then run nixos-rebuild boot.

Step #4: Test!

Reboot into the new kernel, and test whatever is broken. For me I was able to set up a simple test protocol: xset dpms force off to blank my screens, wait 30 seconds, and then wake them. If my kernel panicked then it was a fail.

Step #5: Repeat the bisect

Go into the linux source tree and run git bisect good or git bisect bad depending on whether the test succeeded. Return to step #3.

Step #6: Revert it!

For my case, I eventually found a single commit that introduced the problem, and I was able to revert it from my local kernel. This involves leaving a kernel patch in my NixOS config like this:

  boot.kernelPatches = [
    {
      patch = ./revert-bb2ff6c27b.patch;
      name = "revert-bb2ff6c27b";
    }
  ];

Obviously this isn’t a great long-term solution; it gets my desktop stable right now and I’m OK with that. The best follow-up would be to create a detailed bug report for the impacted kernel module, the steps to reproduce, and the regressing commit. The next best might just be to check occasionally if I can disable the revert and if things still work.

This research was originally published on Google Docs (ABTraceTogether Android Background Capability Analysis) on November 14, 2020, with a short summary described on Twitter. The tweets and research notes are archived and reproduced here with some formatting edits.

Executive Summary

(Reproduced from Twitter)

Today I continued testing ABTraceTogether, focusing on the exposures collected on an Android. There is some defect. It failed to record an Android in close proximity. I don’t know whether it’s an flaw related to one of my devices, or general problem.

Throughout my testing, I had an iPhone and two Android devices in physical proximity. The iPhone was recorded by the Android pretty consistently. Overnight, a second iPhone nearby was also recorded. A good start!

But the Android-to-Android data was so sparse that I revised my test protocol a few times to try to figure it out. No overnight data recorded. 8 hours of the phones snuggled, and they didn’t see each other at all?

I guessed that maybe ABTraceTogether detected that they had registered with the same phone number, and that blocked recording. I didn’t see that in the source code, but, maybe? I borrowed a friend’s spare phone number (thanks Mike). No difference in another hour of testing.

I found that one of my Android devices had automatically configured the ABTraceTogether app to “Intelligent Control” battery optimization. Well, that could have some impact, right? So, I tweaked that to “Don’t Optimize”. Another hour of testing, and no exposures recorded.

I added a third Android device to the test, and I started getting a few Android-to-Android traces. So; maybe one Android device just didn’t work? Despite saying “ABTraceTogether is scanning to keep you safe!” all the time?

Another theory… Android-to-Android only stores data a few times for an exposure (3), and then ignores new records. But… only Android/Android, no other combo? And that won’t allow exposures to age out of the DB accurately. And… the Open Source repo doesn’t do that.

So, I’m stumped for now. I have a few old Android devices; maybe I’ll dust off more and see if this is a single device problem. Unluckily it’s my primary day-to-day phone. 😰

Which I guess is still an interesting result; ABTraceTogether can appear to be working, and not work.

Overview

(Special thanks to my collaborator and iPhone contributor: Amanda Fenniak 😘, and my collaborator and phone number contributor: Mike Roest 😎)

On November 11th, an analysis of ABTraceTogether’s background contact tracing capabilities was conducted with a specific focus on iPhone interactions (https://mathieu.fenniak.net/abtracetogether-iphone-background-capability-analysis/). This allowed investigation of all data that was collected by an iPhone, but not the data that was collected by an Android phone in proximity to other phones.

In this document, testing and analysis is performed on the data collected from an Android phone using ABTraceTogether.

Test Protocol

• ABTraceTogether was installed on two Android phones, and one iPhone
  • ABTraceTogether was run, other applications were opened so that ABTraceTogether was in the “background”, and then the phones were locked
• All phones were kept within close proximity for a 12 hour period; less than 1 meter separation
• Data collected from one Android was analyzed
  • See Appendix for more detailed information on the data collection procedure

Test Details

Dates and times are in Mountain Standard Time

• iPhone #1:
  • iPhone 11 Pro
  • iOS 14.2
  • ABTraceTogether version 1.4.0 installed from the Apple App Store; registered with phone number #1
  • iPhone #1 was a nearby phone during the test, but it is not the iPhone that was specifically kept in physical proximity; it would have varied between 3m and 20m distance throughout the test
• iPhone #2:
  • iPhone 6s
  • iOS 14.1
  • ABTraceTogether version 1.4.0 installed from the Apple App Store; registered with phone number #2
  • Kept within 1m of Android #2 throughout test
• Android #1:
  • OnePlus 7T
  • OxygenOS (Android) 10.0.14.HD65AA
  • ABTraceTogether version 1.4.0 installed from the Google Play Store; registered with phone number #3
  • Kept within 1m of Android #2 throughout test
• Android #2:
  • Google Nexus 5
  • LineageOS 17.1; Android 10
  • ABTraceTogether version 1.4.0 installed from the Google Play Store
    • Originally registered with phone number #3
    • Registered with phone number #4 in test #2 and later
• Android #3:
  • Introduced only in later tests
  • Amazon Fire HD 10 (9th generation)
  • FireOS 7.3.1.6 (Android)
  • ABTraceTogether version 1.40 installed from the Google Play Store; registered with phone number #3
• 2020-11-13
  • 6:52pm
    • Android #2, Android #1, and iPhone #2 have been placed in close physical proximity, had the ABTraceTogether app run, and then put into the background by hitting the home button within the past 2 minutes.
    • All phones are running on battery power.
  • ~10:00pm
    • iPhone #1 & Android #1 are connected to power
• 2020-11-14
  • 5:30am
    • iPhone #1 & Android #1 are disconnected from power
  • 6:27am
    • Android #2 is connected to power
    • Android #2’s ABTraceTogether database is cloned for analysis
• Test period #1: 2020-11-13 6:52pm -> 2020-11-14 6:27am
• Following the completion of the test, data extraction, and analysis above, a second test was conducted to rule out a possible explanation for unexpected results
  • Possible flaw in testing: Android #1 and Android #2 were registered to ABTraceTogether with the same phone number
  • ABTraceTogether was deleted from Android #2
  • ABTraceTogether was installed in Android #2, and re-registered with a fourth phone number, distinct from iPhone #1, #2, and Android #1
  • All devices were charged to >90% battery before test period #2 began
• 2020-11-14
  • 10:49am
    • Test phones Android #1, Android #2, and iPhone #2 are placed in close proximity
    • ABTraceTogether is run in the foreground on the three test phones
    • Continual interactions are required on Android devices to keep screen & app active
  • 11:00am
    • ABTraceTogether is sent to the background on the three test phones by hitting the home button, and locking the phones
  • 12:07pm
    • End of test period; data to be collected at 12:55pm
• Test period #2: 2020-11-14 10:49am -> 12:07pm
• Following the completion of the test, data extraction, and analysis above, a third test was conducted to rule out a possible explanation for unexpected results
  • Android #1 was explicitly configured so that ABTraceTogether app was set to “Don’t Optimize” for battery options
  • Android #3 was introduced into the test environment
• 2020-11-14
  • 1:46pm
    • Android #3: ABTraceTogether installed and registered
  • 1:48pm
    • ABTraceTogether is sent to the background on the four test device sby hitting the home button, and locking the phones
  • 2:34pm
    • End of test period; data collected immediately
• Test period #3: 2020-11-14 1:48pm -> 2:34pm

Data & Analysis

• Raw data from both data sets is available in the below spreadsheet
  • Original analysis format was Google Sheets
  • An archive of the same spreadsheet is available in a Microsoft Excel and OpenDocument format.
• ABTraceTogether Android collects all exposures in a table named record_table with the following fields:
  • id INTEGER PRIMARY KEY
    • Analysis: sequentially incrementing primary key of the record
  • timestamp INTEGER
    • Analysis: epoch-milliseconds; that is, a timestamp in UTC represented by the number of milliseconds past since midnight 1970-01-01
  • v INTEGER
    • Analysis: always contains the value 2
  • msg VARCHAR
    • Data: 61 bytes of BASE64 encoded data; repeated multiple times for each exposure
    • Analysis: This is the exposure ID of the target device; there seems to be no confidential information in the base64 decoded data
    • The data published in this spreadsheet has been trimmed to the first 30 characters in the unlikely event that confidential or sensitive information is contained in the token
  • org TEXT
    • Data: String “CA_AB”
    • Analysis: Possible future expansion to multiple regions, or integration with other OpenTrace-supported applications
  • modelC VARCHAR
    • Data: “iPhone” or “Android”
    • Analysis: In the Bluetooth protocol, “central” devices are host devices, and “peripheral” devices connect to central devices.  The “C” suffix indicates that this suggests which type of device was the “central” device in this exposure.
  • modelP VARCHAR
    • Data: “iPhone” or “Android”
    • Analysis: As per field modelC, this is believed to be the model of the peripheral device.
  • rssi INTEGER
    • Analysis: measurement of the received signal strength indicator, likely in dBm; the values range from -12 to -100
    • txPower INTEGER
    • Data: NULL

Test Period #1

• Data collected
  • 339 exposures were collected
  • Msg: avuP1VvHjDRWCuoVFoAAdTOW2TACdp
    • 58 exposures
    • iPhone central device, Android #2 is peripheral
    • Exposures recorded between 7:07pm and 5:51am
    • Time between exposure:
      • Average: 11 minutes
      • Maximum: 67 minutes
    • RSSI: -100 to -58
  • Msg: KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA
    • 67 exposures
    • iPhone central device, Android #2 is peripheral
    • Exposures recorded between 6:57pm and 5:51am
    • Time between exposure:
      • Average: 9 minutes
      • Maximum: 26 minutes
    • RSSI: -70 to -15
  • Msg: odarSPFz7coBgKtf0mmLFRw7ODGt5O
    • 131 exposures
    • Android #2 central device, iPhone ? is peripheral
    • Exposures recorded between 8:56pm and 5:51am
    • Time between exposure:
      • Average: 4 minutes
      • Maximum: 18 minutes
    • RSSI: -47 to -18
  • Msg: TgT0cEg2mZm6YMd5ejPMv7KRpDVzoK
    • 42 exposures
    • Android #2 is central device, iPhone ? is peripheral
    • Exposures recorded between 9:10pm and 5:51am
    • Time between exposure:
      • Average: 12 minutes
      • Maximum: 71 minutes
    • RSSI: -93 to -60
  • Msg: U00y/jHqnbDh7+wIkl4zCY+D1C99e3
    • 33 exposures
    • Android #2 is central device, iPhone ? is peripheral
    • Exposures recorded between 6:35pm and 8:52pm; includes time before the test began
    • Time between exposure:
      • Average: 4 minutes
      • Maximum: 15 minutes
    • RSSI: -96 to -12
  • Msg: wOPJa8pn/iB8wN6Gawd6olHKeHREfy
    • 1 exposure
    • Android - Android; cannot distinguish the central or peripheral
    • Occurred at 6:49pm; this is before all phones were in the background configuration at the start of the test
    • RSSI: -89
  • Msg: IHGhpOV5XP0sqGfXVlKIaEjzQm+j8z
    • 2 exposures
    • Android - Android; cannot distinguish the central or peripheral
    • All exposures occurred at 6:59pm
    • RSSI: -68 to -47
  • Msg: jhWnG4TnzYe1XOqVej37NWKx2QTgK+
    • 5 exposures
    • Android #2 is central device, iPhone ? is peripheral
    • Exposures recorded between 6:35pm and 8:45pm; includes time before the test began
    • Time between exposure:
      • Average: 32 minutes
      • Maximum: 96 minutes
    • RSSI: -91 to -81
• Analysis
  • On iPhone / Android interactions:
    • TgT0cEg2mZm6YMd5ejPMv7KRpDVzoK & avuP1VvHjDRWCuoVFoAAdTOW2TACdp
      • One of these messages was Android central, other iPhone central
      • Messages had similar start & end times
        • 7:07pm to 5:51am
        • 9:10pm to 5:51am
      • Messages had similar RSSI ranges
        • -93 to -60
        • -100 to -58
      • Given the long-term overnight exposure, this is likely to be either iPhone #1 or iPhone #2 communicating with Android #2
      • Considering the lower RSSI values, which would correlate with greater physical distance between the phones, it is plausible that this is iPhone #1 to Android #2
    • KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA & odarSPFz7coBgKtf0mmLFRw7ODGt5O
      • One of these messages was Android central, other iPhone central
      • Messages had similar start & end times
        • 6:57pm to 5:51am
        • 8:56pm to 5:51am
      • Messages had similar RSSI ranges
        • -70 to -15
        • -47 to -18
      • Given the long-term overnight exposure, this is likely to be either iPhone #1 or iPhone #2 communicating with Android #2
      • Considering the higher RSSI values, which would correlate with shorter physical distance between the phones, it is plausible that this is iPhone #2 to Android #2
    • Note: Although the above message pairs look like they may be mismatched based upon the timestamps, the RSSI signal strength matches better, and all the exposures that started in the 7pm timeframe were iPhone central devices; these messages need to be paired central/peripheral, which they can’t be if the pairing was adjusted.  Regardless the results of the analysis are not impacted by the specific messages that identify a device.
    • U00y/jHqnbDh7+wIkl4zCY+D1C99e3
      • Unidentified iPhone peripheral between 6:35pm and 8:52pm
      • RSSI values were surprisingly strong for an unidentified phone
      • 8:52pm roughly corresponds to Android #2 being moved within the home test environment; my best explanation is that this was a neighbour’s iPhone and we were in close physical proximity, but this explanation seems questionable
    • jhWnG4TnzYe1XOqVej37NWKx2QTgK+
      • Unidentified iPhone peripheral between 6:35pm and 8:45pm
      • RSSI values are weaker than U00y
      • If it weren’t for this device also showing an Android central, it would be likely that this is the inverse relationship of the U00y message above; considering both show the Android central device, this is unexplained
  • On Android / Android interactions:
    • Messages wOPJa8pn/iB8wN6Gawd6olHKeHREfy & IHGhpOV5XP0sqGfXVlKIaEjzQm+j8z
      • Both occurred in the same time window, 6:49pm - 6:59pm
      • Only 3 exposures occurred, and in a small time period
      • RSSI values are not strongly correlated between the exposures, but the number of exposures is so low that this can’t be correlated well
  • Strong interactions between iPhone and Android was recorded throughout the experiment, with a high probability that a 10 minute physical exposure would result in a recorded ABTraceTogether exposure
  • Android-Android interactions appear to be effectively absent from the dataset; despite 12 hours of physical exposure, only 10 minutes of recording between the devices was recorded
    • Part of this exposure recording is outside of the “background” time window in the test, as well; exposures occurred at 6:49pm while the test was being configured and either device may run the application in the foreground

Test Period #2

• Data collected
  • 52 exposures were collected
    • 23 exposures were within the test time period; the remaining exposures occurred and were recorded while all devices were being recharged between tests
    • Non-eligible exposures are coloured grey in the spreadsheet
    • Test period #2: 2020-11-14 10:49am -> 12:07pm
  • Msg: h5Cjy32U67/0fnQeeq05K+NStEElVG
    • 19 exposures
    • Android #2 central device, iPhone #2 is peripheral
    • Exposures recorded between 10:52am and 11:55am
    • Time between exposure:
      • Average: 3 minutes
      • Maximum: 10 minutes
    • RSSI: -47 to -28
  • Msg: KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA
    • 9 exposures
    • iPhone central device, Android #2 is peripheral
    • Exposures recorded between 10:50am and 11:49am
    • Time between exposure:
      • Average: 7 minutes
      • Maximum: 11 minutes
    • RSSI: -77 to -62
  • Analysis
    • On iPhone / Android interactions:
      • KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA & h5Cjy32U67/0fnQeeq05K+NStEElVG
        • One of these messages was Android central, other iPhone central
        • Messages had similar start & end times
          • 10:52am and 11:55am
          • 10:50am and 11:49am
        • When iPhone is central device, the msg code KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA is the same as the code previously identified as iPhone #2; this is consistent with iPhone #2 being the only iPhone present during test period #2
    • On Android / Android interactions:
      • No Android/Android interactions were recorded during the test period, including both the foreground and background execution time

Test Period #3

• Data collected
  • An additional 8 exposures were collected on top of Test Period #2
    • Non-eligible exposures from outside of this test period are coloured grey in the spreadsheet
    • Test period #3: 2020-11-14 1:48pm -> 2:34pm
  • Msg: mIT1/+vA/XhDRFt51WqXtw0pFOy0bK
    • 3 exposures
    • Android - Android; cannot distinguish the central or peripheral
    • Exposures recorded between 1:55pm and 1:59pm
    • Average: 1 minute time between exposure; however very few exposures are captured in this time window
    • RSSI: -61 to -61
• Msg: KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA
  • 5 exposures
  • iPhone central device, Android #2 is peripheral
  • Exposures recorded between 1:55pm and 2:28pm
  • Average: 8 minutes time between exposure; however very few exposures are captured in this time window
  • RSSI: -53 to -50
• Analysis
  • On iPhone / Android interactions:
    • KPZfd0ip8lGWMSUA/IdTeB/ZBZfpGA
      • Consistent with Test Period #2, this appears to be iPhone #2 being recorded from Android #2
  • On Android / Android interactions:
    • A new Android/Android interaction was recorded during the test period
    • Data is inconclusive as to whether Android #2 was the central or peripheral device
    • Data is inconclusive as to whether the other partner in the device was Android #1 or Android #3
    • Regardless of whether Android #1 or #3 was the pair device, the absence of an additional signal indicates that one of the two Android devices did not record within the 46 minute test window

Multi-Test Analysis

• ABTraceTogether failed to record exposures from a nearby Android device during the initial 12 hour test period.
• During a shorter 78 minute Android-Android test, ABTraceTogether failed to record exposures from the Android device.
  • Android #2 was reconfigured to a distinct phone number from Android #1.
  • This change appears to have had no impact.
• During a third 46 minute Android-Android test, one of two Android devices were detected.
  • Android #1 had been reconfigured to disable battery optimization on ABTraceTogether.
  • Android #3 was introduced into the experiment.
  • One of these two changes caused exposures to be recorded, but only to one device.
  • Zero Android-to-Android captures would have indicated that Android #2 is unable to receive Android exposures correctly.
  • Two Android-to-Android captures would have indicated that the battery optimization on Android #1 had been successful, and Android #3 was working as expected.
  • One Android-to-Android capture could theoretically be either change, but, for it to be the battery optimization of Android #1 would be the more complex explanation, as it would indicate Android #3 was not working properly for a new, unexplained reason.
  • It isn’t clear why not all test devices are being recognized.

Conclusion

From the perspective of an Android device in close proximity to iPhone devices, ABTraceTogether effectively records the presence of an iPhone.  A ten minute physical exposure would very likely result in a recorded ABTraceTogether exposure.

There appears to be a condition in which Android-to-Android exposure detection will fail to be recorded during continuous exposure.  However, experimentation has only been able to show the absence of expected functionality, but not the cause of this failure.

ABTraceTogether displays a message on Android continuously that reads “ABTraceTogether is scanning to keep you safe!” and “Restart phone if this notification disappears.”  The condition in which Android/Android exposure was not functioning occurred despite this comforting message on all Android devices.

Follow-Up Work

The results of analyzing Android recorded data is not consistent with observed over-the-air Bluetooth traffic generated from ABTraceTogether on an Android.  Further theories on the possible fault with Android/Android tracing are welcome to be contributed; however future tests that are required are not yet clear.

The experimental testing occurring with phones in an uncontrolled environment has resulted in aberrant, unexpected exposures.  A radio frequency isolating environment would be a great addition to the test configuration, allowing for the removal of all unexpected exposures, and simplifying the analysis of the recorded data.  A short experiment with three phones in a microwave, which is a 2.4GHz isolating container, resulted in zero communication between the three devices.  Further experimentation with an environment that isolates the devices but does not interfere with their communication would be warranted.

Appendix

Collecting Android Exposure Database

ABTraceTogether on an Android stores data in an App-specific file location, which is not typically readable from an external tool.  In order to collect an Android database, it was necessary to configure a device with root-level administrative access.

A Nexus 5 device was chosen from a library of unused Android phones.  The following steps were required to get the device available to use ABTraceTogether, and be able to access the recording database:

• Bootloader unlocked – this allowed the installation of a custom recovery tool
• Installation of TWRP – this is a flexible recovery operating system for Android devices
• Repartitioning of Nexus 5 system & data partitions
  • Additional storage was required on the system partition
  • The Nexus 5 system partition was expanded by 1 GB, by reallocating the data partition space
• Installation of LineageOS 17.1 built for Nexus 5 via TWRP
• Installation of Magisk (https://github.com/topjohnwu/Magisk) via TWRP
• Installation of OpenGApps via TWRP
  • OpenGApps was found to be required to supply Google services, which were required for the successful registration of the ABTraceTogether application
  • “Nano” installation was used
• Android system setup
• Configuration of Developer Tools via Android Settings
• Remote root access by enabling USB debugging and root debugging via Developer Tools
• Installation of ABTraceTogether via Google Play Store
• Registration of ABTraceTogether

Following all that messy work, retrieving the ABTraceTogether exposure database was performed by USB debugging the Android device, and using the “adb” tool to extract the DB via the command: adb pull /data/data/ca.albertahealthservices.contacttracing/databases/record_database .

The database extracted was a SQLite 3.x database, user version 1, last written using SQLite version 3022000.

This research was originally published on Google Docs (ABTraceTogether iPhone Background Capability Analysis) on November 11, 2020, with a short summary described on Twitter. The tweets and research notes are archived and reproduced here with some formatting edits.

Executive Summary

(Reproduced from Twitter)

On the weekend, I tweeted about how ABTraceTogether doesn’t work correctly between iPhones. Steve Buick, press secretary to the Minister of Health of Alberta, replied on Twitter and told me that I was wrong. So, I tested it.

It doesn’t work in the background, iPhone/iPhone. Doesn’t work consistently iPhone/Android, either.

iPhone/iPhone: In over 12 hours of close exposure between two iPhones with ABTraceTogether in the background, zero records of exposure were recorded. After bringing the app into the foreground on one iPhone, exposure records began being collected immediately.

Android/iPhone: While some exposure was recorded while both devices were in the background, it did not occur consistently w/ physical exposure. One hour of physical exposure is more than sufficient to contract COVID-19, but did not trigger ABTraceTogether’s exposure recording.

Overview

(Special thanks to my collaborator and iPhone contributor: Amanda Fenniak 😘)

ABTraceTogether is an application distributed by the Government of Alberta for the purpose of aiding Alberta Health Services contact tracers in reaching out to potential subjects of COVID-19 exposure. ABTraceTogether is based off a Singapore application called TraceTogether, which has been developed based upon the OpenTrace protocol for bluetooth exposure notification.

Concerns have been raised over the effectiveness of ABTraceTogether at detecting exposure when the application is running in the “background” on an iOS device, such as an iPhone.

In this document, a real-world test of the exposure notification capability of an iPhone is performed and analyzed.

Background

When ABTraceTogether was launched, it was clearly communicated that the application did not work in the “background”. For example, Global News reported May 22, 2020, that “An update to Apple’s iOS will allow the Alberta government’s ABTraceTogether app to operate in the background and when the iPhone is locked. However, the ABTraceTogether app is not using the software update yet.” (https://globalnews.ca/news/6969842/ios-update-abtracetogether-app-apple-alberta-coronavirus/)

On September 24 2020, an update to ABTraceTogether was launched which included in the release notes “added capability to do contact tracing while the application is in the background” (https://apps.apple.com/ca/app/abtracetogether/id1508213665).

However, doubts arose that this was an accurate description of the capabilities. The Singaporean application TraceTogether also had an update to allow background utilization, but it included the notes that “Except for very limited circumstances, iOS apps are still unable to exchange signals with other iOS apps if both apps are in the background.” (https://support.tracetogether.gov.sg/hc/en-sg/articles/360050556334-I-see-COVID-19-Exposure-Notifications-in-my-phone-s-operating-system-Do-I-need-to-turn-it-on-for-my-TraceTogether-App-to-work-in-the-background-)

Based upon a review of public literature, it seems unlikely that ABTraceTogether can work correctly as expected. Reference literature:

• https://medium.com/kinandcartacreated/why-bespoke-contact-tracing-apps-dont-work-so-well-on-ios-df0dabd95d42
• https://github.com/opentrace-community/opentrace-ios/issues/4

It seems that there is sufficient reason to believe that Apple enforces app restrictions that would prevent ABTraceTogether from performing the type of background scanning that is required for OpenTrace bluetooth protocol to work correctly.

Experimental testing will resolve whether the ABTraceTogether application has capabilities beyond those documented in the TraceTogether application.

Test Protocol

• ABTraceTogether was installed on two iPhones
• The iPhones were exposed to each other with ABTraceTogether in the “foreground” for a period of time
  • Foreground: Both phones were left unlocked, with the ABTraceTogether application running
• Other apps on the phones were used, to make ABTraceTogether run in the “background” for an extended period of time
  • ABTraceTogether was not closed via the App Switcher
• Data collected from one of the phones was analyzed
  • See Appendix for more detailed information on the data collection procedure

Test Details

• iPhone #1:
  • iPhone 11 Pro
  • iOS 14.2
  • ABTraceTogether version 1.4.0 installed from the Apple App Store
• iPhone #2:
  • iPhone 6s
  • iOS 14.1
  • ABTraceTogether version 1.4.0 installed from the Apple App Store
• Android #1:
  • OnePlus 7T
  • OxygenOS (Android) 10.0.14.HD65AA
  • ABTraceTogether version 1.4.0 installed from the Google Play Store
• After the installation, the two iPhones were left with ABTraceTogether running in the foreground between 7:22am and 8:00am on November 10th
• Other applications were used on both iPhones; the two iPhones were physically within 2 meters of each other throughout November 10th between 8am and 6pm
• Android #1 entered physical proximity of both iPhones around 8am for less than 10 minutes
• Android #1 entered physical proximity of both iPhones around 12pm for about 1 hour
• Android #1 entered physical proximity of both iPhones around 5pm for about 1 hour
• Between 6pm and the next morning around 7am, iPhone #1 and iPhone #2 were physically separated
• The next morning around 6:30am, iPhone #1, iPhone #2, and Android #1 were all physically reunited
• Test data was collected from iPhone #2 at 6:49am by performing a backup to a macOS device, extracting the SQLite tracer.sqlite DB from the application, and exporting all data to a spreadsheet for analysis and reformatting
• Between 7:39am and 7:50am (Nov 11) iPhone #1 started running ABTraceTogether in the foreground
• A second test data set was collected from iPhone #2 at 7:50am, following the same procedure as the first test data set.

Data & Analysis

• Raw data from both data sets is was imported into a spreadsheet for analysis
  • Original analysis format was Google Sheets
  • An archive of the same spreadsheet is available in a Microsoft Excel and OpenDocument format.
• ABTraceTogether collects all exposures in a table named ZENCOUNTER with the following fields:
  • Z_PK INTEGER PRIMARY KEY
    • Analysis: sequentially incrementing primary key of the record
  • Z_ENT INTEGER
    • Analysis: always contains the value 1
  • Z_OPT INTEGER
    • Analysis: always contains the value 1
  • ZV INTEGER
    • Analysis: Contains the value 1 for an initial record “Scanning started”, and contains the value 2 for all exposure records
  • ZRSSI FLOAT
    • Analysis: measurement of the received signal strength indicator, likely in dBm; the values range from -49 to -101
  • ZTIMESTAMP TIMESTAMP
    • Analysis: Cocoa date-time value, which is measured in seconds since midnight, January 1, 2001, in UTC
  • ZTXPOWER FLOAT
    • Data: NULL, 8, 12, or -1
    • Analysis: Not clear
  • ZMODELC VARCHAR
    • Data: “iPhone” or “Android”
    • Analysis: In the Bluetooth protocol, “central” devices are host devices, and “peripheral” devices connect to central devices. The “C” suffix indicates that this suggests which type of device was the “central” device in this exposure.
  • ZMODELP VARCHAR
    • Data: “iPhone” or “Android”
    • Analysis: As per field ZMODELC, this is believed to be the model of the peripheral device.
  • ZMSG VARCHAR
    • Data: 61 bytes of BASE64 encoded data; repeated multiple times for each exposure
    • Analysis: This is the exposure ID of the target device; there seems to be no confidential information in the base64 decoded data
    • The data published in this spreadsheet has been trimmed to the first 30 characters in the unlikely event that confidential or sensitive information is contained in the token
  • ZORG VARCHAR
    • Data: String “CA_AB”
    • Analysis: Possible future expansion to multiple regions, or integration with other OpenTrace-supported applications
• Data collected:
  • 47 exposures were collected between 7:22am Nov 10 and 6:49am Nov 11
  • Three unique ZMSGs were found between iPhone devices
    • One ZMSG occurred 28 times
    • One ZMSG occurred 13 times
    • One ZMSG occurred 2 times
  • Two unique ZMSGs were found between iPhone and Android devices
    • One ZMSG occurred 1 time
    • One ZMSG occurred 2 times
  • An additional 16 exposures were collected between 6:49am Nov 11 and 7:50am Nov 11
• Analysis:
  • The three iPhone exposures are believed to be:
    • iPhone #1 -> iPhone #2
    • iPhone #2 -> iPhone #2
    • An unidentified iPhone -> iPhone #2 (2 times
      • These rarer iPhone to iPhone connections had a much lower RSSI, -101 and -98, as compared to the RSSIs in the range of -68 between the more common exposures
  • The two Android exposures are believed to be:
    • Android #1 -> iPhone #2
    • An unidentified Android -> iPhone #2
  • In test data set #1, all iPhone to iPhone exposures occurred while iPhone #2 had the ABTraceTogether application running in the foreground
    • 44 iPhone/iPhone exposures occurred in the 30 minutes that the app was running in the foreground on both devices
    • 0 iPhone/iPhone exposures occurred in the 23 hours that the app was running in the background on both devices
  • In test data set #2, 16 additional exposures occurred
    • 2 iPhone/iPhone exposures occurred while both devices were believed to have the app running in the background
    • 14 exposures occurred in the 10 minutes that the app was running in the foreground on at least one iPhone device
  • Android/iPhone exposures also occurred rarely considering the proximity of the devices; during two 1hr windows of exposure on Nov 10 no exposures were recorded on the iPhone

Conclusion

ABTraceTogether does not effectively record iPhone-to-iPhone exposures while both devices have ABTraceTogether running in the background. In over 12 hours of close exposure between two iPhones with ABTraceTogether in the background, zero records of exposure were recorded. After bringing the app into the foreground on an iPhone, exposure records began being collected immediately.

Android/iPhone exposure notification is also quite inconsistent. While some exposure was recorded while both devices were in the background, it did not occur consistently at the same time as physical prolonged exposure occurred. One hour of physical exposure is more than sufficient to contract COVID-19, but did not trigger ABTraceTogether’s exposure recording.

Follow-Up Work

iPhone/iPhone exposure notification was collected once when neither device is believed to have the app in the foreground. It is likely a testing error occurred, as this abnormal finding is inconsistent with many hours of previous data collection. Continued data collection will be performed to identify the likelihood of this occurring again.

• Update 2020-11-11 @ 11:05am – Twitter user @ZiadFazel has offered a theory that this exposure may have occurred because one iPhone was plugged in, allowing the power-saving features of Apple’s platform to be disabled. It is true that one phone was plugged in; iPhone #2 was prep’d for another data backup during this time window. This theory is unproven, but a plausible explanation for the background exposure in this small time window.
• Update 2011-11-12 @ 6:00am – An analysis of an additional 24 hours of exposure data from iPhone #2 has shown results consistent with the first 24 hours of data; no iPhone to iPhone exposures were recorded despite iPhone #1 and #2 being in close proximity, with the apps in the background. iPhone #2 to Android #1 exposures were observed in two roughly 10 minute windows, 12 hours apart, despite ongoing daily exposure, which continues to indicate that iPhone-Android exposure is also not consistent.

Android/Android exposure notification could not be tested with the data source used in this testing. Future experimentation on the effectiveness of Android exposure detection is recommended, but a rooted Android device is likely required to collect an Android exposure database.

Appendix

Collecting iPhone Exposure Database

Using the software iMazing, a full iPhone backup was performed. Accessing tracer.sqlite from the application is trivial, as demonstrated in the below screenshot.

As with most Core Data based iOS applications, the data is stored in a SQLite database. This specific database was a SQLite 3.x database, last written using SQLite version 3032003.

Data was exported from the SQLite database for analysis in Google Sheets by exporting the ZENCOUNTER table to CSV:

What HTTP status code does she get back from her response?

Well, it’s not going to be 200 OK, because it wasn’t OK with the server. The server couldn’t find the article that the client requested, because it won’t be published for another 43 years.

“Couldn’t find the article” sounds like a 404 Not Found status code. OK, very reasonable choice.

But, “The server couldn’t find the article” raises a bit of a doubt. A 404 is part of the 4xx-series status codes, which are all for client errors. Was this a client error, if it was the server’s fault for not finding the article? Shouldn’t she be getting 5xx Not Found?

HTTP error status codes fall into two broad categories; 4xx (400 - 499) and 5xx (500 - 599). Understanding these categories is more important than understanding the specific error codes, because the categories communicate a very important piece of information. 4xx status codes represent a problem with the HTTP request, whereas 5xx status codes represent a problem with the HTTP server.

However, the line can be blurry between these two categories, if you just consider them the difference between “HTTP request” or “HTTP server”. If resources can be created in the future, and they can, then the exact same HTTP request that was 404 today could be a 200 tomorrow. So, is it really a problem with the request?

To address this confusion, I like to add an additional rule to clarify between 4xx and 5xx. When servicing an HTTP request, was it possible to give a 200 OK response? Let’s say it was a bug-free application server, or possibly even a human being manually processing HTTP requests. If it is possible, given perfect software or infinite resources or a brilliant mind, to process the request, but an error occurred, then it should be a 5xx code response. On the other hand, if no amount of perfection, resources, or brilliance would have been capable of processing that request, then it is a 4xx error.

Maybe our time-traveler should get back a 505 HTTP Version Not Supported response, because the server could never have understood her attempt to use the HTTP/9.7 protocol. But other than that, she should get a good old 404 Not Found; even with bug-free flawless server software, it was impossible to process her request. A 4xx error code suggests that the HTTP client needs to do something differently; like time-travel forward 43 years.

Because documentation is boring, most software developers will quickly turn towards documentation tools. It’ll be so much faster to write documentation if we can just build or apply some software to do it, and building a software tool is much more fun than just sitting down and writing.

Documentation tools have advantages over hand-written documentation: they’re easier to keep up-to-date, they can produce output much faster, and they can provide much better coverage with less effort. But, before you go looking at API documentation tools, take a step back and figure out some high-level details about what your documentation looks like.

Who is authoring, editing, and maintaining the documentation project, now and in the future?

This is the most important point to consider before you choose a documentation tool. Is your documentation going to be written by a technical writer, a developer, or a product owner? Is it going to be authored and edited in-house, are you going to outsource to someone with more expertise?

If you have a technical writer involved in the documentation of your API, you need to consider the environment, systems, and processes that they’re used to working with. Maybe they’re not comfortable writing Javadoc comments; knowing that will definitely impact upon tool selection.

Example: A first-draft of documentation comes from the API developer, because they know what the API can do and how to describe it to other developers. After that, it’s usually rewritten and then maintained by a technical writer; that will influence my documentation tool search towards tools that a writer can work with.

How and where is it going to be published?

Is your documentation output part of your product, part of your marketing website, part of your support team’s website, or something completely independent? This will vary dramatically between smaller organizations and larger ones, but there are important considerations that need to be figured out.

If you’re a development shop that wants to do continuous deployment, do you want a technical writer to be pushing builds out when they update the documentation project?

Example: In an organization with separated web responsibilities between marketing and product, documentation probably belongs hosted with the marketing web site. It doesn’t have the same availability and compliance requirements as product changes.

When is it going to be updated and released?

Do you release your documentation at the same time as your software upgrades? I think that’s the “default” choice for many people because it allows them to have an integrated release process, but you’ll find that it creates some friction. It can be more difficult to update documentation to cover new capabilities when they aren’t available to be tested, screenshotted, or described by the doc authors.

If you have a rigorous change control process, do documentation changes really need to go through that rigmarole? Does every documentation update need a JIRA issue and a Q/A sign-off?

What formats do you want to present the documentation in?

HTML, obviously. Virtually everyone wants HTML documentation.

But, do you also want a more standalone documentation format? It can be quite handy to have PDF output that can be e-mailed to a prospective client, or can be archived on a network folder somewhere.

Once you’ve gathered all that information, you might start to have a picture of what you want your documentation tools to look like. Maybe it’s an integrated documentation tool, but maybe not.

Personally, I like Sphinx as a documentation tool. It can fit into a wide variety of workflows, it can skirt the edge between automatically-generated and manually-written documentation, and it can output in multiple formats including HTML and PDF. sphinxcontrib-httpdomain is an excellent contrib add-on specifically for documenting HTTP APIs.