Endings and Beginnings

1 Reply

Summary

In this post, I’ll review at the last 2 years of VRTX Systems and talk briefly about the challenges/opportunities that led me to sidelining it.

VRTX Systems; the pitch.

So a little background; a little over two years ago I left a good job and a great team @ Arkeus to set out on my own. There were a few motivating factors; the desire to work for myself, and recognition of some significant gaps in the market when it came to building distributed robot systems. In particular, there was (and continues to be) significant hype around drone swarms and autonomous robot collaboration despite there being at least one significant engineering barrier that is yet to be overcome, at least in the open/civilian market – relative drone positioning & communication.

I was/am mostly excited about using distributed robot collaboration on heavy lift copters – for places where you want a crane, but can’t get one there – coastlines, shipping, mine sites, search-and-rescue and (in the defence setting) contested logisitics. Key feature being that the environment does not have reliable infrastructure, making having portable heavy lift a benefit. By doing this with drones, you might not be as cost-effective as a hiring a helicopter, but you’ll have a smaller per-unit cost, and if swarming technology is good enough, then the ability to scale to fit the problem. All of this assumes that each drone in the fleet knows the relative position of their neighbors.

GPS is by far the most common drone positioning tool, is passive and easily scalable, but vulnerable to spoofing/jamming, and only provides a location for the robot with the GPS receiver – you need a communication network as well. To get drone-to-drone position estimates, you would measure your GPS location, and broadcast that to the network.

Most of the current communication technologies are not designed for distributed operation – both WiFi & Cellular usually require a centralised node. Those that do (e.g. BATMAN) are set up to mimic the existing BSD network stack enforcing a layered approach, which makes less sense in a network of highly mobile communicators. In traditional networking, we decouple the network address from the physical location, for networks of mobile agents, we want the exact opposite.

VRTX Systems first product (VRTX RDIO) was indented to solve these problems by providing an encrypted mesh communication network with relative positioning for drones; meaning low power, hard-realtime, secure, resilient to interference and scalable. The key to solving this problems was to implement a secure decentralised network clock so that network nodes could measure time-of-arrival of packets to estimate distance.

Over the course of VRTX Systems, I got to the point where i had

  1. a mostly working software stack on the embedded radio base on a STM32 LoRa development board & DecaWave unit),
  2. broadcast-based communication over LoRa,
  3. a distributed secure handshake & AES encryption scheme that could handle nodes joining and leaving,
  4. Partial Network clock synchronisation.

I had hit a bit of a wall around November 2024, the key problem was that the network sync algorithm required either a coarse clock to separate the synchronisation into rounds (not scalable), or the development of a fully asynchronous version. The first case was an engineering problem, which would take time to solve (and probably a new board, since i was outgrowing the memory capacity of the STM32L0 chip). The second case required actual research; for me to take off my engineer hat, and put on my mathematician hat. The first cost time & money, the second cost time & risk.

The obvious decision was to implement the coarse clock sync, it was more important to move closer to a product than to “get it right”, but it was a little disheartening nonetheless. Fortunately, around this time I had a opportunity to work with my friends at Arkeus again, which gave me some breathing room, a team environment, and some new experiences.

Learnings

Embedded software is hard.

I knew this going in, but i keep getting reminded of it every time. Embedded software development is a different beast. In most application development, you expect that your vendored code is either a binary blob with header files, or source code at works, or a library provided by your OS or development envirnomnet at best.
This is a lot less standardisation in embedded development. That radio you’re using? Someone from ST has put together a demo application but it’s a demo – heaps of features listed on the device datasheet are missing and there is a lot of drudge-work to get the stuff you want implemented in the way that you want it.

There is also a whole lot of churn. I settled on the Arm M-bed 2.0 ecosystem initially since it seem to have the friendliest development experience, only to have Arm pull the plug on officially supporting it 2-3 months later.

The biggest win was getting familiar with embedded rust, particularly the RTIC and embassy-rs libraries. The next bare-metal project I work on will most likely use one of these. I definitely feel that with a few more tweaks embedded rust will be the best embedded development environment by far.

Don’t underestimate low TRL risk.

Technology Readiness Level (TRL) is a common way to measure how mature a particular technology is. Generally speaking, academic publications are between TRL 1 and TRL 3, while things you can buy on the shelf of your favourite tech store are TRL 9+.
Generally speaking, my personal “sweet spot” is to pick things at low TRL (say 2-3) – academic publications, prototypes etc. – an drag them up to TRL6-ish – i.e. a working system with almost no design/implementation risk left, but still heaps of risk around deployment & sustainment. This gap (TRL-4 to TRL-6) is the so-called “valley of death” because it is here where many technologies fail on their way to commercialisation.

There are many reasons why this happens, but the one I want to focus on is “assumption propagation”. When you bundle a bunch of technologies together making then co-interacting, the operating requirements of one technology puts operating constraints upon another. These interactions cascade out in a way that can be pretty unpredictable. Best case, you can handle it. Worst case, it just straight up won’t work and you need to go back to the drawing board.

The best way around this is to build the smallest, dumbest, dirtiest, “complete system” prototype, but to build it in a way that makes iteration, learning & discovery the objective.

My mistake with VRTX RDIO was that I focused on the engineering risk, and iterated on the embedded software, before appropriately de-risking the clock sync. At the end of the day though, I think it was a damned-if-you-do-damned-if-you-don’t, since there was a whole bunch of risks associated with the radio board development that I had to work through.

The solo founder experience is not for me.

I’m very familiar with working solo for prolonged periods of time – a PhD. in maths will do that – but the solo founder experience is a different beast. When you do a PhD., you focus deep on one topic with the help of a supervisor, the camaraderie of your fellow candidates, and ideally feedback from your school/wider community.

As a solo founder, you’re frequency context switching to solve whatever the most immediate problem is (software, firmware, electronics, design and business), with no established community (you have to take time out of work to do that) and no real feedback unless you’re open-sourcing (different business model) or you are at the point where you have customers/clients. This might work well for some people, but it doesn’t for me. I think it would be fine if i were not “on a timer” (i.e. when the allocated savings run out), but as it was I struggled to stay positive about getting VRTX sustainable quickly enough.

Doing some freelance work really highlighted for me how much being part of a team means to me, and how much more effectively I work when I know what I contribution is part of a larger cause.

The future of VRTX Systems

For now, VRTX Systems is on ice while I focus on other things.
Details are for another time, but I’ve founded a company – Pneumatica.Bio – with some friends/colleagues from my days in the Systems Biology Lab.
You’ll be hearing more about Pneumatica very soon as we have a pretty massive 2026 coming up.

Thanks for reading.

Iron Crosses in Software Engineering

Iron crosses describe the trade-off space of a solution to a problem, and occur due to competing demands on resources. The classic one is for project management: good, fast, cheap, done. It is often assumed that ‘done’ is non-negotiable, so that trade space just ‘good, fast, cheap; pick two’.

In software engineering, an important trade space is between simplicity, generality, efficiency, and punctuality; or more casually; simple, general, efficient, done.

Simple solutions are cheap (in terms of effort) to maintain, cheap to change, cheap to onboard new maintainers, and easy to hand over.
The fundamental constraint is information/understanding/context and how much is required to understand the solution.

General solutions solve a wider variety of problems, including the one in question, such that you only need one solution for a wide class of problems. Maintenance cost is be amortized over many projects, and duplication of effort is minimize.
The fundamental constraint is scope/abstraction and how many different features are expanded when implementing the solution.

Efficient solutions are cheaper (in terms of dollar cost, or energy) to deploy, more responsive and have lower hardware requirements.
The fundamental constraint here is hardware affinity and how tightly coupled the solution is to the particular set of hardware.

Punctual solutions are done on time and under budget.
The fundamental constraint here is resources; how many (and with experience) are allocated to the solution.

You want them all, but can only choose three.

These are competing constraints, so deliberate or otherwise, a choice is always made.
Better tools make the return on investment cheaper, but can’t make those tradeoffs disappear.

Both engineers and product managers have (usually implicit) assumptions/preferences about how these trade offs are to be made for a given problem. Preferences vary depending on background, culture and tech. Application / library developers tend to like general & efficient, systems programmers tend towards simple & efficient, web development towards towards simple & general. Product managers generally want simple and done.

Failures occur when stakeholder trade-off expectations don’t align.

Example: developer wants to implement a general and efficient solution since they’ll expect to need it later for a different project. Product manager wants it done quickly and simply due to timeline and resource pressure.

Possible outcome: Developer sees PM as making future pain, impacting morale and causing frustration. Product manager see developer as not aligned with business goals and having the wrong priorities. Thing gets shipped but nobody is happy about it.

Usual outcome; developer delivers it late, it’s over-complicated and hard to maintain. It’s also buggy and needs performance issues fixed before it’s usable. If it gets shipped, it gets shipped late and nobody is happy about it.

It is crucial to find alignment between stakeholders in order to reach a mutually satisfactory solution.

Beaglebone Black Baremetal

The Beaglebone Black and Baremetal programming.

The BeagleBone Black (BBB) is a system-on-a-module (SoM) based on the TI AM335x CPU series. It’s relatively cheap, extendable via ‘capes’ (expansion boards) and has some interesting features such a programmable real-time co-processor units (PRU). I bought this board to mess around with real-time operating systems (RTOS) and bare-metal programming. Maybe not the ideal stating platform, but there is plenty of room to grow.

Even to someone comfortable programming in C, the embedded space seems somewhat impenetrable and seems like a combination of antiquated tooling and black magic. My aim here is to fix some of that – which will require some experimentation, exploration and iteration.

Host machine setup.

I’ll be using Ubuntu 20.04 as my development environment, not by choice – I would have preferred to use windows subsystem for Linux (WSL) as I’m running on a PC but WSL doesn’t handle USB very well, and I’ll be needing that to talk to the BeagleBone.

Some things we need:

  • A BeagleBone black.
  • A USB to TTL serial converter cable (I bought this one, and it seems to work just fine).
  • A microSD card (and some way to flash it)
  • GNU cross-compiler toolchain (apt package gcc-arm-none-eabi) and newlib
  • GNU make and associated build tools.
  • TI Starterware with the BeagleBone Black patch. (https://www.ti.com/tool/STARTERWARE-SITARA)

I’ll also be using git for source control, but you could just as well use svn.

Issue The First: Installing TI Starterware.

My first step was to ensure that I could build and deploy the TI examples onto the device. Not surprisingly, I ran into issues almost immediately. Firstly, TI Startware package is distributed as a 32-bit ELF binary, which means it wont run out of the box on a modern Linux distribution. I found this out via a bit of digging on stack-overflow, which pointed me at the Linux application file – which spits out information about the file type, surprisingly enough. Further digging on stack-overflow suggested that i needed to run the following to enable the execution of 32-bit binaries;

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386(

(As an aside, one of my frustrations with Linux – and technology in general – is how much working out how to use the software feels like memorizing incantations. More on this another time…)

The i386 issues is a feature of just how old the library is. TI have since replaced this with a new processor SDK which we may end up exploring later. However, the new SDK is tied to TI’s CodeComposer Studio – an Eclipse variant IDE for the TI chips. Personally, I wish TI was a bit more tooling agnostic, and spent more time and resources on documentation instead of ‘helping’ via an IDE. I get the impression that this is standard practice in the embedded world, so I guess there are reasons for it?

TI provides a patch for the BeagleBone Black, which can simply be extracted into the directory where the Starterware package was installed.

Because I like to make things difficult for myself, I elected to install the latest version of the arm GCC cross compiler toolchain (x86_64 to arm-none-eabi) directly from the arm developer website and installed newlib (libnewlib-arm-none-eabi) via apt.

At this point, we’re ready to try building the demo applications.

Issue The Second: Building TI Demo Applications.

Navigating the Startware SDK was pretty confusing at first, and the documentation was not much help. Eventually, I found the entry point to the build process in build/armv7a/gcc/am335x/beaglebone/makefile. Unsurprisingly, it did not ‘just work’ and so begins the task of debugging the build process.

The first thing was an easy fix; the version of cross-compile toolchain needs to be correctly specified. To work out where this is set, we notice that the entry point makefile simply calls make in each of the specified sub-directories. Each makefile in a particular sub-directory then includes build/armv7a/gcc/makedefs which is where we want to look. There, on the first non-comment line, we find the toolchain definition. 

build/armv7a/gcc/makedefs - line 42
# LIB_GCC=${LIB_PATH}/lib/gcc/arm-none-eabi/4.7.2/   <- Old toolchain.
LIB_GCC=${LIB_PATH}/lib/gcc/arm-none-eabi/10.3.1/
LIB_C=${LIB_PATH}/arm-none-eabi/lib/

The other thing to note is that the environment variable LIB_PATH needs to be set so as to point to our cross compile toochain. Since I’ve install this into /opt, I’m exporting LIB_PATH=/opt/gcc-arm-10.3-2021.07-x86_64-arm-none-eabi in .bashrc.

Now, we should be able to compile some of the applications. Running make in the beaglebone directory does build some applications, but our console is being inundated with warnings about -march and -mcpu being incompatible. With a bit of digging, it seems that this is a change in modern GCC (vs the old 4.7 version the SDK shipped with), and that specifying the architecture is redundant. To fix this, simply delete the flag -march=armv7a from the CFLAGS on line 167 of build/armv7a/gcc/makedefs.

The final issue I encountered was a redefinition of strnstr in the demo application. My hypothesis is that the version of GCC/newlib that this SDK was built against didn’t have strnstr implemented, which is required for the lightweight IP (LWIP) stack that the demo application uses. A bit of grep-ing discovered that LWIP provides a compiler define for this exact situation, meaning that the fix was an easy on – just add DLWIP_HTTPD_STRNSTR_PRIVATE=0 to the LWIPCFLAGS make variable in the demo application makefile (line 91 of build/armv7a/gcc/am335x/beaglebone/demo/makefile).

Issue The Third: Booting the Beaglebone from an SD Card.

The Starterware documentation says that any ‘FAT’ filesystem is sufficient for boot loader, and that one simply needs to rename the build artifacts with TI headers (boot_ti.bin, demo_ti.bin) to MLO and app respectively, for the bootloader to pick it up. Again, not so simple. The general approach here is as follows:

  • Format the microSD to some FAT filesystem (W95 FAT32 LBA seems to work just fine.)
  • Copy across the MLO and app binaries.
  • Plug in the USB-TTL cable and point your serial application (minicom for me) at the corresponding tty port (/dev/ttyUSB0 for me).
  • Insert the microSD card, and power cycle the beaglebone, holding the S2 switch (near the SD slot) on boot.

If this works, you should see a bunch of activity on the COM port.

I encountered two fail-states. Firstly, if the card is not flashed correctly, you may receive ‘CCC‘ from the serial port. I couldn’t find out what that means, but I’m sure it’s not great. It was mostly likely an issue with the filesystem, or the onboard bootloader couldn’t find the MLO binary.

In the second fail state, the COM port registered that the bootloader had started as ‘StarterWare AM335x Boot Loader' was sent down the COM port, but it failed to move onto the next stage. A bit of printf debugging isolated that the bootloader was getting stuck calling f_open. I have no idea why, but it does seem that this bug is not present in debug builds. I don’t have a JTAG emulator (which I think is what’s needed to debug this sort of thing) so I can’t really get too much insight into what’s going on. I guess it’s not too much of a concern for me though, since I’ll probably not be using the Starterware bootloader for too long. If worst comes to worst, the BeagleBone Black patch provided by TI comes with a pre-built bootloader, so I could always just use that, or (more likely) piggyback onto Uboot.

Next Steps

With the TI Starterware working, the next steps will be some tooling and planning. In particular:

  • Constantly copying across binaries to an microSD card is pretty inefficient. The existing bootloader (uboot) allows for booting over the network. Can we use that to make development easier.
  • The TI way of building doesn’t suit my tastes and the source is all over the place. Can we factor the ‘BeagleBone Black’-specific parts of the SDK (and bin the rest) into a something more suitable for what I want to explore.
  • What needs to happen to get the video working? Either via HDMI or LCD.

Til next time.

A foray into articulated robots.

The Darkpaw is a hobbyist quadruped kit that I’ve been playing around with for some time now. The goal is to learn how to model, control and program articulated robots for fun and (maybe one day) profit. This post will give a bit of an overview of how i got started on this, where I’m at, and lay some groundwork for future posts.

Into the web of legged robots.

Legged robots are pretty awesome (see this video by Zenta)

(As a aside – I showed my partner this video and now she wants one to keep her company on jogs so as to light her path and further intimidate the locals…)

or this one by Matt Denton…

Anyway, not long after the start of lockdowns during the COVID-19 pandemic in Melbourne, I found myself looking for a new project. I had been working as a post-doc at the Aero-Mech flight lab, and had been learning about underactuated robotics via Russ Tedrakes excellent course from MIT (very highly recommended). Additionally, I had been working mostly in Python (and ROS – ugh – but more on that another time) and developing/re-discovering my C chops. Upon this backdrop, and inspired by a random incursion into the youtube hexapod rabbit hole, I figured it’d would be fun to actually learn how to program such a robot.

Enter the Darkpaw

After a bit of rummaging around ye-olde-internets, I found the Adeept Darkpaw. It seemed to suit my needs; had most of the parts including motor driver hat, servos, camera and light – all that was needed was a Raspberry Pi 3b+ and some batteries. The ‘code’ was all python, as is the trend in hobbyist robotics, so it would be trivial to reverse-engineer to do whatever i wanted with it. So i got it and then got to work.

Two issues were immediately apparent. Firstly; neither my power adapter nor the two 18650 batteries were sufficient to keep the Pi from going under-voltage and shutting the thing down. (I’ve recently upgraded my power supply to a higher power, and i still get undervoltages, but at least the Pi doesn’t reset).

Secondly. Well. The shipped code was… lacking, and we’ll leave it at that.
Of course, this did not really bother me, (other than the general feeling of distaste when looking at it) since the plan was to re-write my own implementation.

The assembly was pretty straight-forward, and there were plenty of spares just-in-case. I wish it was easier to get the Pi in and out, or at least to access the SD card, but one can’t ask too much of such a cheap kit.

The biggest learning curve has been getting to grips with kinematics in practice. Given my background, one might expect me to find rigid body modelling ‘trivial’. The reality is that it took me a number of attempts to get it right – partly due to the fact that the system has algebraic constraints. This is a consequence of the design, as the motors do not drive the leg joints directly, they instead translate force through levers and links. I’ll save the maths for a later post.

Current State

A sweep motion running on all legs.

Currently Implemented:

  • Motor / LED drivers (via user-space i2c).
  • Forward and inverse kinematics.
  • A basic sweep/reset walk cycle for testing.

Next goal is a footstep planner!

About Me

Welcome to the personal site of Dr. Peter Cudmore. Please enjoy your stay.

If you are what you do, then I spend my time:

  • Working as a postdoctoral researcher in optimisation and control with a deep interest in complex systems and nonlinear dynamics as applied to engineering in the broadest sense.
  • Messing around with robotics and embedded computers.
  • Learning/thinking about philosophy, psychology, robotic/systems intelligence and cognitive science.
  • Playing guitar/bass/synth.


I have

  • Obtained a Ph. D. in applied mathematics, studying synchronisation.
  • Played music professionally.
  • Wrote software prototypes for distributed control of quadrotor swarms.
  • Worked on cross-domain modelling tools for biochemical-electrical-mechanical systems.
  • Designed and built an onboard preamp for my bass.
  • Worked on control and camera software for underwater remotely operated vessel.