Iron Crosses in Software Engineering

Iron crosses describe the trade-off space of a solution to a problem, and occur due to competing demands on resources. The classic one is for project management: good, fast, cheap, done. It is often assumed that ‘done’ is non-negotiable, so that trade space just ‘good, fast, cheap; pick two’.

In software engineering, an important trade space is between simplicity, generality, efficiency, and punctuality; or more casually; simple, general, efficient, done.

Simple solutions are cheap (in terms of effort) to maintain, cheap to change, cheap to onboard new maintainers, and easy to hand over.
The fundamental constraint is information/understanding/context and how much is required to understand the solution.

General solutions solve a wider variety of problems, including the one in question, such that you only need one solution for a wide class of problems. Maintenance cost is be amortized over many projects, and duplication of effort is minimize.
The fundamental constraint is scope/abstraction and how many different features are expanded when implementing the solution.

Efficient solutions are cheaper (in terms of dollar cost, or energy) to deploy, more responsive and have lower hardware requirements.
The fundamental constraint here is hardware affinity and how tightly coupled the solution is to the particular set of hardware.

Punctual solutions are done on time and under budget.
The fundamental constraint here is resources; how many (and with experience) are allocated to the solution.

You want them all, but can only choose three.

These are competing constraints, so deliberate or otherwise, a choice is always made.
Better tools make the return on investment cheaper, but can’t make those tradeoffs disappear.

Both engineers and product managers have (usually implicit) assumptions/preferences about how these trade offs are to be made for a given problem. Preferences vary depending on background, culture and tech. Application / library developers tend to like general & efficient, systems programmers tend towards simple & efficient, web development towards towards simple & general. Product managers generally want simple and done.

Failures occur when stakeholder trade-off expectations don’t align.

Example: developer wants to implement a general and efficient solution since they’ll expect to need it later for a different project. Product manager wants it done quickly and simply due to timeline and resource pressure.

Possible outcome: Developer sees PM as making future pain, impacting morale and causing frustration. Product manager see developer as not aligned with business goals and having the wrong priorities. Thing gets shipped but nobody is happy about it.

Usual outcome; developer delivers it late, it’s over-complicated and hard to maintain. It’s also buggy and needs performance issues fixed before it’s usable. If it gets shipped, it gets shipped late and nobody is happy about it.

It is crucial to find alignment between stakeholders in order to reach a mutually satisfactory solution.

Beaglebone Black Baremetal

The Beaglebone Black and Baremetal programming.

The BeagleBone Black (BBB) is a system-on-a-module (SoM) based on the TI AM335x CPU series. It’s relatively cheap, extendable via ‘capes’ (expansion boards) and has some interesting features such a programmable real-time co-processor units (PRU). I bought this board to mess around with real-time operating systems (RTOS) and bare-metal programming. Maybe not the ideal stating platform, but there is plenty of room to grow.

Even to someone comfortable programming in C, the embedded space seems somewhat impenetrable and seems like a combination of antiquated tooling and black magic. My aim here is to fix some of that – which will require some experimentation, exploration and iteration.

Host machine setup.

I’ll be using Ubuntu 20.04 as my development environment, not by choice – I would have preferred to use windows subsystem for Linux (WSL) as I’m running on a PC but WSL doesn’t handle USB very well, and I’ll be needing that to talk to the BeagleBone.

Some things we need:

  • A BeagleBone black.
  • A USB to TTL serial converter cable (I bought this one, and it seems to work just fine).
  • A microSD card (and some way to flash it)
  • GNU cross-compiler toolchain (apt package gcc-arm-none-eabi) and newlib
  • GNU make and associated build tools.
  • TI Starterware with the BeagleBone Black patch. (https://www.ti.com/tool/STARTERWARE-SITARA)

I’ll also be using git for source control, but you could just as well use svn.

Issue The First: Installing TI Starterware.

My first step was to ensure that I could build and deploy the TI examples onto the device. Not surprisingly, I ran into issues almost immediately. Firstly, TI Startware package is distributed as a 32-bit ELF binary, which means it wont run out of the box on a modern Linux distribution. I found this out via a bit of digging on stack-overflow, which pointed me at the Linux application file – which spits out information about the file type, surprisingly enough. Further digging on stack-overflow suggested that i needed to run the following to enable the execution of 32-bit binaries;

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386(

(As an aside, one of my frustrations with Linux – and technology in general – is how much working out how to use the software feels like memorizing incantations. More on this another time…)

The i386 issues is a feature of just how old the library is. TI have since replaced this with a new processor SDK which we may end up exploring later. However, the new SDK is tied to TI’s CodeComposer Studio – an Eclipse variant IDE for the TI chips. Personally, I wish TI was a bit more tooling agnostic, and spent more time and resources on documentation instead of ‘helping’ via an IDE. I get the impression that this is standard practice in the embedded world, so I guess there are reasons for it?

TI provides a patch for the BeagleBone Black, which can simply be extracted into the directory where the Starterware package was installed.

Because I like to make things difficult for myself, I elected to install the latest version of the arm GCC cross compiler toolchain (x86_64 to arm-none-eabi) directly from the arm developer website and installed newlib (libnewlib-arm-none-eabi) via apt.

At this point, we’re ready to try building the demo applications.

Issue The Second: Building TI Demo Applications.

Navigating the Startware SDK was pretty confusing at first, and the documentation was not much help. Eventually, I found the entry point to the build process in build/armv7a/gcc/am335x/beaglebone/makefile. Unsurprisingly, it did not ‘just work’ and so begins the task of debugging the build process.

The first thing was an easy fix; the version of cross-compile toolchain needs to be correctly specified. To work out where this is set, we notice that the entry point makefile simply calls make in each of the specified sub-directories. Each makefile in a particular sub-directory then includes build/armv7a/gcc/makedefs which is where we want to look. There, on the first non-comment line, we find the toolchain definition. 

build/armv7a/gcc/makedefs - line 42
# LIB_GCC=${LIB_PATH}/lib/gcc/arm-none-eabi/4.7.2/   <- Old toolchain.
LIB_GCC=${LIB_PATH}/lib/gcc/arm-none-eabi/10.3.1/
LIB_C=${LIB_PATH}/arm-none-eabi/lib/

The other thing to note is that the environment variable LIB_PATH needs to be set so as to point to our cross compile toochain. Since I’ve install this into /opt, I’m exporting LIB_PATH=/opt/gcc-arm-10.3-2021.07-x86_64-arm-none-eabi in .bashrc.

Now, we should be able to compile some of the applications. Running make in the beaglebone directory does build some applications, but our console is being inundated with warnings about -march and -mcpu being incompatible. With a bit of digging, it seems that this is a change in modern GCC (vs the old 4.7 version the SDK shipped with), and that specifying the architecture is redundant. To fix this, simply delete the flag -march=armv7a from the CFLAGS on line 167 of build/armv7a/gcc/makedefs.

The final issue I encountered was a redefinition of strnstr in the demo application. My hypothesis is that the version of GCC/newlib that this SDK was built against didn’t have strnstr implemented, which is required for the lightweight IP (LWIP) stack that the demo application uses. A bit of grep-ing discovered that LWIP provides a compiler define for this exact situation, meaning that the fix was an easy on – just add DLWIP_HTTPD_STRNSTR_PRIVATE=0 to the LWIPCFLAGS make variable in the demo application makefile (line 91 of build/armv7a/gcc/am335x/beaglebone/demo/makefile).

Issue The Third: Booting the Beaglebone from an SD Card.

The Starterware documentation says that any ‘FAT’ filesystem is sufficient for boot loader, and that one simply needs to rename the build artifacts with TI headers (boot_ti.bin, demo_ti.bin) to MLO and app respectively, for the bootloader to pick it up. Again, not so simple. The general approach here is as follows:

  • Format the microSD to some FAT filesystem (W95 FAT32 LBA seems to work just fine.)
  • Copy across the MLO and app binaries.
  • Plug in the USB-TTL cable and point your serial application (minicom for me) at the corresponding tty port (/dev/ttyUSB0 for me).
  • Insert the microSD card, and power cycle the beaglebone, holding the S2 switch (near the SD slot) on boot.

If this works, you should see a bunch of activity on the COM port.

I encountered two fail-states. Firstly, if the card is not flashed correctly, you may receive ‘CCC‘ from the serial port. I couldn’t find out what that means, but I’m sure it’s not great. It was mostly likely an issue with the filesystem, or the onboard bootloader couldn’t find the MLO binary.

In the second fail state, the COM port registered that the bootloader had started as ‘StarterWare AM335x Boot Loader' was sent down the COM port, but it failed to move onto the next stage. A bit of printf debugging isolated that the bootloader was getting stuck calling f_open. I have no idea why, but it does seem that this bug is not present in debug builds. I don’t have a JTAG emulator (which I think is what’s needed to debug this sort of thing) so I can’t really get too much insight into what’s going on. I guess it’s not too much of a concern for me though, since I’ll probably not be using the Starterware bootloader for too long. If worst comes to worst, the BeagleBone Black patch provided by TI comes with a pre-built bootloader, so I could always just use that, or (more likely) piggyback onto Uboot.

Next Steps

With the TI Starterware working, the next steps will be some tooling and planning. In particular:

  • Constantly copying across binaries to an microSD card is pretty inefficient. The existing bootloader (uboot) allows for booting over the network. Can we use that to make development easier.
  • The TI way of building doesn’t suit my tastes and the source is all over the place. Can we factor the ‘BeagleBone Black’-specific parts of the SDK (and bin the rest) into a something more suitable for what I want to explore.
  • What needs to happen to get the video working? Either via HDMI or LCD.

Til next time.

A foray into articulated robots.

The Darkpaw is a hobbyist quadruped kit that I’ve been playing around with for some time now. The goal is to learn how to model, control and program articulated robots for fun and (maybe one day) profit. This post will give a bit of an overview of how i got started on this, where I’m at, and lay some groundwork for future posts.

Into the web of legged robots.

Legged robots are pretty awesome (see this video by Zenta)

(As a aside – I showed my partner this video and now she wants one to keep her company on jogs so as to light her path and further intimidate the locals…)

or this one by Matt Denton…

Anyway, not long after the start of lockdowns during the COVID-19 pandemic in Melbourne, I found myself looking for a new project. I had been working as a post-doc at the Aero-Mech flight lab, and had been learning about underactuated robotics via Russ Tedrakes excellent course from MIT (very highly recommended). Additionally, I had been working mostly in Python (and ROS – ugh – but more on that another time) and developing/re-discovering my C chops. Upon this backdrop, and inspired by a random incursion into the youtube hexapod rabbit hole, I figured it’d would be fun to actually learn how to program such a robot.

Enter the Darkpaw

After a bit of rummaging around ye-olde-internets, I found the Adeept Darkpaw. It seemed to suit my needs; had most of the parts including motor driver hat, servos, camera and light – all that was needed was a Raspberry Pi 3b+ and some batteries. The ‘code’ was all python, as is the trend in hobbyist robotics, so it would be trivial to reverse-engineer to do whatever i wanted with it. So i got it and then got to work.

Two issues were immediately apparent. Firstly; neither my power adapter nor the two 18650 batteries were sufficient to keep the Pi from going under-voltage and shutting the thing down. (I’ve recently upgraded my power supply to a higher power, and i still get undervoltages, but at least the Pi doesn’t reset).

Secondly. Well. The shipped code was… lacking, and we’ll leave it at that.
Of course, this did not really bother me, (other than the general feeling of distaste when looking at it) since the plan was to re-write my own implementation.

The assembly was pretty straight-forward, and there were plenty of spares just-in-case. I wish it was easier to get the Pi in and out, or at least to access the SD card, but one can’t ask too much of such a cheap kit.

The biggest learning curve has been getting to grips with kinematics in practice. Given my background, one might expect me to find rigid body modelling ‘trivial’. The reality is that it took me a number of attempts to get it right – partly due to the fact that the system has algebraic constraints. This is a consequence of the design, as the motors do not drive the leg joints directly, they instead translate force through levers and links. I’ll save the maths for a later post.

Current State

A sweep motion running on all legs.

Currently Implemented:

  • Motor / LED drivers (via user-space i2c).
  • Forward and inverse kinematics.
  • A basic sweep/reset walk cycle for testing.

Next goal is a footstep planner!

About Me

Welcome to the personal site of Dr. Peter Cudmore. Please enjoy your stay.

If you are what you do, then I spend my time:

  • Working as a postdoctoral researcher in optimisation and control with a deep interest in complex systems and nonlinear dynamics as applied to engineering in the broadest sense.
  • Messing around with robotics and embedded computers.
  • Learning/thinking about philosophy, psychology, robotic/systems intelligence and cognitive science.
  • Playing guitar/bass/synth.


I have

  • Obtained a Ph. D. in applied mathematics, studying synchronisation.
  • Played music professionally.
  • Wrote software prototypes for distributed control of quadrotor swarms.
  • Worked on cross-domain modelling tools for biochemical-electrical-mechanical systems.
  • Designed and built an onboard preamp for my bass.
  • Worked on control and camera software for underwater remotely operated vessel.