How we validate our Toolchains
It’s important that we provide LibreEMS users/developers a stable and reliable toolchain for compiling the LibreEMS sources into firmware for their EMS/ECU, so how do we validate that our Toolchains generate consistent output across all of the Linux OS’s we support in both 32 and 64 bit variants? (We support Debian: sid, jessie, wheezy, Ubuntu: lucid, precise, quantal, raring, saucy)?
Well we could do it the hard and error prone (and frankly insane way) of installing each of those OS’s by hand on either physical or virtual machines in both 32 and 64 bit forms, downloading he packages from our repo, installing them and checking out the firmware and compiling it by hand, then verifying the md5sum of the firmware match up.
We don’t know how much time you have on your hands, but we don’t have THAT much free time, so being the geeks that we are we leveraged automation as much as possible and made use of some excellent OSS tools. Enter “sbuild” which is what Debian uses to build uploaded source packages across all of their supported releases in a completely automated manner.
We have 16 pristine minimally configured chroot’s build (lucid, precise, quantal, raring, saucy, sid, jessie, wheezy) in both 32 and 64 bit variants, that are updated daily, and we wrote a shell script that loops through each one, doing basically what an end user would need to to to build the firmware on their end.
- Install APT key for our repo
wget -q http://apt.libreems.org/apt.libreems.org-pub.asc -O- | sudo apt-key add -
- add the apt.libreems.org repo
sudo wget -q -O /etc/apt/sources.list.d/apt.libreems.org.list http://apt.libreems.org/lists/apt.libreems.org_DIST.list
NOTE: this URL and filename will vary based on the target distribution by changing the DIST value for the appropriate one for the chroot..
apt-get -y install mc9s12x-toolchain
- checkout the firmware from git (we use a specific branch as shown and we know it compiles to a specific known hash)
git checkout -b addons-xgate-sequential-stagedInjection-MAF http://git.libreems.org/libreems-suite/libreems-firmware.git firmware
- Build it
cd firmware ; HASHCHECK=1 make SEANKLT1
- Check the sum against the known default.
making sure it matches with
NOTE: the “HASHCHECK=1” string before “make”. This is done because the makefiles inject dynamic strings in the sourcecode indicating the release version, Build OS and Build date, since any change in those strings will result in a different output md5sum that environment variable forces the Makefile to inject FIXED STATIC STRINGS in place of those variables resulting in a consistent md5sum in the target firmware.
We do this procedure any time the Toolchain packages are modified and released to the public. For OS-X and Windows Builds, this test is still done manually, as we have not yet found a reliable and easy way to virtualize OS-X on KVM, and the amount of effort needed to implement the CLI automation in windows to do the same test wasn’t worthwhile due to how infrequently the tools are released.
NOTE: firmware compiled on Windows will NOT having matching md5sums for the .s19 files compared to Linux/OS-X. However converting the .s19 files taht were compiled with the windows toolchain into .bin files using s19tool and comparing those results in a perfect matchup. We suspect this has to do with line-endings in windows for strings (CRLF vs LF). If you know more on this, please let us know in the comments below.