Building ARMV6 Ports on FreeBSD with Poudriere

Introduction

While FreeBSD is a great OS, as of this writing (mid-2015) it currently lacks official support for the increasingly popular ARM architectures, which means freebsd.org (or its mirrors) do not provide pre-built ports (packages for those new to BSD) in the same way they are available for x86 / amd64 architectures. This means most any package you likely want to install on your newly built ARM target is not available in binary form and therefore accessible via the convenient pkgng tool.

While some developers provide public access to the repositories they've managed to cobble together the reality is there is no guarantee the repository will have built all the packages required for your application, or built those packages with needed options. This leaves the obvious choice, which is, not surprisingly, aligned with the BSD philosophy -- build everything from source yourself! Of course, it's not exactly that simple, as building on the target could easily overwhelm the relatively modest CPU and memory resources of the typical ARM platform. In short, it would take forever, if it completed at all.

The best way to build packages (for ARM or other architectures) is with the poudriere tool. Poudriere facilitates the creation of jails in which packages can be built in a pristine environment. It intelligently handles the build process, logs everything, and even provides its outputs in a format that can be readable by a common web browser. In short, it rocks. Combined with a trick command to tell the kernel to use the appropriate qemu emulator to execute any binaries built for ARM, poudriere provides the ability to cross compile packages at speeds far exceeding what is practical or possible natively. In short, poudriere makes building on the target unnecessary.

This document outlines my experience configuring a freshly installed FreeBSD 10.1 server with ZFS filesystem to build packages for the armv6 architecture appropriate for Raspberry PI, Freescale Solo/Dual/Quad (i.e. Wandboard), and similar SoCs.

Requirements

This process requires an active Internet connection (the faster the better) and root access to the build host.

Build servers should have as much memory as possible (my host has 16G at this writing) and at least several hundred GB of conventional hard drive (i.e. non-SSD) space available. I recommend avoiding SSDs as they will be beat to death prematurely. I love SSDs but they are best saved for activities that require high I/O performance. Building code isn't one of them.

FreeBSD is closely married with the subversion source control system. This package must be present on the server and may be installed via pkgng ("pkg") as follows:

# pkg install subversion

Reference Articles

These articles were instrumental in getting me started, but they did not provide all of the information necessary to get my build server up and running. While it is not essential to read them prior to reading my article, you may find them helpful.

EvilCoder: Package Building Farm for AMD64 and ARMv6

IgnorantHack: Cross Building Ports with QEMU and Poudriere Devel on FreeBSD

Installing the Ports Tree

The poudriere tool must be installed from source but before that is possible on a virgin FreeBSD installation a copy of the ports tree must be downloaded and extracted as follows:

# portsnap fetch
# portsnap extract

The fetch command will snarf about 75MB from the net, while the extract command will install the ports tree in /usr/ports. Note that the extract command can be executed from anywhere and the ports tree will always wind up in /usr/ports.

This ports tree provides the ability to build packages for the native build host only (amd64 in my case). This should not be confused with the one that will be installed with poudriere later.

Building Poudriere From Source

The binary version of Poudirere provided by freebsd.org for the x86/amd64 architectures is built without the qemu option. This means poudriere must be built from source. With the ports tree installed this is as simple as executing the following commands:

# cd /usr/ports/ports-mgmt/poudriere-devel/ && make config
# make install clean

The first command will present a curses (command line based) menu. Select the qemu option and continue.

Note that this process will ask to configure options for several more dependencies that need to be built, at least on a virgin ports tree. I accepted all default settings.

[Edit]: Some readers have brought to my attention that it is not necessary to build Poudriere from source to get the qemu option. Apparently, installing the package qemu-user-static will do the job, e.g.:

# pkg install qemu-user-static

I have not tested this, however. If you install the package and it works for you please send me a brief note to let me know.

Configuring Poudriere

The poudriere configuration file must be manually edited before the tool may be used. The poudriere config file is located here:

/usr/local/etc/poudriere.conf

Rather than copy the entire file here I offer a list of the variables I changed or uncommented (i.e. made active):

ZPOOL=zroot
ZROOTFS=/poudriere
FREEBSD_HOST=ftp://ftp2.us.FreeBSD.org/pub/FreeBSD/
BASEFS=/usr/local/poudriere
SVN_HOST=svn0.us-east.FreeBSD.org
POUDRIERE_DATA=${BASEFS}/data
NOLINUX=yes
USE_COLORS=yes
PRESERVE_TIMESTAMP=yes
BUILDER_HOSTNAME=build

I picked FREEBSD_HOST from the list of ftp mirrors.

I picked SVN_HOST from the list of subversion mirrors.

Creating the Jail

The poudriere "jail" command along with a few options is used create the jail in which packages will be built. I used the following command syntax:

# poudriere jail -c -j 101armv6 -m svn -a arm.armv6 -v release/10.1.0

This means:

• (-c) create the jail
• (-j) name the jail "101armv6", which is my convention for "FreeBSD Version 10.1" and the "ARMV6 architecture"
• (-m) use the subversion method to acquire stuff. This is why we installed the subversion package earlier.
• (-a) build for ARMV6 architecture. Note that this must be "arm.armv6", rather than just "armv6" or a build error will occur.
• (-v) the release version in the FreeBSD subversion repository from which we want to pull source. As this version is ultimately translated into a path within the repository it cannot be abbreviated (e.g. 10.1) or the path will not be found. As you can see I used "10.1.0".

This command is executed once during the setup process and then the jail is reused for all future builds.

See also:

https://github.com/freebsd/poudriere/issues/255

Selecting Packages to Build

To build all packages (i.e. 24000) one could theoretically issue this command:

# poudriere bulk -j 101armv6 -a

This is unwise for a couple reasons. First, it will take forever, as in days (or perhaps weeks if the only chip you have is a Dorito). Second, it's unlikely all ports will build on ARM at this writing. I'm sure this will improve over time but at the moment requesting a build of all packages is just a fool's errand. Since I was only interested in building a few specific packages I needed for my application I created this file:

/usr/local/etc/poudriere.d/pkglist.txt

In that file I wrote the the list of packages to build, one per line, in "section/packagename", format, e.g.:

www/nginx
lang/php56-5.6.8

Naturally, it's important to get the package sections and names correct. You can search for the correct values here:

https://www.freebsd.org/ports/searching.html
http://www.freshports.org/

Configuring QEMU

Here's the special sauce that makes it possible to run cross-compiled binaries in the qemu emulator on the fly...

# binmiscctl add armv6 --interpreter "/usr/local/bin/qemu-arm-static"
      --magic
      "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00"
      --mask
      "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"
      --size 20 --set-enabled

It's important that all of the backslashes depicted are included with the command.

It's also important to make sure that the path to qemu is correct for your machine. I saw several examples and all used "qemu-arm", while my system had "qemu-arm-static" installed instead. If the correct path is not provided when this command is issued poudriere will ultimately issue this error:

[00:00:00] ====>> Cross-building ports for armv6 on amd64 requires QEMU 
[00:00:00] ====>> Error: You need to setup an emulator with binmiscctl(8) for armv6

If you get the path wrong the existing emulator definition (just added) must be removed first:

# binmiscctl remove armv6

If you merely try to issue the "add" command again verbatim without removing the existing definition, binmiscctl will output the error:

'armv6' is not unique in activator list

plus the binmiscctl usage syntax. I'm not sure why the tool outputs usage, as the syntax isn't incorrect. It should instead tell you what you what's really wrong: you attempted to redefine an existing definition in the activator list and that is (apparently) not possible.

Note that this command is not persistent so it must be issued prior to building any code after every system boot. This can be placed in a script that is manually executed prior to a build or automated by placing it in an init script as follows:

# vi /usr/local/etc/rc.d/qemu-setup.sh
# chown root:root /usr/local/etc/rc.d/qemu-setup.sh
# chmod 555 /usr/local/etc/rc.d/qemu-setup.sh

The recommended content of the init script is documented on the FreeBSD site.

Installing the Default Ports Tree

Poudriere requires the installation of a ports tree under its data hierarchy that it can reference in the jails. This is most conveniently installed via:

# poudriere ports -c

This will create a ports tree called "default" here:

/usr/local/poudriere/ports/default

If you attempt to run a build without the "default" port tree installed you'll see the following error:

/usr/local/share/poudriere/bulk.sh: cannot open /usr/local/etc/poudriere.d/ports/default/mnt: No such file or directory

Provisioning Poudriere Options

While I suppose it is not strictly necessary to perform this step separately, doing it before building makes building a Ronco operation (set it and forget it). While located in /usr/local/etc/poudriere.d I issued the poudriere options command:

# poudriere options -cf pkglist.txt

That took me through a series of build configuration dialogs, all of which I accepted as default except for nginx in which I enabled the spdy extension for my application.

Starting The Build

The build is then started easily enough with this command:

# poudriere bulk -j 101armv6 -f /usr/local/etc/poudriere.d/pkglist.txt

This will instruct poudriere to build the packages listed in the pkglist.txt file as well as their dependencies. For what it's worth, including the two packages outlined above ultimately resulted in the build of 19 packages which took about 1.5 hours on my unlocked Corei7 with hyperthreading enabled. My advice? Be careful what you ask for, and keep the first build short. Start with a single package with a minimum of dependencies.

While poudriere is what I'd call sufficiently chatty about what's going on during the build, some packages can take a while and it may appear that the build is hung. I used [ctrl]-t to check on the status of the build periodically just for giggles. You can see a snippet of typical console output in the header picture of this article.

Configuring Optional Web Service

Poudriere provides build status on the command line, obviously, but it also formats data in a manner that is suitable for delivery to a common web browser. To provide that access a web server must be provisioned to look in the right places in the poudriere data hierarchy.

This could easily be done with Apache, but I needed to gain some experience with nginx which I intended to use on my ARM target so I decided to use that on my build server instead. Should you decide to follow in my footsteps you can do what I did and update the nginx configuration file here:

/usr/local/etc/nginx/nginx.conf

with the following stanza:

server {
        listen       1.2.3.4:80;
        server_name  pkg.feld.me;
        root         /usr/local/share/poudriere/html;

        # Allow caching static resources
        location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|woff|css|js|html)$ {
                add_header Cache-Control "public";
                expires 2d;
        }

        location /data {
                alias /usr/local/poudriere/data/logs/bulk;

                # Allow caching dynamic files but ensure they get rechecked
                location ~* ^.+\.(log|txz|tbz|bz2|gz)$ {
                        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                }

                # Don't log json requests as they come in frequently and ensure
                # caching works as expected
                location ~* ^.+\.(json)$ {
                        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                        access_log off;
                        log_not_found off;
                }

                # Allow indexing only in log dirs
                location ~ /data/?.*/(logs|latest-per-pkg)/ {
                        autoindex on;
                }

                break;
        }

        location /repo {
                alias /usr/local/poudriere/data/packages;
        }
}

Note that this must replace the existing (default) server stanza in the configuration file or the server will fail with several "failed to bind to port" error messages. Once installed I restarted the service via:

# service nginx onerestart

and pointed my browser at the IP address of the build server. Worked like a charm.

See also:

Makefile.feld: FreeBSD poudriere Cheat Sheet

Conclusion

This represents a solid day of banging my head against the wall and most of that was due to simple inexperience with FreeBSD. (Forgive me, I'm coming off a 20 year bender with Unix and Linux). Hopefully this guide will allow you to produce the same results in far less time. If you have found this article helpful, please consider a small donation to my site support fund to encourage me to create more content like this.