Cross-compiling

Cross-compiling for the Raspberry Pi

There are some notes here on cross-compiling two programs for the Raspberry Pi - one is the operating system kernel and associated modules (takes many, many hours on the Raspberry Pi itself), and the other is the NTP timekeeping software. This latter is still in progress, but it's reported here as at least a working set of NTP programs have been produced. I made the kernel compile first, and thus will have installed tools essential to the NTP cross-compile, so I suggest you follow the steps for the kernel recompile even if you don't actually do the final time-intensive, compiling step.

Kernel cross-compile

I did this based on the notes here and here, using a Debian 7.2, 64-bit install, running in a VMware workstation virtual machine, hosted on a Windows-7 64-bit PC. I would recommend a 20 GB virtual disk - I initially used 10 GB and it only allowed about one Raspberry Pi kernel compilation. This is much quicker than doing the same operation on the Raspberry Pi itself, but I have yet to test the kernel and modules I have compiled on a real Raspberry Pi...

Notes on VMware configuration and tools installation

I used Linux Debian 7.2 with a 20 GB disk allowance, 1 GB memory and 2 processors. Having created the installation, I gave the I had created "david" rights to us the sudo command by using the root terminal application from the menu, and entering:

adduser david sudo

Then I needed to log out and back in. To install VMware tools, there are a number of utilities which may be required, so install them now. I found I kept getting error messages about the install CDROM not being available, so I needed to edit the file:

sudo nano /etc/apt/sources.list

to comment out the line pointing to the CDROM. There seemed to be two copies of the line referring to "cdrom" in my file, and only the first was already commented out. Now we can install more of the required packages.

apt-get install make
apt-get install gcc
apt-get install libc-dev
apt-get install git
apt-get install ncurses-dev

Use the uname -r command to find out what your kernel version is, and then apt-get install the relevant kernel headers as follows…

uname -r
apt-get install linux-headers-3.2.0-4-amd64

Once you have clicked the VM menu, install VM tools, you can click the Help button on the bar below the Linux screen display to find out the exact procedure to install the tools. Everything is a lot easier once you can use the right-click, Paste menu to copy commands into the terminal window.

Host update for 32-bit libraries on a 64-bit Linux OS

These steps are only needed on a 64-bit Linux, I was using a Debian 7.2 system.

Since Wheezy introduces multiarch, the ia32-libs package in now deprecated. It is now possible to install 32bit packages directly:

sudo  dpkg  --add-architecture  i386 # enable multi-arch
sudo apt-get  update

Then run:

sudo apt-get install ia32-libs

Downloading the OS source and required tools

I created a new directory "kern" in my home directory, and then downloaded the OS source and required cross-compiling tools. I had problems with my first git download of the 698 MB source, but you may be luckier. It's nearer 106 MB when compressed into a .gz archive and without a lot of the dross.

Either, preferred by me....

mkdir kern
cd kern
wget https://github.com/raspberrypi/linux/archive/rpi-3.6.y.tar.gz
tar xvfz rpi-3.6.y.tar.gz

or, although this seems to create many more files than are needed ...

mkdir kern
cd kern
git init
git fetch git://github.com/raspberrypi/linux.git rpi-3.6.y:refs/remotes/origin/rpi-3.6.y
git checkout rpi-3.6.y

.. and then the tools directory, with the cross-compilation tools (about 220 MB):

git clone git://github.com/raspberrypi/tools.git

Patch to add PPS support over GPIO pin 18

Starting from the kern directory, with the tools and linux-rpi-3.6.y sub-directories:

wget  https://raw.github.com/lampeh/rpi-misc/master/linux-pps/linux-rpi-pps-gpio-bcm2708.diff
HERE=$(pwd)
cd  linux-rpi-3.6.y/arch/arm/mach-bcm2708/
patch --backup bcm2708.c < $HERE/linux-rpi-pps-gpio-bcm2708.diff

should produce a message like:

got Hunk #3 succeeded at 756 (offset 39 lines)

then back to where we started:

cd $HERE

Command script to compile kernel and modules

You will need either to define your configuration from scratch, which has literally hundreds of options, or to use one from a working Raspberry Pi. On the Raspberry Pi:

zcat /proc/config.gz > raspi.config

and then transfer the text file raspi.config to your Linux PC, placing it in the kern directory.

I wrote a small script named build-k to automate the steps to compile the kernel and the OS modules. Your home directory will be different, of course. As I was working on a quad-core PC, I allowed the compilation to use 2 processors, adding the -j3 switch. To make the script executable:

    chmod a+x build-k

To run the script, enter:

./build-k

#!/bin/bash

KERNEL_SRC=/home/david/kern/linux-rpi-3.6.y
export KERNEL_SRC

CCPREFIX=/home/david/kern/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-
export CCPREFIX

cd $KERNEL_SRC
make mrproper
cp /home/david/kern/raspi.config .config
make ARCH=arm CROSS_COMPILE=${CCPREFIX} oldconfig
# make ARCH=arm CROSS_COMPILE={$CCPREFIX} menuconfig
make ARCH=arm CROSS_COMPILE=${CCPREFIX} -j3
make ARCH=arm CROSS_COMPILE=${CCPREFIX} modules -j3

Changes for kernel version 3.10

2013-Dec-21 tried compiling kernel 3.10. The .gz download is 113 MB.

mkdir kern
cd kern
wget https://github.com/raspberrypi/linux/archive/rpi-3.10.y.tar.gz
tar xvfz rpi-3.10.y.tar.gz

and then the tools directory if you don't already have it (about 220 MB):

git clone git://github.com/raspberrypi/tools.git

Next, the patch:

wget  https://raw.github.com/lampeh/rpi-misc/master/linux-pps/linux-rpi-pps-gpio-bcm2708.diff
HERE=$(pwd)
cd  linux-rpi-3.6.y/arch/arm/mach-bcm2708/
patch --backup bcm2708.c < $HERE/linux-rpi-pps-gpio-bcm2708.diff

.. and you get two messages rather than the one on 3.6.y

Hunk #2 succeeded at 379 (offset -41 lines).
Hunk #3 succeeded at 782 with fuzz 1 (offset 56 lines).

.. and back to where we started:

$ cd $HERE

To make a default configuration file

I don't have a default configuration file for 3.10.y, but someone suggested the following to make the default kernel configuration into the "active" one:

#v+
# arch/arm/configs/bcmrpi_defconfig
export PLATFORM=bcmrpi
ARCH=arm CROSS_COMPILE=${CCPREFIX} make ${PLATFORM}_defconfig
#v-

I interpreted this as the following steps:

Go into the kernel source directory:
cd /home/david/kernel/linux-rpi-3.10.y
Create and export the CCPREFIX variable:
CCPREFIX=/home/david/kern/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-
export CCPREFIX
Create and export the PLATFORM variable:
PLATFORM=bcmrpi
export PLATFORM
Run make to create a .config file based in the defaults:
ARCH=arm CROSS_COMPILE=${CCPREFIX} make ${PLATFORM}_defconfig
You then end up with a file .config in the source directory with all the defaults set.
You can then run:
make ARCH=arm CROSS_COMPILE={$CCPREFIX} menuconfig
to set the kernel options.

I actually ran the commands from the script I give below, commenting out the commands one at a time, starting with them all commented out, so I could run the procedure and gradually build up confidence that all as OK. If you are happy with the changes, it may be a good idea to save the changed file in your kernel directory - something like:

cp linux-rpi-3.10.y/.config raspi-3.10.config

Use the script as a guide!

Setting the kernel options

Setting the better kernel options for using NTP, not tick-less system, add PPS support:

Make the system not tick-less:

General setup --->
  Timers subsystem --->
    Timer tick handling --->
      Periodic timer ticks: checked

Add PPS support:

General setup --->
  Device Drivers --->
    PPS support --->
      <M> PPS support
        *** PPS clients support
      <M> PPS client using GPIO

Add memory mapped GPIO driver support:

General setup --->
  Device Drivers --->
    GPIO support --->
      <*> Generic memory-mapped GPIO controller support

#!/bin/bash

KERNEL_SRC=/home/david/kernel/linux-rpi-3.10.y
export KERNEL_SRC

CCPREFIX=/home/david/kernel/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-
export CCPREFIX

PLATFORM=bcmrpi
export PLATFORM

cd $KERNEL_SRC
make mrproper
# cp /home/david/kernel/raspi.config .config
ARCH=arm CROSS_COMPILE=${CCPREFIX} make ${PLATFORM}_defconfig
# make ARCH=arm CROSS_COMPILE=${CCPREFIX} oldconfig
make ARCH=arm CROSS_COMPILE={$CCPREFIX} menuconfig
make ARCH=arm CROSS_COMPILE=${CCPREFIX} -j3
make ARCH=arm CROSS_COMPILE=${CCPREFIX} modules -j3

MODULES_TEMP=~/modules
export MODULES_TEMP
make ARCH=arm CROSS_COMPILE=${CCPREFIX} INSTALL_MOD_PATH=${MODULES_TEMP} modules_install

Installing the newly compiled kernel

Well this is the part i had been putting off, as there seemed to be a number of obstacles in the way, including the fact that the test system I had was installed from a NOOBS download, which was reported to partition the SD card in a different way to usual. Of course, I had taken an image backup of the working card just in case....

I used a USB SD card reader, and attached it to the Windows PC while the Linux VM was powered up with me logged in. Imagine my surprise when a file manager window opened, pointing to all the required directories on the SD card! This is going to be easy, or so I thought!

The first step was to locate the newly-compiled kernel - it's in the file Image in the directory arch/arm/boot/. On my compile of 3.10.24 the file was 6.1 MB in size. I copied the file to a more meaningful name:

cp Image kernel-3.10-pps.img

I could then take the steps below to make the Raspberry Pi boot from the new kernel;

Copy the file Image kernel-3.10-pps.img into the Boot directory. You should see other files there, including one named kernel, so that you know that's the right directory.
Edit the file config.txt to change (or in my case, add) an entry pointing to the new kernel:
kernel=kernel-3.10-pps.img

Copying the compiled modules was a little more difficult as I didn't seem to have permission to write the the root directory. Open another file manager window in superuser mode (sudo nautilus from the terminal). You should then be able to copy the lib directory from the modules directory to the lib directory in the root lib directory on the card. I actually copied the two sub-directories firmware and modules one at a time, being careful I was in the right location each time. In each case I used the "merge" option so that my newly compiled stuff was added to that already on the card (which was at 3.10.24+).

It worked! And I was able to use the PPS on the Raspberry Pi once I had edited /etc/modules to include the line: pps-gpio.

As I was installing a 3.10.24 kernel on an existing 3.10.24+ system, I didn't bother updating the firmware.

NTP cross-compile

This work is based on the cross-compile instructions given here, and with an awful of of help from the comp.protocols.time.ntp Usenet newsgroup (also available on Google Groups). The Linux installation used was the same as the kernel cross-compile described above, so all the tools and system settings mentioned above will apply. You will also find it helpful to have built NTP at least once on the Raspberry Pi itself, as described here. Downloading and unzipping the NTP source is likely to be essential to determining the host variable mentioned below.

Online instructions

I started with the online instructions here:

https://support.ntp.org/bin/view/Dev/Cross-compilingNTP

Downloading the NTP source

My home directory on the Linux box is /home/david/. I first create a sub-directory ntp, and download the current development version (4.2.7p430 in the example below):

cd ~
mkdir ntp
cd ntp
wget http://archive.ntp.org/ntp4/ntp-dev/ntp-dev-4.2.7p430.tar.gz
tar xvfz ntp-dev-4.2.7p430.tar.gz
cd ntp-dev-4.2.7p430

Determining the strings for the host and build machines

What do you mean by Host and build machines?

Host machine: where you expect the binaries to run
Build machine: the machine you will be compiling on

You need to run the config.guess script in the build directory of the relevant PC. If the config.guess script is not in the build direcory, you will find it in sntp/libevent/build-aux/config.guess. The results for my own systems were:

Host: armv6l-unknown-linux-gnueabihf
Build: x86_64-unknown-linux

(Online references seem to have: arm-bcm2708hardfp-linux-gnueabi for the host when cross-compiling the kernel).

Creating further directories

The instructions then suggest creating an appropriately named sub-directory, based on the host name, which I did, but it results in a rather long path!

mkdir A.armv6l-unknown-linux-gnueabihf
cd A.armv6l-unknown-linux-gnueabihf

This is referred to as the build directory.

Compiling - is a CCPREFIX needed?

From my experience in cross-compiling the kernel, it was necessary to add the path of the cross-compiling tool in front of the existing path, so I made a script which did that. From the CCPREFIX part of the kernel cross-compile, I determined that the appropriate tools were in:

/home/david/kernel/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin

(I am uncertain now why I chose that particular set. I recall a Web page showing different sets for different versions of the kernel, but I can't find it now!). You will recognise this directory from my previous kernel cross-compile. Having prefixed the path, I then export it for the subsequent scripts to use. I then run a make clean to remove any existing compiled code, and follow it with a ../configure command with two important parameters:

--host=arm-bcm2708hardfp-linux-gnueabi
--build=x86_64-unknown-linux

corresponding to the host and build settings discovered above. I use a third parameter (--enable-linuxcaps) to try and make the configuration of NTP the same as on the Raspberry Pi, but that parameter appears to be ignored at the moment. It's an outstanding issue. However, I got Intel binaries when following this obvious route, so I needed to dig further.

Configuring the NTP compile

I needed to run the NTP compile configuration command telling it what the build and host machines were:

../configure --host=armv6l-unknown-linux-gnueabihf --build=x86_64-unknown-linux-gnu

However, this failed, with the following lines just prior to failure:

checking if pthread_create() works... yes
checking if select yields when using pthreads... cross
configure: error: crossing compiling: use --with-yielding_select=yes|no|manual

Following advice in the NTP newsgroup, I discovered that I needed to add a further definition to the configure command, having first run configure on the Raspberry Pi to determine what the select() function did. Note that you may need to run "make distclean" if you have already configured the compile on that Raspberry Pi. I redirected the configure output on the RPi to a file, and found that yielding_select needed to be set to "yes". I could then run the configure command again:

../configure --host=armv6l-unknown-linux-gnueabihf --build=x86_64-unknown-linux-gnu --with-yielding_select=yes

but unfortunately the same error message resulted.

Notes from the Raspberry Pi configure

Having redirected the main text from configure on the Raspberry Pi to a file, I was left with the following warnings:

pi@raspi-6 ~/ntp/ntp-dev-4.2.7p422 $ ./configure --enable-linuxcaps > _djt.txt
configure: WARNING: libcrypto and libssl not found in any of /usr/lib /usr/lib/openssl /usr/sfw/lib /usr/local/lib /usr/local/ssl/lib /lib
configure: WARNING: did not find openssl/evp.h in any of /usr/include /usr/sfw/include /usr/local/include /usr/local/ssl/include
configure: WARNING: net-snmp-config present but net-snmp headers are not available!
configure: WARNING: libcrypto and libssl not found in any of /usr/lib /usr/lib/openssl /usr/sfw/lib /usr/local/lib /usr/local/ssl/lib /lib
configure: WARNING: did not find openssl/evp.h in any of /usr/include /usr/sfw/include /usr/local/include /usr/local/ssl/include
pi@raspi-6 ~/ntp/ntp-dev-4.2.7p422 $

Whilst I don't believe they are relevant to the current problem, they may be of interest.

To be resolved

After the configure, the make command does the compiling and linking, and the make install command puts the compiled software into the subdirectories expected for Linux. Why the instructions suggest a directory ":Built" I don't know, but I tried to deviate from the instructions as little as possible!

Finally, I check the contents of the bin and sbin sub-directories - you can look at the date and time to be sure that have just compiled something, and finally I use the file command to determine that the compiled program is actually for the Raspberry Pi, and not for the Intel system on which has been built!

My complete, but non-working, script is here:

PATH=/home/david/kernel/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/:$PATH
export PATH

make clean
../configure --host=arm-bcm2708hardfp-linux-gnueabi --build=x86_64-unknown-linux --enable-linuxcaps
make
make DESTDIR=`pwd`/:Built install

ls -l :Built/usr/local/bin
ls -l :Built/usr/local/sbin
file :Built/usr/local/sbin/ntpd

and I made a script to upload the compiled NTP binaries to a location from which the Raspberry Pi card themselves could access and upload the newly compiled version. I didn't appreciate that the lcd command is like the cd command, and will use relative paths unless absolute paths are specified. Well, they say you learn something every day!

#!/bin/sh USER=<user-name> PASSWD={user-password> ftp -n <ftp-server-address> <<SCRIPT user $USER $PASSWD binary passive prompt off cd /RaspberryPi/ntp-cross/ lcd :Built/usr/local/sbin/ mput * lcd ../bin/ mput * quit SCRIPT