Kernel changes - November 2013

These tests were in response to NTP bug 2314, where I reported that the Raspberry Pi sometimes reported NTP jitter as zero in loopstats, making it somewhat more difficult to see whether any change had brought about an improvement or not.  They involve recompiling the kernel, which is a long-winded process, but one I have documented here (with notes and help from Rob Windgassen).

There are two results presented here, one is a Raspberry Pi card which was using the user-mode PPS option from Folkert van Heusden, where the switch from user-mode to kernel mode is expected to reduce jitter and offset, and the other is a Raspberry Pi which was using the kernel-mode PPS from Hauke Lampe.  Here, the improvements are due to the configuration changes made to the kernel and suggested by Rob Windgassen.

Using the improved kernel configuration from Rob Windgassen

The RMS offset has been reduced from about 0.6 µs to less than 0.3 µs, and the jitter is now being recorded (proving that bug 2314 was due to the kernel configuration, and not a fault in NTP).

However, looking at the frequency reported by the loopstats it seems that it is now being adjusted in steps rather than smoothly, and perhaps this needs further investigation.

Switching from user-mode to kernel-mode

The RMS offset has been reduced from over 1.5 µs to a much lower value (perhaps the 0.3 µs seen above), and the averaged jitter has been reduced from 3.8 µs to about 1.2 µs.  Once again we see the frequency adjustment is no longer smooth, but stepped.

Some thoughts from Jorge Amaral, Portugal

Dear David

I found your webpage this morning and first I would like to congratulate you for your effort in explaining everything very well on how to build an ntp server with a raspi.  Now I want to share with you my effort in trying to improve the precision of the ntp server.

My hardware is composed of a raspi model b and a Adafruit ultimate v3 GPS modem.  While I was copying your work, i decided to use driver 20 for ntp (ntp speaks directly with the gps modem, no more shm).  You can forget running the gpsd.  The only changes that I have made were:

To compile ntp server with NMEA support:

./configure  --disable-all-clocks  --disable-parse-clocks  --without-lineeditlibs  --enable-NMEA  --enable-LOCAL-CLOCK  --enable-SHM  
--enable-linuxcaps  --enable-ATOM

Then after ntp compiled and installed i create a symlink for ntp to read the gps and the pps devices:

cd /dev
ln -sf /dev/ttyAMA0 /dev/gps0
ln -sf /dev/pps0 /dev/gpspps0

and the only thing missing is the ntp.conf slightly changed:

server 127.127.1.0 fudge 127.127.1.0 stratum 10 server 127.127.20.0 mode 18 minpoll 4 maxpoll 4 prefer fudge 127.127.20.0 flag1 1 fudge 127.127.20.0 flag2 0 fudge 127.127.20.0 flag3 1 fudge 127.127.20.0 time2 0.470 server ntp02.oal.ul.pt minpoll 6 maxpoll 8 server ntp04.oal.ul.pt minpoll 6 maxpoll 8 #server 127.127.22.0 minpoll 4 maxpoll 4 prefer #fudge 127.127.22.0 flag2 0 driftfile /var/lib/ntp/ntp.drift logfile /var/log/ntp logconfig=all statsdir /var/log/ntpstats/ statistics loopstats peerstats clockstats

Comment from Pete Stephenson:

Jorge created the necessary symlinks using: cd /dev ln -sf /dev/ttyAMA0 /dev/gps0 ln -sf /dev/pps0 /dev/gpspps0 While that certainly works, there's a slight issue: the symlinks get purged on reboot. A more permanent solution can be achieved using udev, which is used by Raspbian, to automatically create the needed symlinks at boot: Specifically, create the file: /etc/udev/rules.d/09.pps.rules with the following contents: KERNEL=="ttyAMA0", SYMLINK+="gps0" KERNEL=="pps0", OWNER="root", GROUP="tty", MODE="0660", SYMLINK+="gpspps0" Source: http://linuxpps.org/wiki/index.php/LinuxPPS_NTPD_support

Two other things that I have changed were the cpu_scaling and the serial latency:

echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
setserial /dev/ttyAMA0 low_latency

As you can see, it's working with GPS and PPS support, and precision is 2^-20 (1µs)

Once again thank you very much for your work and I hope that this can help you too :-)

With thanks,
Jorge

Cumprimentos
Jorge Amaral, Portugal
Departamento Técnico

Reducing the Ethernet latency

I was delighted to receive this note from Philip Gladstone about reducing the Ethernet latency on the Raspberry Pi.

David

Thanks for putting together all your pages on making a stratum-1 server  on a Raspberry Pi.

It appears that the Ethernet latency on the RPi is fairly large, but it  can be reduced at a slight cost in performance. To do this, edit your:

/boot/cmdline.txt

and add:

smsc95xx.turbo_mode=0

or

smsc95xx.turbo_mode=N

along with all the other parameters that are being set.

This disables a feature of the Ethernet controller that tries to pack  multiple received frames into a single USB transfer. In order to do  this, the controller waits after receiving a frame (in the hope of  receiving another frame) before passing any data back to the host.  Setting turbo_mode to 0 reduced a ping RTT (Round Trip Time) from 500-600 µs down to 300-350 µs.

Enjoy.

Philip

p.s. I was the person that added the kernel timekeeping PLL to Linux and  the adjtimex system call back in '93? Linux 0.99pl13

DJT: I was able to put this to the test recently when I was doing some NTP measurements, and from a Raspberry Pi which had been running for a while, I made the change to cmdline.txt and rebooted, with the following effect on NTP delay as reported by a remote PPS-locked server:

The delay has been reduced from ~0.51 to ~0.35 milliseconds.  Linux ping times showed a similar improvement:

For comparison, here is a similar plot from the BeagleBone Black, which does not have the Ethernet via USB configuration:

Considerations when compiling recent kernel versions

Matthew George NG7M in Syracuse Utah sent the following notes:

Thanks for all your effort and details on your website.  I did want to pass along some info that stumped me for weeks now.  I started into the NTPD and GPSDO obsession after the 3.18.7+ Raspbian kernel source was out... so it was all about device tree and the modules if you are compiling your own version of the kernel.

I have been stumped as to why I could never get modules loading when I compiled my own version of 3.18.13 on a Raspberry Pi 2.  So there are a couple of things you might want to add to your website if someone is hell bent on getting kernel level PPS going with the newer kernel source and device tree etc.

1. For the Raspberry Pi, you need to name the kernel image: kernel7.img by default if you are not overriding the name in /boot/config.txt.  This made me scratch my head for a while... not a big deal, but it was confusing trying to see that I was rebooting with the new Image / zImage that I had just compiled.
     
2. The bigger issue for a dummy like me was trying to figure out why none of the modules would load with my new compiled kernel that I had enabled kernel user PPS support for... Updated the Timer Options and enabled PPS etc... down the device settings in menuconfig.  The usual routine.

Here is the magic trick which maybe you already know about if you are using the RPi 2.  After you compile and link the kernel image, modules will not load unless you run another script that is part of the Linux source distribution to enable Device Tree modules in the kernel.

You must do the following after you have your new image in ../arch/arm/boot:

$ ../../../scripts/mkknlimg Image kernel7.img

Then copy the new kernel7.img to your /boot directory as root...  Bingo!  Shazam...  Finally, I have kernel PPS working with a couple of different GPSDO devices... a Trimble Thunderbolt and a Motorola Oncore UT+ in my case.

Before I was lucky to get consistent microsecond offsets and now I'm consistent with sub microseconds and clearly in the land of nanosecond offsets!  Here is some output from my loopstats file for example with kernel PPS running with 3.18.13: (ntpd up maybe 20 minutes):

57165 85453.930 -0.000000019 -6.178 0.000001206 0.001265 3
57165 85461.930 0.000000198 -6.178 0.000001418 0.001184 3
57165 85469.930 0.000000820 -6.177 0.000002364 0.001117 3
57165 85477.930 0.000000200 -6.177 0.000001489 0.001045 3
57165 85485.930 0.000000625 -6.177 0.000001315 0.000983 3
57165 85493.930 0.000000490 -6.177 0.000002125 0.000924 3
57165 85501.930 0.000000559 -6.177 0.000001567 0.000870 3
57165 85509.930 0.000000860 -6.176 0.000002316 0.000826 3
57165 85517.930 0.000000269 -6.176 0.000002903 0.000775 3
57165 85525.930 -0.000000412 -6.176 0.000002248 0.000728 3
57165 85533.930 -0.000000479 -6.176 0.000002118 0.000686 3
57165 85541.930 0.000000179 -6.176 0.000001374 0.000643 3
57165 85549.930 0.000000644 -6.176 0.000001699 0.000612 3
57165 85557.930 0.000000009 -6.176 0.000001122 0.000572 3
57165 85565.930 -0.000000176 -6.176 0.000000947 0.000536 3
57165 85573.930 -0.000000222 -6.153 0.000002364 0.008075 3

This is with a maxpoll at 3 as you can see... not much difference with maxpoll 4 with PPS now in my experience.

Anyway Google: *mkknlimg* and you will find the posts about needing to run this script on your new 3.18.7+ kernel for the raspi 2 to get modules working / Device Tree working.  My first RPi was an RPi 2, so no experience with the previous RPi models.

I hope this might help someone else that comes along.  Your website is a fantastic resource and I hope you find my details above worth posting... :)

Best Regards David.

Thank You

Matthew George
NG7M
Syracuse Utah
USA

 

Copyright © David Taylor, Edinburgh

@gm8arv on Twitter

Last modified: 2016 Mar 06 at 09:01