Feb 19

Intergalactics Source Code

Does anybody out there have the original source code to the Java game Intergalactics?  I was able to pull the (compiled) client off of SourceForge, but without a server it’s not too useful.  I did start updating the client to work properly over the summer along with a new server implementation, but it would still be interesting to get all of the original source code.

Anyway, if you do happen to have the original code, I would be grateful.  Intergalactics was always a nice fun timewaster.  It wasn’t too complicated but it did require a certain amount of strategy.

Dec 09

APT Repo is now live

Today, I created an APT repo for my projects.  At the moment, this hosts only CSerial, however the intention is to put some other projects up at some point.  Note that because CSerial is built as both amd64 and armhf in the same repository, you may need to give the exact version to APT when installing: apt-get install cserial-dev=version

Versions can be seen by using the following APT command: apt-cache policy cserial-dev

There are actually two APT repos: one for nightly builds, and one for the releases.  As of right now, nothing is in the releases, as I need to fix a few bugs before that happens.

Everything in these APT repos is built from Jenkins.

The main website can be seen here: http://apt.rm5248.com/

Sep 03

The Demise of Juicero

As you may have heard recently, Juicero is shutting down.  While not entirely unexpected, their business model just seemed insane to start with.  As recounted in some of the Slashdot comments, Juicero went through $100+ million dollars in funding.

This brings up an obvious question:


Really.  Over 100 million dollars to create a device that does nothing but squeeze juice?  Give me only ten million and I will fail faster for you!  I’m not quite sure what these guys were doing with all that money(although AvE’s video seems enlightening), but I’m beginning to think that they were simply trying to go too far at once.  Instead of making a small, feasible product at first, they went and sunk a whole lot of money into design and not a whole lot into making an actually reasonable device.  This is just insane the amount of money spent on this.

From the VC point of view, probably what they were thinking is “continuous revenue stream”, which would be reasonable from their view of getting money back.  While the premise may have good, the actual execution was very lacking.  After all, “aim for the stars, hope not to get the exact polar opposite of what you were actually trying to accomplish.”

Jul 21

NTP Woes

For the past year or so, I’ve been battling trying to get NTP fed from GPSD.  This has proven to be somewhat harder than I imagined.  GPSD is supposed to feed NTP directly from shared memory, however this was not happening.  I haven’t been trying to do this the entire time, but it’s been a problem for at least a year!

The simple situation is as follows: I have a device that has a GPS receiver on it in order to get the time.  NTP is also running to sync the time whenever it connects to the internet(for most of the time, we are not on the network).  This is also a very vanilla Debian 8 (Jessie) system on an ARM processor, so we’re not custom-compiling any of the standard packages.  The GPS is also a hotplug, so we have a udev rule that calls gpsctl with the proper TTY when the USB is activated:

(udev match options here) SYMLINK+="gps%n", TAG+="systemd", ENV{SYSTEMD_WANTS}="gpsdctl@%k.service"

According to all of the literature on the web, we should just be able to do this by adding the following to /etc/ntp.conf:

#gpsd SHM
server prefer
fudge refid GPS flag1 1

(note that we are setting flag1 here, as the system that this is running on has no clock backup).

This allows NTP to read from a shared memory address that is fed by GPSD.  You can check this with ntpq -p:

     remote           refid      st t when poll reach   delay   offset  jitter
 SHM(0)          .GPS.            0 l    -   64    0    0.000    0.000   0.000
-altus.ip-connec    2 u    7   64    1  189.620    6.525   5.288
+ks3370497.kimsu    2 u    5   64    1  186.252   17.282  14.504
+head1.mirror.ca    2 u    5   64    1  186.503   15.792  14.691
*de.danzuck.eu   2 u    4   64    1  155.953    8.786  13.366

As you can see, in this situation there is no connection to the SHM segment, but we are synced with another NTP server.  However, we can verify that GPSD is running and getting GPS data by running the ‘gpsmon’ program(in the ‘gpsd-clients’ package).

The other important thing to note about the SHM segement is that the ‘reach’ value is always 0, and will never increase.  You can also check the output of NTP trying to reach it with:

 ntpq -p

This was very confusing to me: how are we not talking with the SHM segment?  This would also seem to work on some systems, but not other systems.  No matter how long I left it running, NTP would never get any useful data from the SHM segment.  Finally, I stumbled across a link, which I can’t find now, that said that GPSD must be started before NTP.  I then tried that on my system, by stopping my software(which is always reading from GPSD), stopping NTP and then stopping GPSD.  I then started GPSD, NTP, and my software(which will also initialize the GPS system to start sending commands).  This causes NTP to consistently sync with GPSD.

Do this like the following:

$ systemctl stop your-software
$ systemctl stop ntp
$ systemctl stop gpsd
$ .. make sure everything is dead..
$ systemctl start gpsd
$ systemctl start ntp
$ systemctl start your-software

For reference, here’s the ntp.conf that I am using(we only added the SHM driver from the standard Debian install):

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift

# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example

#gpsd SHM
server prefer
fudge refid GPS flag1 1

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict ::1

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict mask notrust

# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth

TL;DR Having trouble feeding NTP from GPSD when you have a good GPS lock?  Make sure that GPSD starts before NTP.

Apr 09

QTest Project Setup

Recently, I was looking into how to setup a Qt project with unit tests.  The Qt-esque way to do this is to use the QTest framework within Qt in order to do this.  While not a bad framework, the problem is that there isn’t a good way to integrate it within your project.  What I mean with that is that you must link with the QTest library and have a special main() method in order to run the tests.  This brings up two problems:

  1. How do we run the tests if we have only one executable?
  2. How do we only link with QTest at certain times(e.g. we don’t want to link with QTest when we send the executable out)

Searching for a solution to the problem, one thing that some people did was to create a .pri file with the source files to compile into two projects: one for the main application, and one for the unit tests.  The disadvantages to this are that it makes it hard to work with Qt Creator to add in the files, as it can’t add them in automatically to the SOURCES and HEADERS list.  So that’s not good.

The other solution that I came across was to add in the files to each project.  However, this adds in a lot of overhead, and you must be sure to add in the files to both projects when something changes.  So that’s not good either.

The solution that I came up with was to have three separate projects:

  • One project for the main code of our project.  This gets built as a static library.
  • One project to run the main code of our project.  This is the normal executable.
  • One project to run the unit tests.  This links with QTest, and doesn’t get installed by default.

This is basically the same project setup as described here, although it wasn’t until after I had created my project that I realized it was the exact same thing(I did come across the link at first, but didn’t understand it at the time since it wasn’t using QTest).

With this project setup, you can edit the main code in Qt creator, do anything that you want, and still have everything link and test in a clean manner(no need to link with QTest in your main executable!)

The code for the same project is up on github, to give you a general overview of how it all fits together.

Feb 04

Microchip Harmony and Bootloaders

I’ve been messing around with a Digilent Max32, and have been using Microchip’s MPLAB Harmony framework to program the device.  The framework is rather nice, but it can be a little confusing at times.  It took me a while to figure out how to work the bootloader on it, so this post will go into how to setup your projects.


  1. PIC32 based device.  In our case, this will be the Max32, but any PIC based device should work fine.
  2. MPLAB X, Microchip’s dev platform based on Netbeans.
  3. Microchip XC32 compiler.  This should be a part of your MPLAB X install, but if not you can download it from Microchip’s website.  I am using version 1.42.
  4. PIC Programmer.  I have an ICD3, but it should be possible to use a PICKit programmer as well.

Part I: Install Harmony

Before we can mess around with Harmony, we need to install the latest version.  In our case, we will be using the current beta version of 2.0.

  1. Download the new Harmony Configurator from Microchip’s website.
  2. Go to the ‘Downloads’ tab, and download the proper version for your system

    Step 1 and 2 – Download Harmony and Configurator

  3. Run the installer for Harmony.  Make sure that you remember where you installed it, you will need this information later.
  4. In MPLAB X, go to Tools->Plugins.

    Plugins location

  5. Go to the ‘Downloaded’ tab
  6. Click ‘Add Plugins’
  7. Browse to where you downloaded the Harmony plug-in to, and select it.

    How to install the downloaded plugin

  8. The plugin will now show up in your Plugins window.  Click the ‘Install’ button.
  9. The plugin will install.  Once it does, you will have to restart MPLAB X.

Part II: Create Bootloader project

Note that before you come here, you must have MPLAB X Harmony and the proper Harmony plugin installed.

  1. Create a new project by going to File->New Project.
  2. Select ’32-bit MPLAB Harmony Project’ and click ‘Next’

    Basic project configuration

  3. Fill in all of the information on this screen.  Don’t forget to set the PIC device that you are using.
  4. If the Harmony configurator does not open up for you automatically, go to Tools -> Embedded -> MPLAB Harmony Configurator

    Open up the Harmony Configurator if it closes

  5. Under the ‘Application Configuration’ section, we want to create an application(we will call it max32_bl) and click ‘Generate application code for selected Harmony components’.  We want to generate code for the bootloader in this area.

    Bootloader configuration #1

  6. Under the ‘Harmony Framework Configuration’ section, select ‘Bootloader Library’.  We will be using the USART bootloader.  Under the ‘Bootloader or Applicaton?’ section, select ‘Bootloader’.  Also select ‘Ports’ under ‘System Services’

    Bootloader configuration #2

  7. Click the ‘Generate Code’ button on the Configurator screen.  We will have a number of new C/H files for us to edit.  Also, there is now a custom linker script for us under the ‘Linker Files’ folder.
  8. Now that code has been generated, there are a few things that we want to do to make it clear what is happening.  First, we don’t need the legacy bootloader options.  So under ‘Header Files’, go to app/system_config/default/system_config.h, and comment out the preprocessor macros BOOTLOADER_LEGACY and BTL_SWITCH

    Delete/comment out these macros

  9. The first thing for us to edit in the code is to open up our max32_bl.c file.  This file contains generated code for us to modify.  The first thing is that we need to move the line that says

    to be in the


    function.  Otherwise, the bootloader code will not call our ForceEvent function.

  10. Now, let’s add a splash screen.  For now, we can simply add a simple function to print out to the UART, and then call that function once in the MAX32_BL_Tasks function, in the MAX32_BL_STATE_INIT case statement of our switch.  Here’s the simple UART printing function:
    static void writeStringToUART( const char* string ){
     int pos = 0;
     while( string[ pos ] != 0 ){
     while( DRV_USART0_TransmitBufferIsFull() ){}
     DRV_USART0_WriteByte( string[ pos++ ] );
  11. You should now be able to program your device with the bootloader.  Open up a terminal with a terminal emulator such as Putty, and if you have done everything correctly you should see a splash screen whenever you reset the processor.  On the MAX32, this is accomplished by pressing the reset switch.
  12. Let’s also make sure that we can get to the bootloader manually.  To do this, we will have to configure one pin on the chip as a GPIO input, with a pull-up resistor.  Go back to the Harmony Configurator screen, and go to the ‘Pin Settings’ tab.

    Pin settings

  13. Choose a pin to use as an input.  In my case, I chose RG6.  This corresponds to pin 52 on the Max32 board.  I will also enable the pull up resistor on this pin, so that when the pin is shorted to ground the bootloader will start.
  14. Go back to our max32_bl file, and modify MAX32_BL_Bootloader_ForceEvent to look like the following(change your pin if you used a different one).
    int MAX32_BL_Bootloader_ForceEvent(void)
     if( !rg6 ){
     writeStringToUART( "Button held, forcing bootloader\n" );
     return 1;
     /* Check the trigger memory location and return true/false. */
     return (1);
     return (0);
  15. Build and re-flash your device.  When you short pin RG6 to GND, the string “Button held, forcing bootloader” should now print out on your console when the bootloader starts.

Here’s the entire file for reference:

  MPLAB Harmony Application Source File
    Microchip Technology Inc.
  File Name:

    This file contains the source code for the MPLAB Harmony application.

    This file contains the source code for the MPLAB Harmony application.  It 
    implements the logic of the application's state machine and it may call 
    API routines of other MPLAB Harmony modules in the system, such as drivers,
    system services, and middleware.  However, it does not call any of the
    system interfaces (such as the "Initialize" and "Tasks" functions) of any of
    the modules in the system or make any assumptions about when those functions
    are called.  That is the responsibility of the configuration-specific system

Copyright (c) 2013-2014 released Microchip Technology Inc.  All rights reserved.

Microchip licenses to you the right to use, modify, copy and distribute
Software only when embedded on a Microchip microcontroller or digital signal
controller that is integrated into your product or third party product
(pursuant to the sublicense terms in the accompanying license agreement).

You should refer to the license agreement accompanying this Software for
additional information regarding your rights and obligations.


// *****************************************************************************
// *****************************************************************************
// Section: Included Files 
// *****************************************************************************
// *****************************************************************************

#include "max32_bl.h"

#include <stdio.h>

// *****************************************************************************
// *****************************************************************************
// Section: Global Data Definitions
// *****************************************************************************
// *****************************************************************************

// *****************************************************************************
/* Application Data

    Holds application data

    This structure holds the application's data.

    This structure should be initialized by the APP_Initialize function.
    Application strings and buffers are be defined outside this structure.

MAX32_BL_DATA max32_blData;

static void writeStringToUART( const char* string ){
    int pos = 0;
    while( string[ pos ] != 0 ){
        while( DRV_USART0_TransmitBufferIsFull() ){}
        DRV_USART0_WriteByte( string[ pos++ ] );

static void writeIntAsCharToUART( int i ){
    while( DRV_USART0_TransmitBufferIsFull() ){}
    DRV_USART0_WriteByte( i + 48 );

static void waitForUARTFlush(){
    while( !PLIB_USART_TransmitterIsEmpty( USART_ID_1 ) ){}

// *****************************************************************************
// *****************************************************************************
// Section: Application Callback Functions
// *****************************************************************************
// *****************************************************************************
    static void MAX32_BL_Bootloader_ForceEvent (void)
    Sets a trigger to be passed to force bootloader callback.
	Run bootloader if memory location == '0xFFFFFFFF' otherwise jump to user 
int MAX32_BL_Bootloader_ForceEvent(void)
    if( !rg6 ){
        writeStringToUART( "Button held, forcing bootloader\n" );
        return 1;
    /* Check the trigger memory location and return true/false. */
        return (1);
		return (0);

/* TODO:  Add any necessary callback functions.

// *****************************************************************************
// *****************************************************************************
// Section: Application Local Functions
// *****************************************************************************
// *****************************************************************************

/* TODO:  Add any necessary local functions.

// *****************************************************************************
// *****************************************************************************
// Section: Application Initialization and State Machine Functions
// *****************************************************************************
// *****************************************************************************

    void MAX32_BL_Initialize ( void )

    See prototype in max32_bl.h.

void MAX32_BL_Initialize ( void )
    /* Place the App state machine in its initial state. */
    max32_blData.state = MAX32_BL_STATE_INIT;

    /* TODO: Initialize your application's state machine and other
     * parameters.

static void printSplash(){    
    writeStringToUART( "MAX32 Bootloader\n" );
    writeStringToUART( "Version " );
    writeIntAsCharToUART( MAJOR_VERSION );
    writeStringToUART( "." );
    writeIntAsCharToUART( MINOR_VERSION );
    writeStringToUART( "\n" );
    writeStringToUART( __DATE__ );
    writeStringToUART( " " );
    writeStringToUART( __TIME__ );
    writeStringToUART( "\n" );

    void MAX32_BL_Tasks ( void )

    See prototype in max32_bl.h.

void MAX32_BL_Tasks ( void )

    /* Check the application's current state. */
    switch ( max32_blData.state )
        /* Application's initial state. */
        case MAX32_BL_STATE_INIT:
            bool appInitialized = true;
            if (appInitialized)
                max32_blData.state = MAX32_BL_STATE_SERVICE_TASKS;
                //print splash screen.  this will happen only once


        /* TODO: implement your application state machine.*/

        /* The default state should never be executed. */
            /* TODO: Handle error in application's state machine. */


 End of File

Part III: Create our main program

Now that we have our bootloader setup, we need to create the main program that will get loaded by the bootloader.

  1. Create a new project by going to File->New Project.
  2. Select ’32-bit MPLAB Harmony Project’ and click ‘Next’
  3. Fill in all of the information on this screen.  Don’t forget to set the PIC device that you are using.
  4. If the Harmony configurator does not open up for you automatically, go to Tools -> Embedded -> MPLAB Harmony Configurator
  5. Under the ‘Application Configuration’ section, we want to create an application(we will call it max32_program) and click ‘Generate application code for selected Harmony components’.  We want code for the Timer and USART to give us some code to work with.

    Main program configuration #1

  6. Under the ‘Harmony Framework Configuration’ section, select ‘Bootloader Library’.  We will be using the USART bootloader.  Under the ‘Bootloader or Applicaton?’ section, select ‘Application’

    Main program configuration #2

  7. Under the ‘Harmony Framework Configuration’ section, also make sure that you have drivers for the timer and USART 0 selected.  The USART should be running at the same baud rate as the bootloader.
  8. Go to the ‘Pin Settings’ page as we did before.  For the MAX32, pin RC1 is an output that drives an LED.  We will use RG6 as an input as we did before on the botloader.  Setup those settings.

    Pin diagram for the main program

  9. Click ‘Generate code’
  10. Your project is now created with some basic information.  Since we set up our timer to use interrupts, we now have a TimerCallback function in mainapp.c.  Let’s modify it to toggle the RC1 LED every time the timer is called.  The code will now look like this:
    static void TimerCallback ( uintptr_t context, uint32_t alarmCount )
     static int state = 0;
     state = !state;
  11. Build your project.  The hex file that is generated is what you want to send to the bootloader.  Note that this hex file can’t be sent as-is, you must convert it to binary and use Microchip’s protocol.  See the next section for some discussion.

Part IV: Bootloading

Now that we have our bootloader and a program, we should try to actually load information onto the device using the bootloader.  To do this, we will need a program to write the information.  You can use either Microchip’s version in AN1388, or I wrote a version that is cross-platform compatible using my CSerial library.

Whichever method you use, once you have built the actual program you need to send the hex file across to get the bootloader to write the data.  If everything has gone correctly, you will then be able to re-set the board and have your application running.

One more tip: If you can program the entire chip at once(e.g. you have an ICD3), if you set the bootloader project as a loadable of your main project you can flash both projects to the chip at the same time.  Since they get flashed in different locations, there’s no conflict.  Note that this won’t work with the lower-end programmers.


This tutorial probably skims over a few steps, but it should at least be enough to get people started and working with Harmony and its bootloader.  Questions? Add a comment and I will see about expanding upon either this post or a later one.

Jan 21

Thoughts on Star Wars and Jedi

So I saw Rogue One last week, and I found it to be quite interesting.  It wasn’t my favorite movie, but it was a good lead in to A New Hope.  Once I got back home, I had to look up the crawl for A New Hope, which of course I did on TV tropes.  Looking at the TV tropes page for A New Hope, there is this one section that I feel is somewhat out of place:

  • Admiral Motti’s description of the Force as a “sad devotion to that ancient religion” seems downright bizarre, given that its existence was treated as common knowledge in the Prequel Trilogy, a timeframe in which Motti would likely have been alive (albeit very young). This is largely because when the film was written Lucas envisioned exact knowledge of the Force and Jedi powers to be something which only a select few had knowledge of, which was gradually contradicted by the Expanded Universe novels and comics, and then jettisoned altogether by The Phantom Menace. Some later novels, such as the Republic Commando novels, took some steps to try and square the two perspectives, but with limited success.

The “sad devotion to that ancient religion” actually seems like it could be a plausible thing for him to say, even though the bullet point goes on to say that this was contradicted with later works.  Why would this work?  Well, let’s do some basic math here and get some rough estimates of how many people there are.

First order of business: How many Jedi are there?  The movies don’t say for certain, but the Jedi temple seems to be quite large.  Making grand assumptions here, let’s also assume that it is large enough to fit every Jedi at once in it.  Given that it seems to encompass an entire city block at least, let us say that the Jedi temple can fit 100,000 people.  (According to scifi.stackexchange.com, there were about 10,000 Jedi at the time of Order 66).  Regardless, 10,000 vs 100,000 Jedi will make no difference here in a moment.

Next question: How large is Coruscant?  Assuming that it is the same size as Earth, and has a population density at least as high as the highest city on Earth(according to Wikipedia, that is Manila at 41,515 people/km²).  According to Wolfram|Alpha, the Earth is 5.1×10^8 square kilometers.  Now, multiplying these two numbers together we get a population of ~2.1×10^13.  According to Wookieepedia, there is more than 1 trillion(1×10^12) people on Coruscant.  If I am doing the math correctly here, my estimate for the number of people on Coruscant is (one order of magnitude * 2 ) from what Wookiepedia says.

Now that we have our numbers(both unofficial and estimated), let’s do some quick math here to figure out about how many Jedi there are per-person on Coruscant.

Number of Jedi Population of Coruscant People-per-Jedi
10,000 1×10^12 100,000,000
10,000 2.1×10^13 2,100,000,000
100,000 1×10^12 10,000,000
100,000 2.1×10^13 210,000,000

Now, obviously these numbers are rather rough estimates.  But at a minimum, there are ten million people on Coruscant per Jedi!

Now, back to the original question: Why is the statement “sad devotion to that ancient religion” completely plausible in my mind?  Because given the ratio of people to Jedi on Coruscant alone, it is likely that you could go an entire lifetime without seeing a real Jedi.  Don’t forget as well, there are more planets than just Coruscant.  Basically, the number of Jedi for the population of the galaxy is laughably small.  If you combine the lack of population with potentially a good PR campaign, it is completely plausible that a person could grow up in the galaxy and never see a Jedi, only hear about them, and then be convinced that Jedi are bad people who were controlling the fate of the galaxy in their hands and/or they were just pretending.  Reportedly, there was a minister in Pakistan who fooled the entire government for 6 years that he wasn’t an actual minister(note: I can’t find any other sources to this from Western media, so I don’t know the veracity of this).

So, to conclude: It is reasonable to assume that because there were so few Jedi for the entire population of the galaxy, any effort to discredit them could be very effective at wiping their knowledge from the population’s memory.

Jan 07

Jenkins in the Cloud

At work, we use Jenkins to automatically build our software.  We use it to build both Java and C++ programs.  Since it’s useful to have a continuous integration server setup to build and test software, I figured that I would document a bit about what I have done on a new VM that I have setup.

This post is going to go into several different directions and technologies, so it may be a little rambling.

Setting up the VM

In order for us to create a hosted instance of Jenkins, the easiest thing to do is to create a VM for us to install Jenkins on.  Since many hosting providers don’t allow you to run Java on shared hosting, our options are to either create a VPS or create a cloud server.  I chose to create a cloud server with DreamCompute, as my web hosting is already through Dreamhost.  This allows us to create a new VM that we have full control over.  The normal VPS with Dreamhost is at some level still a shared system – Ubuntu only.  My preference is for Debian(I have had many weird issues with Ubuntu before).  Note that Jenkins will probably fail if it has less than 2 GB of RAM.  My first instance only had 512 MB of RAM, and as soon as I started to build something that required a bit of memory the VM immediately started thrashing.  My load average went to 11 with only 1 CPU!  (this means that I had enough jobs to keep 11 CPUs busy)

I launched a new Debian 8 image on DreamCompute and created a little script to install the needed things for Jenkins and a few other dependencies that I needed.  This also sets Jenkins up to run on port 80, so that we can access it directly through our browser.

Now that we have Jenkins installed, we need to SSH in to finish the install.

Finish Setup VM

Now that the VM has been mostly setup, we need to SSH in to finish the install.  The first thing I had to do was to finish the install of some needed parts(obviously, you have to wait until the setup script finishes before you can go to this step).

apt-get install cowbuilder

The next thing that we have to do is to setup Jenkins for the first time.  Get the initial admin password from the server:

root@jenkins:/home/debian# cat /var/lib/jenkins/secrets/initialAdminPassword

This hex string is our default password for logging into Jenkins.  We need to go to our public IP address to set this up.  In my case, it is  Once I got there, I put in the admin password and setup Jenkins for the first time.

DNS Setup

Accessing the server by the IP is not particularly useful.  Fortunately, DreamHost has a quick guide on how to setup your DNS entries.  I then put the server at jenkins.rm5248.com.

Jenkins Jobs

I have quite a few projects on my GitHub.  Half of this endeavor is to make a nice place where I can make sure that software is building cleanly.  So I have added a few projects to have them build automatically.

C/C++ Projects

Now to the other big reason for doing this: enabling automatic builds and automatic packaging.  Building Debian packages can be a little tricky. Some tools, like jenkins-debian-glue exist, but they are not the greatest for configuration and usage.  Problems that exist with jenkins-debian-glue:

  • Script-based.  This is not too bad, but it means that it is hard to split builds across multiple nodes.  For example, building ARM packages on an ARM node in my testing is almost 10x faster than emulating ARM on x86
  • Configuration is not clear.  There are separate scripts to generate git snapshots and svn snapshots.   Not all of the configuration variables are documented either.
  • Requires a lot of manual configuration.
  • The default setup requires you to have two Jenkins jobs per project.  This can quickly go overboard and lead to an explosion of jobs.
  • Outputs of the build are not automatically archived to be able to be downloaded through the web interface

To be clear, jenkins-debian-glue is not the worst way to build Debian packages – but it could be a lot better.  So I wrote my own Jenkins plugin to do it.

Debian Pbuilder

This plugin is designed to make building of Debian packages very simple and straightforward.  Because it’s a plugin, it also makes running on other Jenkins nodes practical.  It is also in some ways a port of jenkins-debian-glue to Java, instead of being shell scripts.  In that regard, it uses cowbuilder to create the chroot environment and does build everything in a clean environment.  The actual Jenkins plugin for building packages doesn’t support this behavior at all, which makes builds awkward.

Anyway, this plugin fixes these issues and does everything in a nice way.  If there are people who would like to test out this plugin, that would be very useful.  Since it is built through our Jenkins instance, we can create a new build of the project at any time.  Note that as of right now, you have to install the plugin manually through Jenkins, it is not on the main Jenkins git to be easily installed at this moment.

More information on how to use Debian-pbuilder will be coming soon. 🙂

Other Nodes

This is not strictly related to Jenkins in the cloud, but one thing that we have done at work is to get a dedicated ARM computer in order to build our ARM packages with Debian-pbuilder. We use the NVidia Jetson TK1 board as a Jenkins node.  This greatly decreases our build time from almost 1 hour to only 10 minutes!  The problem with what we were doing before was that because of how Debian-pbuilder and jenkins-debian-glue work, they create an (emulated) chroot environment on the system.  This is slooooooooooow.  Making the chroot on actual hardware greatly speeds up the building of projects.

Dec 01

SVN Commit Problems – Update

An update to my previous problem with checking into SVN.  I had the same problem just now:

$ svn commit -m "message"
Sending        debian/control
Transmitting file data .

Committed revision 9488.

Warning: post commit FS processing had error:
sqlite[S5]: database is locked

I SSH’d into the server to try and figure out the problem.  I then stumbled across this post on the SVN mailing list.  Checking the rep-cache.db, it has two commits less than what the current revision number is, which would match up with my two failed commits, so that seemed suspicious.  I then figured out if some process had the rep-cache.db open at all:

$ lsof | grep rep-cache

As it turns out, Atlassian Fisheye had a bunch of connections open to this database for some reason.  I restarted Fisheye to kill all those connections, so we will see if that fixes the problem.  Note that this is also running an old version of Fisheye(3.2.3), so it could be a problem with just this version.

Nov 28

SVN Checkout By Date Problems

Have you ever had to checkout an SVN revision by the date and gotten an error like the following?

bash$ svn co -r {2016-11-28} https://example.com/svn/foo
svn: E175002: Unexpected HTTP status 500 'Internal Server Error' on '/svn/!svn/me'

svn: E175002: Additional errors:
svn: E175002: REPORT request on '/svn/!svn/me' failed: 500 Internal Server Error

(This is often the case with Jenkins, see this issue for some more details)

Why Jenkins checks out revisions by date is another matter(it’s stupid), but what is the original problem in the first place?  Well, if we look at the Apache error_log on the server, it will look something like the following:

[Mon Nov 28 08:01:24 2016] [error] [client] Could not access revision times.  [500, #175002]

Well, that’s not good.  What causes the problem exactly?  I’m not sure; but this only started after I committed a revision and the SVN command exited after printing out an error message, something about not being able to lock the database(I think this came from the server).  Things didn’t break all at once for some reason, but eventually all of the automated builds on Jenkins were failing.  The workaround specified in JENKINS-17431(edit the URLs to add @HEAD) works, but the fact that it randomly broke was bothering me.

Since this seemed to be related to a revision time problem, I tried to find out what revision was broken, which eventually led me to this blog post with the solution.  I’ve copied the relevant portions of the post below incase the post is ever removed/deleted:

In order to correct the problem the entries with the missing dates will need to be located.  This is easily done with the following command string which needs to be run on the Subversion server itself.

svn log --xml -v -r {2010-12-01T00:00:00Z}:{2011-01-13T15:00:00Z} file:///svn/repo

If there is any issue related to the time on a revision then the following error will be shown when you run the above command.

<?xml version="1.0"?>
svn: Failed to find time on revision 34534

The first step in making the necessary corrections is to edit the /svn/repos/mosaic/hooks/post-revprop-change file so that it contains the following information.

exit 0

If the previous steps had not be performed the hook files would have prevented this step from actually occurring. So to make the change to the revision entry the svn command is used on the server itself using the propset subcommand. When entering the date try to select a date that is in between the dates that proceed and follow this revision entry.

svn propset -r34534 --revprop svn:date '2010-11-29T19:55:44.000220Z' file:///svn/repo

The interesting thing about this was that in my case, the particular revision that was bad had been committed over 3 years ago, so why that one revision was bad is still a mystery to me.