Jan 07

Jenkins in the Cloud

At work, we use Jenkins to automatically build our software.  We use it to build both Java and C++ programs.  Since it’s useful to have a continuous integration server setup to build and test software, I figured that I would document a bit about what I have done on a new VM that I have setup.

This post is going to go into several different directions and technologies, so it may be a little rambling.

Setting up the VM

In order for us to create a hosted instance of Jenkins, the easiest thing to do is to create a VM for us to install Jenkins on.  Since many hosting providers don’t allow you to run Java on shared hosting, our options are to either create a VPS or create a cloud server.  I chose to create a cloud server with DreamCompute, as my web hosting is already through Dreamhost.  This allows us to create a new VM that we have full control over.  The normal VPS with Dreamhost is at some level still a shared system – Ubuntu only.  My preference is for Debian(I have had many weird issues with Ubuntu before).  Note that Jenkins will probably fail if it has less than 2 GB of RAM.  My first instance only had 512 MB of RAM, and as soon as I started to build something that required a bit of memory the VM immediately started thrashing.  My load average went to 11 with only 1 CPU!  (this means that I had enough jobs to keep 11 CPUs busy)

I launched a new Debian 8 image on DreamCompute and created a little script to install the needed things for Jenkins and a few other dependencies that I needed.  This also sets Jenkins up to run on port 80, so that we can access it directly through our browser.

Now that we have Jenkins installed, we need to SSH in to finish the install.

Finish Setup VM

Now that the VM has been mostly setup, we need to SSH in to finish the install.  The first thing I had to do was to finish the install of some needed parts(obviously, you have to wait until the setup script finishes before you can go to this step).

apt-get install cowbuilder

The next thing that we have to do is to setup Jenkins for the first time.  Get the initial admin password from the server:

root@jenkins:/home/debian# cat /var/lib/jenkins/secrets/initialAdminPassword

This hex string is our default password for logging into Jenkins.  We need to go to our public IP address to set this up.  In my case, it is  Once I got there, I put in the admin password and setup Jenkins for the first time.

DNS Setup

Accessing the server by the IP is not particularly useful.  Fortunately, DreamHost has a quick guide on how to setup your DNS entries.  I then put the server at jenkins.rm5248.com.

Jenkins Jobs

I have quite a few projects on my GitHub.  Half of this endeavor is to make a nice place where I can make sure that software is building cleanly.  So I have added a few projects to have them build automatically.

C/C++ Projects

Now to the other big reason for doing this: enabling automatic builds and automatic packaging.  Building Debian packages can be a little tricky. Some tools, like jenkins-debian-glue exist, but they are not the greatest for configuration and usage.  Problems that exist with jenkins-debian-glue:

  • Script-based.  This is not too bad, but it means that it is hard to split builds across multiple nodes.  For example, building ARM packages on an ARM node in my testing is almost 10x faster than emulating ARM on x86
  • Configuration is not clear.  There are separate scripts to generate git snapshots and svn snapshots.   Not all of the configuration variables are documented either.
  • Requires a lot of manual configuration.
  • The default setup requires you to have two Jenkins jobs per project.  This can quickly go overboard and lead to an explosion of jobs.
  • Outputs of the build are not automatically archived to be able to be downloaded through the web interface

To be clear, jenkins-debian-glue is not the worst way to build Debian packages – but it could be a lot better.  So I wrote my own Jenkins plugin to do it.

Debian Pbuilder

This plugin is designed to make building of Debian packages very simple and straightforward.  Because it’s a plugin, it also makes running on other Jenkins nodes practical.  It is also in some ways a port of jenkins-debian-glue to Java, instead of being shell scripts.  In that regard, it uses cowbuilder to create the chroot environment and does build everything in a clean environment.  The actual Jenkins plugin for building packages doesn’t support this behavior at all, which makes builds awkward.

Anyway, this plugin fixes these issues and does everything in a nice way.  If there are people who would like to test out this plugin, that would be very useful.  Since it is built through our Jenkins instance, we can create a new build of the project at any time.  Note that as of right now, you have to install the plugin manually through Jenkins, it is not on the main Jenkins git to be easily installed at this moment.

More information on how to use Debian-pbuilder will be coming soon. 🙂

Other Nodes

This is not strictly related to Jenkins in the cloud, but one thing that we have done at work is to get a dedicated ARM computer in order to build our ARM packages with Debian-pbuilder. We use the NVidia Jetson TK1 board as a Jenkins node.  This greatly decreases our build time from almost 1 hour to only 10 minutes!  The problem with what we were doing before was that because of how Debian-pbuilder and jenkins-debian-glue work, they create an (emulated) chroot environment on the system.  This is slooooooooooow.  Making the chroot on actual hardware greatly speeds up the building of projects.

Nov 28

SVN Checkout By Date Problems

Have you ever had to checkout an SVN revision by the date and gotten an error like the following?

bash$ svn co -r {2016-11-28} https://example.com/svn/foo
svn: E175002: Unexpected HTTP status 500 'Internal Server Error' on '/svn/!svn/me'

svn: E175002: Additional errors:
svn: E175002: REPORT request on '/svn/!svn/me' failed: 500 Internal Server Error

(This is often the case with Jenkins, see this issue for some more details)

Why Jenkins checks out revisions by date is another matter(it’s stupid), but what is the original problem in the first place?  Well, if we look at the Apache error_log on the server, it will look something like the following:

[Mon Nov 28 08:01:24 2016] [error] [client] Could not access revision times.  [500, #175002]

Well, that’s not good.  What causes the problem exactly?  I’m not sure; but this only started after I committed a revision and the SVN command exited after printing out an error message, something about not being able to lock the database(I think this came from the server).  Things didn’t break all at once for some reason, but eventually all of the automated builds on Jenkins were failing.  The workaround specified in JENKINS-17431(edit the URLs to add @HEAD) works, but the fact that it randomly broke was bothering me.

Since this seemed to be related to a revision time problem, I tried to find out what revision was broken, which eventually led me to this blog post with the solution.  I’ve copied the relevant portions of the post below incase the post is ever removed/deleted:

In order to correct the problem the entries with the missing dates will need to be located.  This is easily done with the following command string which needs to be run on the Subversion server itself.

svn log --xml -v -r {2010-12-01T00:00:00Z}:{2011-01-13T15:00:00Z} file:///svn/repo

If there is any issue related to the time on a revision then the following error will be shown when you run the above command.

<?xml version="1.0"?>
svn: Failed to find time on revision 34534

The first step in making the necessary corrections is to edit the /svn/repos/mosaic/hooks/post-revprop-change file so that it contains the following information.

exit 0

If the previous steps had not be performed the hook files would have prevented this step from actually occurring. So to make the change to the revision entry the svn command is used on the server itself using the propset subcommand. When entering the date try to select a date that is in between the dates that proceed and follow this revision entry.

svn propset -r34534 --revprop svn:date '2010-11-29T19:55:44.000220Z' file:///svn/repo

The interesting thing about this was that in my case, the particular revision that was bad had been committed over 3 years ago, so why that one revision was bad is still a mystery to me.

Oct 16

Endianess in Computers

The other month, I was working on some code where I had to be concerned about the endianess of the data.  I came across this article from IBM which greatly cleared up some things(I got to the article at least partially from this Stackoverflow quesetion).

Now I’m going to deviate here as that was almost a month ago and I forget what I was going to talk about here.

Anyway, the point here is that when you do any sort of bit-shift or bit-masking, it is always in the code as big-endian format.  When you do an operation, it is always big-endian but how it gets stored differs on what machine you are on.  To get the actual layout of the byte as they are in-memory, I generally cast the data as a uint8_t*.  So for example, let’s take the following code:

uint32_t value = 0x12345678;
uint8_t* value_memory = (uint8_t*)&value;

for( int x = 0; x < 4; x++ ){
    printf( "0x%02X ", value_memory[ x ] );

With this code, there will be two different outputs depending on what endianess your machine is in.  On a big-endian machine(like MIPS) this will print out something like the following:

0x12 0x34 0x56 0x78

However, on a little-endian machine(x86, ARM) it will print out the values in the opposite direction:

0x78 0x56 0x34 0x12

The thing to remember here is that the ‘value’ variable, when printed out on both big and little-endian systems, will print out as the same value.

If you need to force a particular byte order, make an array in the code(this is shown in the IBM article).  If you need to convert a particular byte sequence into the proper endianess for your processor, you can simply OR the two bytes together.

For example, let’s assume that we get the following two bytes in big-endian order:

uint8_t bytes[] = { 0x12, 0x34 };

To convert this to your processor endianess as a 16-bit value, you only have to OR the two bytes together, and since OR-ing will always act as though the bytes were in a big-endian format, your data has magically become the proper endianess.

uint16_t proper_endianess = (bytes[ 0 ] << 8 ) | bytes[ 1 ];

Anyway I hope that somewhat clears some things up.  (Although I think I may have simply confused people more).

Aug 20

Libraries and Dependencies

Have you ever used a library in a project, only to find that it depends on other libraries, only to depend on other libraries?

I’m willing to bet that you have.  Fortunately, there exist tools in the world to help manage these dependencies.  In the Java world, Maven is the de facto standard in this regard.  It’s quite convenient to manage your dependencies.  On Windows, NuGet is the de facto standard.  (Note: I have never used NuGet.  The number of dependencies can quickly get staggeringly huge though, so that is at least part of the reason that these tools exist.

Why do I bring this up? Well, at work we use jOOQ for our database operations.  It’s an awesome library that you should use if you need to do database operations in Java.  They also have a blog that I read from time to time, as there is some good information included in it.  One of their more recent blog posts talks about libraries and having zero dependencies.  Basically, the argument here is that all libraries should have zero dependencies, because of getting into a dependency hell(although they do carve out an exception for frameworks such as Spring and Java EE).

I don’t fully agree with this statement, but it is a good time to talk about these dependencies.  As far as I am able, none of the libraries that I have made have any dependencies(that being said, I only have one library on Maven: JavaSerial).

Why don’t I fully agree with the statement?  Well, in many cases I don’t think that it is entirely unreasonable to have a dependency, although this can get into a very gray area.  In the case of jOOQ, it would not be entirely unreasonable to have a dependency on commons-lang, because there are two different artifacts for commons-lang(one for version 2.x[commons-lang] and one for version 3.x[commons-lang3]).  On the other hand though, if there is only one class that you need from a certain dependency, it is reasonable to simply wrap that class as jOOQ does in order to provide the same level of support.

With that being said, here is a set of questions you should ask yourself when deciding if you need a hard dependency of a library:

  • What is the dependency attempting to provide?  For example, SLF4J is a reasonable dependency to have, since many libraries already use it for logging(although, if at all possible, you should use java.util.logging instead)
  • How much of the dependency do you need?  If you only need a single class from commons-lang as jOOQ does, pulling in the full dependency does not make sense.  If you use a significant portion, then it may make sense.  This is probably the most important question you have to ask.  If you are doing a lot of reflection, then it may be worth it to get a library to help you do reflection.  If you only need to do it once, it probably doesn’t make sense.
  • How widespread is the library?  Commons-lang is a widely used library, so having it as a dependency is not likely to cause problems.
  • Are there different versions of the library that people may be using?  As shown in the jOOQ blog post, if you are using Guava 15 while another library uses Guava 18, will that cause problems?  I have never used Guava, so I don’t know if it would actually cause a problem, depending on how the API has changed.

I realize that this is not a comprehensive list, nor is it a perfect guide.  Ultimately, as a library writer you should try as much as possible to keep the number of dependencies down as low as possible.  Otherwise, it can quickly lead to dependency hell for the end users, and potentially for you as well.

Dec 30

SQLite and debugging

I have just spent the last few days and weeks trying to track down a very annoying, yet somewhat subtle bug in a program that I have been making for work.  So, first a little background.  We are making a gateway that talks over radio to other, remote lamps in order to control them and get status back.  One thing that it must be able to do is to change the frequency of a remote lamp.  In order to do this, you must first change your local frequency(if it has to be changed) and then tell the lamp the new frequency to go to, and ensure that you can still talk to the lamp once that has happened.  This is all controlled by a web page, so the sequence looks something like this:

Web Request --> (http) --> Gateway --> (serial, 115200) --> Radio --> (RF) --> Lamp Radio

However, we were having a problem with this sequence.  Almost all the time, the responses that we were getting back from the radio were short by a few bytes.  Sometimes it would be 1, other times it would be 15 bytes that we missed.  Moreover, the bytes that were dropped changed position.  Generally it wouldn’t really matter, since there were a bunch of unused bytes that were all initialized to 0x00, but we couldn’t tell where in that sequence the bytes were getting dropped.  Now let’s go into some pseudo-code here.  This is what the update looked like:

void update( Map<String,Object> JSONData ){
    if( JSONData[ "attr1" ] )
        writeToDatabase( JSONData[ "attr1" ] )
    // ..repeat the above a few times

    if( JSONData[ "frequency" ] ){
        //the frequency needs to be updated
        new ThreadToUpdateTheFrequency().start();

    if( JSONData[ "attr4" ] )
        writeToDatabase( JSONData[ "attr4" ] )
    //repeat the above a few more times

There’s not anything wrong with this code.  Sure, it’s probably inefficient, but that’s somewhat besides the point.  This code works sometimes, however a large portion of the time it will randomly drop bytes in the new thread that is started if the frequency is changed.

To debug this, I tried everything that I could think of.  Timing the sequence between us writing out data to the radio and receiving it back, writing data out at random times, etc.  Many hours were spent on this in order to figure out what was going on.  One of the things that we noticed though was that this only happened at certain times – when we were actually writing to the database.  If we were writing out commands without a write to the database, everything worked properly.  Moreover, if we stepped through the process in the debugger, everything would work properly.

What?  Why would the database cause a problem?

Well, let’s back up a bit and talk about the hardware that this is running on.  Linux is using an SD card as the root filesystem, and the processor is an Atmel SAMA5D3.  This is not a particularly fast processor, in addition to being only single-core.  What finally fixed the problem is the following:

void update( Map<String,Object> JSONData ){
    if( JSONData[ "attr1" ] )
        writeToDatabase( JSONData[ "attr1" ] )
    // ..repeat the above a few times for ALL ATTRIBUTES

    if( JSONData[ "frequency" ] ){
        //the frequency needs to be updated
        new ThreadToUpdateTheFrequency().start();

Why would moving the new thread to the end of the updates fix anything?  Well, here’s the basic theory that I have come up with:

SQLite is writing out to the database.  When it does that, it attempts to lock the database(see the documentation for some more details).  This takes quite a bit of time, especially talking to the SD card which is running at just a few megabytes per second at the most.  At some point during this time, a new interrupt comes in.  However, the kernel does not process the interrupt fast enough before a new character comes in on the serial line, and thus some data will get dropped since the processor is assuming that the data has been fully read.

Oddly enough, dmesg does not show many problems; I only get a few input overruns.  It looks like there could be a kernel bug here compounded by how fast the SD card can be read from.

Aug 29

Simple JAX-RS example & Maven configuration

So I have been using JAX-RS for the past few months now, and it was a little confusing to get started with.  To fix that, I’ve created a simple project on github that contains a minimal configuration for Maven in order to get a war file and have Jersey be the JAX-RS provider.

The other reason to do this is because the Jersey documentation is not very clear as to what you have to pull in from Maven Central in order to actually use the dependencies in a program.  Turns out, it’s just a single dependency, but finding it and making sure that it works is another matter.

Lastly, I just came across this page, which also has some good JAX-RS examples.  I hope that they help somebody!

May 27

udev rules for multiple interfaces

I just had to write a udev rule for Linux to interpret between several different endpoints on a single cellular modem, specifically this modem.  This shows up in Linux as 5 ttyUSB ports.  In order to make sure that the ports do not change around on me, I checked the following with udevadm:

$ udevadm info -a -n /dev/ttyUSB1

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

 looking at device '/devices/pci0000:00/0000:00:1d.0/usb4/4-1/4-1.1/4-1.1:1.0/ttyUSB1/tty/ttyUSB1':

 looking at parent device '/devices/pci0000:00/0000:00:1d.0/usb4/4-1/4-1.1/4-1.1:1.0/ttyUSB1':

 looking at parent device '/devices/pci0000:00/0000:00:1d.0/usb4/4-1/4-1.1/4-1.1:1.0':
 ATTRS{bAlternateSetting}==" 0"

 looking at parent device '/devices/pci0000:00/0000:00:1d.0/usb4/4-1/4-1.1':
 ATTRS{bNumInterfaces}==" 8"
 ATTRS{version}==" 2.00"

 looking at parent device '/devices/pci0000:00/0000:00:1d.0/usb4/4-1':
 ATTRS{bNumInterfaces}==" 1"
 ATTRS{version}==" 2.00"

 looking at parent device '/devices/pci0000:00/0000:00:1d.0/usb4':
 ATTRS{bNumInterfaces}==" 1"
 ATTRS{version}==" 2.00"
 ATTRS{manufacturer}=="Linux 3.16.0-4-amd64 ehci_hcd"
 ATTRS{product}=="EHCI Host Controller"

 looking at parent device '/devices/pci0000:00/0000:00:1d.0':

 looking at parent device '/devices/pci0000:00':

After I got the info, I noticed that the one thing that was different between all of the ttyUSB devices was ATTRS{bInterfaceNumber}.  So, I wrote a rule that looked like this:

ATTRS{idProduct}=="1201", ATTRS{idVendor}=="1bc7", ATTRS{bInterfaceNumber}=="00", SYMLINK+="cellular_1"

However, this did not produce the expected result of a new symlink.  It appears as though you can’t get the udev information for a product/vendor and the interface number in the same rule.  However, there are environment variables that are set which you can use.  If you run udevadm test, you can see what exactly the environment variables are that are set:

$ udevadm test --action=add `udevadm info -q path -n /dev/ttyUSB1`

...skip a bunch of data...
DEVLINKS=/dev/serial/by-id/usb-Android_Android_0123456789ABCDEF-if00-port0 /dev/serial/by-path/pci-0000:00:1d.0-usb-0:1.1:1.0-port0
ID_VENDOR_FROM_DATABASE=Telit Wireless Solutions

These variables can also be used in udev rules to do differentiate between different devices.  The two that we are interested in are ID_VENDOR_ID and ID_MODEL_ID, as these correspond to ATTRS{idProduct} and ATTRS{idVendor}.  Our new rule becomes:

DRIVERS=="option", ATTRS{bInterfaceNumber}=="00", ENV{ID_MODEL_ID}=="1201", ENV{ID_VENDOR_ID}=="1bc7", SYMLINK+="cellular_1"

The DRIVERS part of the rule may not be strictly necessary, but it can’t hurt it.

Feb 03

Web Design and Programmers

On a project at work, we’re starting into web development from being entirely desktop applications.  It’s a new thing to learn because of that.  We’re working with a contractor, who is creating the majority of the website for us.  He has suggested, and we have implemented, a REST-based service for sending data to and from the server.  Fortunately, making a REST-based web service in Java is very easy using Genson and JAX-RS.

However, this post is not about that.  It is about web technologies in general.  Originally, I spent a fair amount of time researching different web technologies that use Java as the backend, and I came up with a few such as JSP, JSF, as well as some frameworks, such as PrimeFaces.  We’re not using JSF technology at the moment, but that’s not really the point either.

There are a lot of different web technologies out there, and each has its own pros and cons.  My real question here is why do we treat web design as separate from programming the website?  While researching the advantages/disadvantages of a REST-only website, I came across this good post on programmers.SE about the pros and cons, which then led me to this interesting post about using tag libraries in web pages.  Essentially, what this boils down to is:

The benefit here is that web designers are supposed to be able to work with these tags, or at least that’s what they say. In practice, in all my years of software development and programming, I have yet to ever see this magical unicorn walk out of the shadows and into the sunlight.

My question here though is: why are we treating web development in a different light than we are treating desktop application development?

When developing a desktop application, generally the programmers are the people who create the design.  Even if somebody creates the design in a GUI designer, the programmer has to be involved in how to tie all the events together – they don’t exist in separate paradigms, which is what web technologies seem to do.

In fact, I make all of my Java GUIs by hand.  I have never made a QT gui by hand, but from the generated code that I have looked at it follows a very similar programming model(composite design pattern).  What exactly is preventing a web designer from doing code, and what is preventing a programmer from making a website?

It seems to me that this is not something where the programmer and the web designer are two different people.  What seems to be better would be to have people that can straddle both areas.  After all, we are generally not pigeon-holed into just one area.  If you work mostly on backend code, it is not unreasonable to also touch the GUI code at some point, so you must know how the GUIs are set-up.  But for web development, this seems to be just the opposite.

Perhaps what we need is a swing-like web framework that you don’t need to make any HTML files for, just create the entire website through code.  This sounds complicated, and I don’t know if it would solve the problem.

Jan 19

PIC32 and I2C

I’ve been struggling with Microchip’s plib library trying to get I2C working.  First, the example I2C code is horrible.  Second, I’m not even sure what sample hardware from Microchip actually uses it.  It’s just so confusing.

Anyway, there are a few things to note here about the I2C library and the quirks that it has:

  • Make sure that you have pull-up resistors on the I2C bus.  This was problem #1.
  • Make sure you configure all pins as digital, if they are analog pins.  This was problem #2.
  • There are two methods you need to acknowledge the I2C: I2CAcknowledgeByte and I2CByteWasAcknowledged.
  • When reading a byte from the bus when you are the master, you need to call I2CReceiverEnable for each byte that you need to get.

These functions are not very special; the problem is that they are poorly documented.  For the most part, the plib functions simply access the registers directly in memory and return an appropriate value after checking the bits that may be set as a side effect.  It’s not particularly hard, they are just oddly named often times.

Anyway, you can see more about this both here and… some other place on the Microchip forums, I can’t find it now.

Hope that helps.

Dec 22

Adventures in accidental chown

So earlier today I had to get Virtualbox up and running, and it was complaining about permissions not being set right.  For some reason, the permissions on my /usr directory were slightly messed up and not owned by root.  So, I did:

sudo chown root:root /usr

That didn’t fix the problem.  Going into the /usr directory, I also noticed that a few of the other folders weren’t owned by root.  So, let’s just go fix everything at once:

sudo chown -R root:root /usr

Okay, now let’s try starting Virtualbox!

Virtualbox now complains about not being setuid.  Okay, fine.  Let’s reinstall virtualbox, since it says that should fix it.

sudo dpkg -i virtualbox.deb
sudo: must be setuid root



Turns out, doing the chroot dropped the setuid bit from all binaries in the /usr directory.

To fix this, you’ll have to boot from a rescue disk/CD of some kind, and change the permissions on sudo:

chmod 4755 /path/to/mounted/filesystem/usr/bin/sudo

Other programs will also be messed up at that point, but now that you have sudo you can change the permissions to what they should be, and/or reinstall. 🙂