At work, we use Jenkins to automatically build our software.  We use it to build both Java and C++ programs.  Since it’s useful to have a continuous integration server setup to build and test software, I figured that I would document a bit about what I have done on a new VM that I have setup.

This post is going to go into several different directions and technologies, so it may be a little rambling.

Setting up the VM

In order for us to create a hosted instance of Jenkins, the easiest thing to do is to create a VM for us to install Jenkins on.  Since many hosting providers don’t allow you to run Java on shared hosting, our options are to either create a VPS or create a cloud server.  I chose to create a cloud server with DreamCompute, as my web hosting is already through Dreamhost.  This allows us to create a new VM that we have full control over.  The normal VPS with Dreamhost is at some level still a shared system – Ubuntu only.  My preference is for Debian(I have had many weird issues with Ubuntu before).  Note that Jenkins will probably fail if it has less than 2 GB of RAM.  My first instance only had 512 MB of RAM, and as soon as I started to build something that required a bit of memory the VM immediately started thrashing.  My load average went to 11 with only 1 CPU!  (this means that I had enough jobs to keep 11 CPUs busy)

I launched a new Debian 8 image on DreamCompute and created a little script to install the needed things for Jenkins and a few other dependencies that I needed.  This also sets Jenkins up to run on port 80, so that we can access it directly through our browser.

Now that we have Jenkins installed, we need to SSH in to finish the install.

Finish Setup VM

Now that the VM has been mostly setup, we need to SSH in to finish the install.  The first thing I had to do was to finish the install of some needed parts(obviously, you have to wait until the setup script finishes before you can go to this step).

apt-get install cowbuilder

The next thing that we have to do is to setup Jenkins for the first time.  Get the initial admin password from the server:

root@jenkins:/home/debian# cat /var/lib/jenkins/secrets/initialAdminPassword
<hex-string-here>

This hex string is our default password for logging into Jenkins.  We need to go to our public IP address to set this up.  In my case, it is 208.113.129.94.  Once I got there, I put in the admin password and setup Jenkins for the first time.

DNS Setup

Accessing the server by the IP is not particularly useful.  Fortunately, DreamHost has a quick guide on how to setup your DNS entries.  I then put the server at jenkins.rm5248.com.

Jenkins Jobs

I have quite a few projects on my GitHub.  Half of this endeavor is to make a nice place where I can make sure that software is building cleanly.  So I have added a few projects to have them build automatically.

C/C++ Projects

Now to the other big reason for doing this: enabling automatic builds and automatic packaging.  Building Debian packages can be a little tricky. Some tools, like jenkins-debian-glue exist, but they are not the greatest for configuration and usage.  Problems that exist with jenkins-debian-glue:

• Script-based.  This is not too bad, but it means that it is hard to split builds across multiple nodes.  For example, building ARM packages on an ARM node in my testing is almost 10x faster than emulating ARM on x86
• Configuration is not clear.  There are separate scripts to generate git snapshots and svn snapshots.   Not all of the configuration variables are documented either.
• Requires a lot of manual configuration.
• The default setup requires you to have two Jenkins jobs per project.  This can quickly go overboard and lead to an explosion of jobs.
• Outputs of the build are not automatically archived to be able to be downloaded through the web interface

To be clear, jenkins-debian-glue is not the worst way to build Debian packages – but it could be a lot better.  So I wrote my own Jenkins plugin to do it.

Debian Pbuilder

This plugin is designed to make building of Debian packages very simple and straightforward.  Because it’s a plugin, it also makes running on other Jenkins nodes practical.  It is also in some ways a port of jenkins-debian-glue to Java, instead of being shell scripts.  In that regard, it uses cowbuilder to create the chroot environment and does build everything in a clean environment.  The actual Jenkins plugin for building packages doesn’t support this behavior at all, which makes builds awkward.

Anyway, this plugin fixes these issues and does everything in a nice way.  If there are people who would like to test out this plugin, that would be very useful.  Since it is built through our Jenkins instance, we can create a new build of the project at any time.  Note that as of right now, you have to install the plugin manually through Jenkins, it is not on the main Jenkins git to be easily installed at this moment.

More information on how to use Debian-pbuilder will be coming soon. 🙂

Other Nodes

This is not strictly related to Jenkins in the cloud, but one thing that we have done at work is to get a dedicated ARM computer in order to build our ARM packages with Debian-pbuilder. We use the NVidia Jetson TK1 board as a Jenkins node.  This greatly decreases our build time from almost 1 hour to only 10 minutes!  The problem with what we were doing before was that because of how Debian-pbuilder and jenkins-debian-glue work, they create an (emulated) chroot environment on the system.  This is slooooooooooow.  Making the chroot on actual hardware greatly speeds up the building of projects.

Have you ever had to checkout an SVN revision by the date and gotten an error like the following?

bash$ svn co -r {2016-11-28} https://example.com/svn/foo
svn: E175002: Unexpected HTTP status 500 'Internal Server Error' on '/svn/!svn/me'

svn: E175002: Additional errors:
svn: E175002: REPORT request on '/svn/!svn/me' failed: 500 Internal Server Error

(This is often the case with Jenkins, see this issue for some more details)

Why Jenkins checks out revisions by date is another matter(it’s stupid), but what is the original problem in the first place?  Well, if we look at the Apache error_log on the server, it will look something like the following:

[Mon Nov 28 08:01:24 2016] [error] [client 127.0.0.1] Could not access revision times.  [500, #175002]

Well, that’s not good.  What causes the problem exactly?  I’m not sure; but this only started after I committed a revision and the SVN command exited after printing out an error message, something about not being able to lock the database(I think this came from the server).  Things didn’t break all at once for some reason, but eventually all of the automated builds on Jenkins were failing.  The workaround specified in JENKINS-17431(edit the URLs to add @HEAD) works, but the fact that it randomly broke was bothering me.

Since this seemed to be related to a revision time problem, I tried to find out what revision was broken, which eventually led me to this blog post with the solution.  I’ve copied the relevant portions of the post below incase the post is ever removed/deleted:

In order to correct the problem the entries with the missing dates will need to be located.  This is easily done with the following command string which needs to be run on the Subversion server itself.

svn log --xml -v -r {2010-12-01T00:00:00Z}:{2011-01-13T15:00:00Z} file:///svn/repo

If there is any issue related to the time on a revision then the following error will be shown when you run the above command.

<?xml version="1.0"?>
<log>
svn: Failed to find time on revision 34534

The first step in making the necessary corrections is to edit the /svn/repos/mosaic/hooks/post-revprop-change file so that it contains the following information.

#!/bin/sh
exit 0

If the previous steps had not be performed the hook files would have prevented this step from actually occurring. So to make the change to the revision entry the svn command is used on the server itself using the propset subcommand. When entering the date try to select a date that is in between the dates that proceed and follow this revision entry.

svn propset -r34534 --revprop svn:date '2010-11-29T19:55:44.000220Z' file:///svn/repo

The interesting thing about this was that in my case, the particular revision that was bad had been committed over 3 years ago, so why that one revision was bad is still a mystery to me.

The other month, I was working on some code where I had to be concerned about the endianess of the data.  I came across this article from IBM which greatly cleared up some things(I got to the article at least partially from this Stackoverflow quesetion).

Now I’m going to deviate here as that was almost a month ago and I forget what I was going to talk about here.

Anyway, the point here is that when you do any sort of bit-shift or bit-masking, it is always in the code as big-endian format.  When you do an operation, it is always big-endian but how it gets stored differs on what machine you are on.  To get the actual layout of the byte as they are in-memory, I generally cast the data as a uint8_t*.  So for example, let’s take the following code:

uint32_t value = 0x12345678;
uint8_t* value_memory = (uint8_t*)&value;

for( int x = 0; x < 4; x++ ){
    printf( "0x%02X ", value_memory[ x ] );
}

With this code, there will be two different outputs depending on what endianess your machine is in.  On a big-endian machine(like MIPS) this will print out something like the following:

0x12 0x34 0x56 0x78

However, on a little-endian machine(x86, ARM) it will print out the values in the opposite direction:

0x78 0x56 0x34 0x12

The thing to remember here is that the ‘value’ variable, when printed out on both big and little-endian systems, will print out as the same value.

If you need to force a particular byte order, make an array in the code(this is shown in the IBM article).  If you need to convert a particular byte sequence into the proper endianess for your processor, you can simply OR the two bytes together.

For example, let’s assume that we get the following two bytes in big-endian order:

uint8_t bytes[] = { 0x12, 0x34 };

To convert this to your processor endianess as a 16-bit value, you only have to OR the two bytes together, and since OR-ing will always act as though the bytes were in a big-endian format, your data has magically become the proper endianess.

uint16_t proper_endianess = (bytes[ 0 ] << 8 ) | bytes[ 1 ];

Anyway I hope that somewhat clears some things up.  (Although I think I may have simply confused people more).

Have you ever used a library in a project, only to find that it depends on other libraries, only to depend on other libraries?

I’m willing to bet that you have.  Fortunately, there exist tools in the world to help manage these dependencies.  In the Java world, Maven is the de facto standard in this regard.  It’s quite convenient to manage your dependencies.  On Windows, NuGet is the de facto standard.  (Note: I have never used NuGet.  The number of dependencies can quickly get staggeringly huge though, so that is at least part of the reason that these tools exist.

Why do I bring this up? Well, at work we use jOOQ for our database operations.  It’s an awesome library that you should use if you need to do database operations in Java.  They also have a blog that I read from time to time, as there is some good information included in it.  One of their more recent blog posts talks about libraries and having zero dependencies.  Basically, the argument here is that all libraries should have zero dependencies, because of getting into a dependency hell(although they do carve out an exception for frameworks such as Spring and Java EE).

I don’t fully agree with this statement, but it is a good time to talk about these dependencies.  As far as I am able, none of the libraries that I have made have any dependencies(that being said, I only have one library on Maven: JavaSerial).

Why don’t I fully agree with the statement?  Well, in many cases I don’t think that it is entirely unreasonable to have a dependency, although this can get into a very gray area.  In the case of jOOQ, it would not be entirely unreasonable to have a dependency on commons-lang, because there are two different artifacts for commons-lang(one for version 2.x[commons-lang] and one for version 3.x[commons-lang3]).  On the other hand though, if there is only one class that you need from a certain dependency, it is reasonable to simply wrap that class as jOOQ does in order to provide the same level of support.

With that being said, here is a set of questions you should ask yourself when deciding if you need a hard dependency of a library:

• What is the dependency attempting to provide?  For example, SLF4J is a reasonable dependency to have, since many libraries already use it for logging(although, if at all possible, you should use java.util.logging instead)
• How much of the dependency do you need?  If you only need a single class from commons-lang as jOOQ does, pulling in the full dependency does not make sense.  If you use a significant portion, then it may make sense.  This is probably the most important question you have to ask.  If you are doing a lot of reflection, then it may be worth it to get a library to help you do reflection.  If you only need to do it once, it probably doesn’t make sense.
• How widespread is the library?  Commons-lang is a widely used library, so having it as a dependency is not likely to cause problems.
• Are there different versions of the library that people may be using?  As shown in the jOOQ blog post, if you are using Guava 15 while another library uses Guava 18, will that cause problems?  I have never used Guava, so I don’t know if it would actually cause a problem, depending on how the API has changed.

I realize that this is not a comprehensive list, nor is it a perfect guide.  Ultimately, as a library writer you should try as much as possible to keep the number of dependencies down as low as possible.  Otherwise, it can quickly lead to dependency hell for the end users, and potentially for you as well.