At work, we use Jenkins to automatically build our software.  We use it to build both Java and C++ programs.  Since it’s useful to have a continuous integration server setup to build and test software, I figured that I would document a bit about what I have done on a new VM that I have setup.

This post is going to go into several different directions and technologies, so it may be a little rambling.

Setting up the VM

In order for us to create a hosted instance of Jenkins, the easiest thing to do is to create a VM for us to install Jenkins on.  Since many hosting providers don’t allow you to run Java on shared hosting, our options are to either create a VPS or create a cloud server.  I chose to create a cloud server with DreamCompute, as my web hosting is already through Dreamhost.  This allows us to create a new VM that we have full control over.  The normal VPS with Dreamhost is at some level still a shared system – Ubuntu only.  My preference is for Debian(I have had many weird issues with Ubuntu before).  Note that Jenkins will probably fail if it has less than 2 GB of RAM.  My first instance only had 512 MB of RAM, and as soon as I started to build something that required a bit of memory the VM immediately started thrashing.  My load average went to 11 with only 1 CPU!  (this means that I had enough jobs to keep 11 CPUs busy)

I launched a new Debian 8 image on DreamCompute and created a little script to install the needed things for Jenkins and a few other dependencies that I needed.  This also sets Jenkins up to run on port 80, so that we can access it directly through our browser.

Now that we have Jenkins installed, we need to SSH in to finish the install.

Finish Setup VM

Now that the VM has been mostly setup, we need to SSH in to finish the install.  The first thing I had to do was to finish the install of some needed parts(obviously, you have to wait until the setup script finishes before you can go to this step).

apt-get install cowbuilder

The next thing that we have to do is to setup Jenkins for the first time.  Get the initial admin password from the server:

root@jenkins:/home/debian# cat /var/lib/jenkins/secrets/initialAdminPassword
<hex-string-here>

This hex string is our default password for logging into Jenkins.  We need to go to our public IP address to set this up.  In my case, it is 208.113.129.94.  Once I got there, I put in the admin password and setup Jenkins for the first time.

DNS Setup

Accessing the server by the IP is not particularly useful.  Fortunately, DreamHost has a quick guide on how to setup your DNS entries.  I then put the server at jenkins.rm5248.com.

Jenkins Jobs

I have quite a few projects on my GitHub.  Half of this endeavor is to make a nice place where I can make sure that software is building cleanly.  So I have added a few projects to have them build automatically.

C/C++ Projects

Now to the other big reason for doing this: enabling automatic builds and automatic packaging.  Building Debian packages can be a little tricky. Some tools, like jenkins-debian-glue exist, but they are not the greatest for configuration and usage.  Problems that exist with jenkins-debian-glue:

• Script-based.  This is not too bad, but it means that it is hard to split builds across multiple nodes.  For example, building ARM packages on an ARM node in my testing is almost 10x faster than emulating ARM on x86
• Configuration is not clear.  There are separate scripts to generate git snapshots and svn snapshots.   Not all of the configuration variables are documented either.
• Requires a lot of manual configuration.
• The default setup requires you to have two Jenkins jobs per project.  This can quickly go overboard and lead to an explosion of jobs.
• Outputs of the build are not automatically archived to be able to be downloaded through the web interface

To be clear, jenkins-debian-glue is not the worst way to build Debian packages – but it could be a lot better.  So I wrote my own Jenkins plugin to do it.

Debian Pbuilder

This plugin is designed to make building of Debian packages very simple and straightforward.  Because it’s a plugin, it also makes running on other Jenkins nodes practical.  It is also in some ways a port of jenkins-debian-glue to Java, instead of being shell scripts.  In that regard, it uses cowbuilder to create the chroot environment and does build everything in a clean environment.  The actual Jenkins plugin for building packages doesn’t support this behavior at all, which makes builds awkward.

Anyway, this plugin fixes these issues and does everything in a nice way.  If there are people who would like to test out this plugin, that would be very useful.  Since it is built through our Jenkins instance, we can create a new build of the project at any time.  Note that as of right now, you have to install the plugin manually through Jenkins, it is not on the main Jenkins git to be easily installed at this moment.

More information on how to use Debian-pbuilder will be coming soon. 🙂

Other Nodes

This is not strictly related to Jenkins in the cloud, but one thing that we have done at work is to get a dedicated ARM computer in order to build our ARM packages with Debian-pbuilder. We use the NVidia Jetson TK1 board as a Jenkins node.  This greatly decreases our build time from almost 1 hour to only 10 minutes!  The problem with what we were doing before was that because of how Debian-pbuilder and jenkins-debian-glue work, they create an (emulated) chroot environment on the system.  This is slooooooooooow.  Making the chroot on actual hardware greatly speeds up the building of projects.