Jan 21

Thoughts on Star Wars and Jedi

So I saw Rogue One last week, and I found it to be quite interesting.  It wasn’t my favorite movie, but it was a good lead in to A New Hope.  Once I got back home, I had to look up the crawl for A New Hope, which of course I did on TV tropes.  Looking at the TV tropes page for A New Hope, there is this one section that I feel is somewhat out of place:

  • Admiral Motti’s description of the Force as a “sad devotion to that ancient religion” seems downright bizarre, given that its existence was treated as common knowledge in the Prequel Trilogy, a timeframe in which Motti would likely have been alive (albeit very young). This is largely because when the film was written Lucas envisioned exact knowledge of the Force and Jedi powers to be something which only a select few had knowledge of, which was gradually contradicted by the Expanded Universe novels and comics, and then jettisoned altogether by The Phantom Menace. Some later novels, such as the Republic Commando novels, took some steps to try and square the two perspectives, but with limited success.

The “sad devotion to that ancient religion” actually seems like it could be a plausible thing for him to say, even though the bullet point goes on to say that this was contradicted with later works.  Why would this work?  Well, let’s do some basic math here and get some rough estimates of how many people there are.

First order of business: How many Jedi are there?  The movies don’t say for certain, but the Jedi temple seems to be quite large.  Making grand assumptions here, let’s also assume that it is large enough to fit every Jedi at once in it.  Given that it seems to encompass an entire city block at least, let us say that the Jedi temple can fit 100,000 people.  (According to scifi.stackexchange.com, there were about 10,000 Jedi at the time of Order 66).  Regardless, 10,000 vs 100,000 Jedi will make no difference here in a moment.

Next question: How large is Coruscant?  Assuming that it is the same size as Earth, and has a population density at least as high as the highest city on Earth(according to Wikipedia, that is Manila at 41,515 people/km²).  According to Wolfram|Alpha, the Earth is 5.1×10^8 square kilometers.  Now, multiplying these two numbers together we get a population of ~2.1×10^13.  According to Wookieepedia, there is more than 1 trillion(1×10^12) people on Coruscant.  If I am doing the math correctly here, my estimate for the number of people on Coruscant is (one order of magnitude * 2 ) from what Wookiepedia says.

Now that we have our numbers(both unofficial and estimated), let’s do some quick math here to figure out about how many Jedi there are per-person on Coruscant.

Number of Jedi Population of Coruscant People-per-Jedi
10,000 1×10^12 100,000,000
10,000 2.1×10^13 2,100,000,000
100,000 1×10^12 10,000,000
100,000 2.1×10^13 210,000,000

Now, obviously these numbers are rather rough estimates.  But at a minimum, there are ten million people on Coruscant per Jedi!

Now, back to the original question: Why is the statement “sad devotion to that ancient religion” completely plausible in my mind?  Because given the ratio of people to Jedi on Coruscant alone, it is likely that you could go an entire lifetime without seeing a real Jedi.  Don’t forget as well, there are more planets than just Coruscant.  Basically, the number of Jedi for the population of the galaxy is laughably small.  If you combine the lack of population with potentially a good PR campaign, it is completely plausible that a person could grow up in the galaxy and never see a Jedi, only hear about them, and then be convinced that Jedi are bad people who were controlling the fate of the galaxy in their hands and/or they were just pretending.  Reportedly, there was a minister in Pakistan who fooled the entire government for 6 years that he wasn’t an actual minister(note: I can’t find any other sources to this from Western media, so I don’t know the veracity of this).

So, to conclude: It is reasonable to assume that because there were so few Jedi for the entire population of the galaxy, any effort to discredit them could be very effective at wiping their knowledge from the population’s memory.

Jan 07

Jenkins in the Cloud

At work, we use Jenkins to automatically build our software.  We use it to build both Java and C++ programs.  Since it’s useful to have a continuous integration server setup to build and test software, I figured that I would document a bit about what I have done on a new VM that I have setup.

This post is going to go into several different directions and technologies, so it may be a little rambling.

Setting up the VM

In order for us to create a hosted instance of Jenkins, the easiest thing to do is to create a VM for us to install Jenkins on.  Since many hosting providers don’t allow you to run Java on shared hosting, our options are to either create a VPS or create a cloud server.  I chose to create a cloud server with DreamCompute, as my web hosting is already through Dreamhost.  This allows us to create a new VM that we have full control over.  The normal VPS with Dreamhost is at some level still a shared system – Ubuntu only.  My preference is for Debian(I have had many weird issues with Ubuntu before).  Note that Jenkins will probably fail if it has less than 2 GB of RAM.  My first instance only had 512 MB of RAM, and as soon as I started to build something that required a bit of memory the VM immediately started thrashing.  My load average went to 11 with only 1 CPU!  (this means that I had enough jobs to keep 11 CPUs busy)

I launched a new Debian 8 image on DreamCompute and created a little script to install the needed things for Jenkins and a few other dependencies that I needed.  This also sets Jenkins up to run on port 80, so that we can access it directly through our browser.

Now that we have Jenkins installed, we need to SSH in to finish the install.

Finish Setup VM

Now that the VM has been mostly setup, we need to SSH in to finish the install.  The first thing I had to do was to finish the install of some needed parts(obviously, you have to wait until the setup script finishes before you can go to this step).

apt-get install cowbuilder

The next thing that we have to do is to setup Jenkins for the first time.  Get the initial admin password from the server:

root@jenkins:/home/debian# cat /var/lib/jenkins/secrets/initialAdminPassword
<hex-string-here>

This hex string is our default password for logging into Jenkins.  We need to go to our public IP address to set this up.  In my case, it is 208.113.129.94.  Once I got there, I put in the admin password and setup Jenkins for the first time.

DNS Setup

Accessing the server by the IP is not particularly useful.  Fortunately, DreamHost has a quick guide on how to setup your DNS entries.  I then put the server at jenkins.rm5248.com.

Jenkins Jobs

I have quite a few projects on my GitHub.  Half of this endeavor is to make a nice place where I can make sure that software is building cleanly.  So I have added a few projects to have them build automatically.

C/C++ Projects

Now to the other big reason for doing this: enabling automatic builds and automatic packaging.  Building Debian packages can be a little tricky. Some tools, like jenkins-debian-glue exist, but they are not the greatest for configuration and usage.  Problems that exist with jenkins-debian-glue:

  • Script-based.  This is not too bad, but it means that it is hard to split builds across multiple nodes.  For example, building ARM packages on an ARM node in my testing is almost 10x faster than emulating ARM on x86
  • Configuration is not clear.  There are separate scripts to generate git snapshots and svn snapshots.   Not all of the configuration variables are documented either.
  • Requires a lot of manual configuration.
  • The default setup requires you to have two Jenkins jobs per project.  This can quickly go overboard and lead to an explosion of jobs.
  • Outputs of the build are not automatically archived to be able to be downloaded through the web interface

To be clear, jenkins-debian-glue is not the worst way to build Debian packages – but it could be a lot better.  So I wrote my own Jenkins plugin to do it.

Debian Pbuilder

This plugin is designed to make building of Debian packages very simple and straightforward.  Because it’s a plugin, it also makes running on other Jenkins nodes practical.  It is also in some ways a port of jenkins-debian-glue to Java, instead of being shell scripts.  In that regard, it uses cowbuilder to create the chroot environment and does build everything in a clean environment.  The actual Jenkins plugin for building packages doesn’t support this behavior at all, which makes builds awkward.

Anyway, this plugin fixes these issues and does everything in a nice way.  If there are people who would like to test out this plugin, that would be very useful.  Since it is built through our Jenkins instance, we can create a new build of the project at any time.  Note that as of right now, you have to install the plugin manually through Jenkins, it is not on the main Jenkins git to be easily installed at this moment.

More information on how to use Debian-pbuilder will be coming soon. 🙂

Other Nodes

This is not strictly related to Jenkins in the cloud, but one thing that we have done at work is to get a dedicated ARM computer in order to build our ARM packages with Debian-pbuilder. We use the NVidia Jetson TK1 board as a Jenkins node.  This greatly decreases our build time from almost 1 hour to only 10 minutes!  The problem with what we were doing before was that because of how Debian-pbuilder and jenkins-debian-glue work, they create an (emulated) chroot environment on the system.  This is slooooooooooow.  Making the chroot on actual hardware greatly speeds up the building of projects.

Dec 01

SVN Commit Problems – Update

An update to my previous problem with checking into SVN.  I had the same problem just now:

$ svn commit -m "message"
Sending        debian/control
Transmitting file data .

Committed revision 9488.

Warning: post commit FS processing had error:
sqlite[S5]: database is locked

I SSH’d into the server to try and figure out the problem.  I then stumbled across this post on the SVN mailing list.  Checking the rep-cache.db, it has two commits less than what the current revision number is, which would match up with my two failed commits, so that seemed suspicious.  I then figured out if some process had the rep-cache.db open at all:

$ lsof | grep rep-cache

As it turns out, Atlassian Fisheye had a bunch of connections open to this database for some reason.  I restarted Fisheye to kill all those connections, so we will see if that fixes the problem.  Note that this is also running an old version of Fisheye(3.2.3), so it could be a problem with just this version.

Nov 28

SVN Checkout By Date Problems

Have you ever had to checkout an SVN revision by the date and gotten an error like the following?

bash$ svn co -r {2016-11-28} https://example.com/svn/foo
svn: E175002: Unexpected HTTP status 500 'Internal Server Error' on '/svn/!svn/me'

svn: E175002: Additional errors:
svn: E175002: REPORT request on '/svn/!svn/me' failed: 500 Internal Server Error

(This is often the case with Jenkins, see this issue for some more details)

Why Jenkins checks out revisions by date is another matter(it’s stupid), but what is the original problem in the first place?  Well, if we look at the Apache error_log on the server, it will look something like the following:

[Mon Nov 28 08:01:24 2016] [error] [client 127.0.0.1] Could not access revision times.  [500, #175002]

Well, that’s not good.  What causes the problem exactly?  I’m not sure; but this only started after I committed a revision and the SVN command exited after printing out an error message, something about not being able to lock the database(I think this came from the server).  Things didn’t break all at once for some reason, but eventually all of the automated builds on Jenkins were failing.  The workaround specified in JENKINS-17431(edit the URLs to add @HEAD) works, but the fact that it randomly broke was bothering me.

Since this seemed to be related to a revision time problem, I tried to find out what revision was broken, which eventually led me to this blog post with the solution.  I’ve copied the relevant portions of the post below incase the post is ever removed/deleted:

In order to correct the problem the entries with the missing dates will need to be located.  This is easily done with the following command string which needs to be run on the Subversion server itself.

svn log --xml -v -r {2010-12-01T00:00:00Z}:{2011-01-13T15:00:00Z} file:///svn/repo

If there is any issue related to the time on a revision then the following error will be shown when you run the above command.

<?xml version="1.0"?>
<log>
svn: Failed to find time on revision 34534

The first step in making the necessary corrections is to edit the /svn/repos/mosaic/hooks/post-revprop-change file so that it contains the following information.

#!/bin/sh
exit 0

If the previous steps had not be performed the hook files would have prevented this step from actually occurring. So to make the change to the revision entry the svn command is used on the server itself using the propset subcommand. When entering the date try to select a date that is in between the dates that proceed and follow this revision entry.

svn propset -r34534 --revprop svn:date '2010-11-29T19:55:44.000220Z' file:///svn/repo

The interesting thing about this was that in my case, the particular revision that was bad had been committed over 3 years ago, so why that one revision was bad is still a mystery to me.

Oct 16

Endianess in Computers

The other month, I was working on some code where I had to be concerned about the endianess of the data.  I came across this article from IBM which greatly cleared up some things(I got to the article at least partially from this Stackoverflow quesetion).

Now I’m going to deviate here as that was almost a month ago and I forget what I was going to talk about here.

Anyway, the point here is that when you do any sort of bit-shift or bit-masking, it is always in the code as big-endian format.  When you do an operation, it is always big-endian but how it gets stored differs on what machine you are on.  To get the actual layout of the byte as they are in-memory, I generally cast the data as a uint8_t*.  So for example, let’s take the following code:

uint32_t value = 0x12345678;
uint8_t* value_memory = (uint8_t*)&value;

for( int x = 0; x < 4; x++ ){
    printf( "0x%02X ", value_memory[ x ] );
}

With this code, there will be two different outputs depending on what endianess your machine is in.  On a big-endian machine(like MIPS) this will print out something like the following:

0x12 0x34 0x56 0x78

However, on a little-endian machine(x86, ARM) it will print out the values in the opposite direction:

0x78 0x56 0x34 0x12

The thing to remember here is that the ‘value’ variable, when printed out on both big and little-endian systems, will print out as the same value.

If you need to force a particular byte order, make an array in the code(this is shown in the IBM article).  If you need to convert a particular byte sequence into the proper endianess for your processor, you can simply OR the two bytes together.

For example, let’s assume that we get the following two bytes in big-endian order:

uint8_t bytes[] = { 0x12, 0x34 };

To convert this to your processor endianess as a 16-bit value, you only have to OR the two bytes together, and since OR-ing will always act as though the bytes were in a big-endian format, your data has magically become the proper endianess.

uint16_t proper_endianess = (bytes[ 0 ] << 8 ) | bytes[ 1 ];

Anyway I hope that somewhat clears some things up.  (Although I think I may have simply confused people more).

Aug 20

Libraries and Dependencies

Have you ever used a library in a project, only to find that it depends on other libraries, only to depend on other libraries?

I’m willing to bet that you have.  Fortunately, there exist tools in the world to help manage these dependencies.  In the Java world, Maven is the de facto standard in this regard.  It’s quite convenient to manage your dependencies.  On Windows, NuGet is the de facto standard.  (Note: I have never used NuGet.  The number of dependencies can quickly get staggeringly huge though, so that is at least part of the reason that these tools exist.

Why do I bring this up? Well, at work we use jOOQ for our database operations.  It’s an awesome library that you should use if you need to do database operations in Java.  They also have a blog that I read from time to time, as there is some good information included in it.  One of their more recent blog posts talks about libraries and having zero dependencies.  Basically, the argument here is that all libraries should have zero dependencies, because of getting into a dependency hell(although they do carve out an exception for frameworks such as Spring and Java EE).

I don’t fully agree with this statement, but it is a good time to talk about these dependencies.  As far as I am able, none of the libraries that I have made have any dependencies(that being said, I only have one library on Maven: JavaSerial).

Why don’t I fully agree with the statement?  Well, in many cases I don’t think that it is entirely unreasonable to have a dependency, although this can get into a very gray area.  In the case of jOOQ, it would not be entirely unreasonable to have a dependency on commons-lang, because there are two different artifacts for commons-lang(one for version 2.x[commons-lang] and one for version 3.x[commons-lang3]).  On the other hand though, if there is only one class that you need from a certain dependency, it is reasonable to simply wrap that class as jOOQ does in order to provide the same level of support.

With that being said, here is a set of questions you should ask yourself when deciding if you need a hard dependency of a library:

  • What is the dependency attempting to provide?  For example, SLF4J is a reasonable dependency to have, since many libraries already use it for logging(although, if at all possible, you should use java.util.logging instead)
  • How much of the dependency do you need?  If you only need a single class from commons-lang as jOOQ does, pulling in the full dependency does not make sense.  If you use a significant portion, then it may make sense.  This is probably the most important question you have to ask.  If you are doing a lot of reflection, then it may be worth it to get a library to help you do reflection.  If you only need to do it once, it probably doesn’t make sense.
  • How widespread is the library?  Commons-lang is a widely used library, so having it as a dependency is not likely to cause problems.
  • Are there different versions of the library that people may be using?  As shown in the jOOQ blog post, if you are using Guava 15 while another library uses Guava 18, will that cause problems?  I have never used Guava, so I don’t know if it would actually cause a problem, depending on how the API has changed.

I realize that this is not a comprehensive list, nor is it a perfect guide.  Ultimately, as a library writer you should try as much as possible to keep the number of dependencies down as low as possible.  Otherwise, it can quickly lead to dependency hell for the end users, and potentially for you as well.

Jul 08

Debug Qt on remote device

Have you ever had to debug a Qt program running on a different computer?  There are a few special things that you must do in Qt Creator in order to properly get all of the debug information out from the device.

In my case, I’m debugging a Qt program running on an ARM processor, while debugging through my x86 computer.

First of all, you will need a root filesystem on your local computer that matches what is installed on your board.  See my guide on cross-compling Qt for ARM for information on how to use qemu-debootstrap to create a new rootfilesystem.  Alternatively, this should also work with any other filesystem that you create(e.g. through yocto).

Now that you have your root filesystem, here’s how to setup Qt Creator correctly:

  1. (optional) install the qt sources.  On Debian, you can easily do this with:
    $ apt-get source qt5-default
  2. (optional) Once you have the Qt sources, go to Tools -> Options -> Debugger and click ‘Add Qt sources’ at the bottom and add the source path for Qt.  The source path should be /var/tmp/qt-src, the target path should be the directory in which you executed the apt-get source line above, for example it may be /home/username/qt/qtbase-opensource-src-5.3.2+dfsg/src
  3. Under Tools -> Options -> Debugger, go to the GDB tab and add these additional startup commands:
    set sysroot /path/to/rootfs
    set debug-file-directory /path/to/rootfs/usr/lib/debug
  4. Install the debug symbols in your rootfs:
    $ sudo chroot /path/to/rootfs
    # apt-get install qtbase5-dbg
  5. Go to Window -> Views -> Debugger log.  This will open up a new panel in Qt Creator that gives you the output from GDB.
  6. Run the program with GDB through Qt Creator as you would normally.
  7. When you hit a breakpoint or encounter an exception in the program, send the command ‘sharedlibrary’ to GDB.  This will load the shared libraries, but it will not update your stack trace immediately.  Either switch threads, or through the program for the stack trace to be updated in the GDB view.

You should now have a fully functioning remote debugging system.

Jun 12

Sequel Escalation

I’d like to take a moment to talk to you today about sequel escalation.  In general, this is when the next movie in a series has an even worse bad guy than the previous one.

(This is not limited to only movies though)

I was thinking about this as I watched the Fast and Furious franchise, although this also happens in the new Bond films.  I know that it is in someways inevitable; after all, if the new bad guy was easier to defeat, that would make for a bit of a dull movie, right?  But that is in some ways the problem.  There can only be so many international terrorists and criminal masterminds.  Once you get rid of one of them, a new one does not magically appear.

There has to be some kind of limit here.

But that never seems to happen.

Anyway, just some random thoughts.

Apr 25

About Script Kiddies

I don’t know anything about hacking.  It’s true.

I know the basic theory behind some of the attacks, but in-depth knowledge is beyond me.  I went to school for almost five years to learn how to program, and I can’t even tell you how to hack into a computer.

 

I like to think that the software that I write is immune to being hacked, but I don’t know for sure without having somebody else look for vulnerabilities.  At this point, if I were to try to hack something, I would be a script kiddie.  And that’s not something that you want to be.

At least there are a few guides on how to not become a script kiddie.

Apr 06

Setup Tomcat 8 on Debian 8 on port 80

Alright, that’s a confusing title.

Anyway, the point of this is to show you how to setup Tomcat 8 on Debian 8 (Jessie), and to have it bind to port 80.  This uses just the default packages from Debian, so if you are using a version of Tomcat that you downloaded directly from their website you may be out of luck.

First, install the needed packages(note: this causes Tomcat to start automatically):

$ apt-get install tomcat8 authbind

Now, we need to change the configuration settings for Tomcat and authbind to make sure that they work properly.

  1. Edit /etc/default/tomcat8.  At the very end, there should be a line that says AUTHBIND=no.  Change this to say AUTHBIND=yes
  2. Edit /etc/tomcat8/server.xml.  In this file, there is an XML tag that starts out with <Connector port=”8080″ ….  Change this to be 80.  There are two sections that you want to change this.
  3. Go to /etc/authbind/byport.  Do the following(note: this assumes that tomcat is running as the tomcat8 user, which it will do if you installed it through apt)
    $ cd /etc/authbind/byport
    $ touch 80
    $ chown tomcat8 80
    $ chmod 744 80
    
  4. Restart Tomcat
    $ systemctl restart tomcat8
    

You should now be able to access Tomcat on port 80.