Building in isolation with Docker

by Bruce Szalwinski


Over at Device Detection, I wrote about creating an Apache Handler that could be used to do real time device detection.  The handler has a number of dependencies on other Perl modules, 51Degrees, JSON, Apache2::Filter, Apache2:RequestRec, Apache2::RequestUtil, etc.  And those modules have dependencies as well.  I wanted our build server, Bamboo, to do the building of my module without the nasty side effects of having to install third party libs into all of the build agents.  In the Maven world, I would just add all of these dependencies to my pom.xml and maven would download the dependencies from the repository into my local environment.  At build time, Bamboo would take care of establishing a clean environment, download my dependencies and most importantly, when the build was complete, the plates would be wiped clean ready to serve another guest leaving no traces of my build party.  The challenge then, how to do this in Perl world.  Spoiler alert, full source code is available at DeviceDetection.

Enter Docker


The fancy new way for developers to deliver applications to production is via Docker.  Developers define the application dependencies via a plain old text file, conventionally named Dockerfile.  Using the Docker toolkit, the Dockerfile is used to build a portable image that can be deployed to any environment.  A running image is known as a container, which behaves like an operating system running inside of a host operating system.  This is a lot like VMs but lighter weight.  Developers are now empowered to deliver an immutable container to production.  Let’s see if this can also be used to provide an isolated build environment.


For the big picture folks, here is what we are trying to do.  We’ll start by defining all of the application dependencies in our Dockerfile.  We’ll use the Docker toolkit to build an image from this Dockerfile.  We’ll run the image to produce a container.  The container’s job will be to build and test the Perl modules and when all tests are successful, produce an RPM.

Building the image was an iterative process and I had fun dusting off my sysadmin hat.  Here is where I finally ended up.

FROM google/debian:wheezy
RUN    apt-get -y install make gcc build-essential sudo
RUN    apt-get -y install apache2-threaded-dev
RUN    apt-get -y install libapache2-mod-perl2
RUN    apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl
RUN apt-get -y install rpm

Let’s break this down.   The first non-comment line in the Dockerfile must be the “FROM” command.  This defines the image upon which our image will be based.  I’m using the “google/debian” image tagged as “wheezy”.   Think of images as layers.  Each image may have dependencies on images below it.  Eventually, you get to a base image, which is defined as an image without a parent.

FROM google/debian:wheezy

The RUN command is used to add layers to the image, creating a new image with each successful command.  The 51Degrees Perl module is built using the traditional Makefile.PL process, so we start by installing the make, gcc and build-essentials.  Containers generally run as root so we wouldn’t normally need to install sudo, but our handler uses Apache::Test for its unit test and Apache::Test doesn’t allow root to create the required httpd process.  So we will end up running our install as a non-root user and give that user sudo capabilities.  More about that in a bit.

RUN     apt-get -y install make gcc build-essential sudo

Next, we install our apache environment.  With Apache2, there is a pre-fork and a threaded version which has to do with how apache handles multi-processing.  For my purposes, I didn’t really care which one I picked. It was important however to pickup the -dev version as this includes additional testing features.

RUN     apt-get -y install apache2-threaded-dev

Next, we install mod-perl since the device detector is a mod perl handler.

RUN     apt-get -y install libapache2-mod-perl2

Next, add our Perl dependencies.  Each Linux distro has its own way of naming Perl modules.  Why?  Because they can.  Debian uses “lib” prefix and “-perl” suffix, converts “::” to “-“, and lower cases everything.  To install the Perl module known as “Test::Harness”, you would request “libtest-harness-perl”.

RUN     apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl

And since we’ll be delivering a couple of RPMs at the end of this, we install the rpm package.

RUN apt-get -y install rpm

With the Dockerfile in place, it is time to build our image.  We tell docker to build our image and tag it as “device-detection”.  We tell docker to look in the current directory for a file named Dockerfile.

$ docker build -t device-detection .

Time for some coffee as docker downloads the internet and builds our image.  Here is the pretty version of the log produced after the initial construction of the image.  If there are no changes to the Dockerfile, then the image is just assembled from the cached results.  The 12 character hex strings are the ids (full ids are really  64 characters long) of the images that are saved after each step.

Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM google/debian:wheezy
 ---> 11971b6377ef
Step 1 : RUN apt-get -y install make gcc build-essential sudo
 ---> Using cache
 ---> 2438117da917
Step 2 : RUN apt-get -y install apache2-threaded-dev
 ---> Using cache
 ---> 41f878809025
Step 3 : RUN apt-get -y install libapache2-mod-perl2
 ---> Using cache
 ---> 43eadc4ec9eb
Step 4 : RUN apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl
 ---> Using cache
 ---> 106d5f017b5c
Step 5 : RUN apt-get -y install rpm
 ---> Using cache
 ---> fd0dc5f192d6
Successfully built fd0dc5f192d6

Use the docker images to see the images that have been built.  The ubuntu/14.04 was before I got religion and started using google/debian.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
device-detection    latest              2f2b3e38e8c1        29 minutes ago      358.6 MB
ubuntu              14.04               d0955f21bf24        2 weeks ago         188.3 MB
google/debian       wheezy              11971b6377ef        9 weeks ago         88.2 MB

Running the Image

At this point, we have an image that contains our isolated build environment.  Now we are ready to do some building by running the image.  In Docker terms, a running image is known as a container.  The build-docker script will be used to produce a container.   When we create our Bamboo build plan, this is the script that we will execute.

docker run --rm -v $PWD:/opt/51d device-detection:latest /opt/51d/

The –rm removes the container when finished. The -v mounts the current directory as /opt/51d inside of the container.  The device-detection:latest refers to our image that we just built.  And finally, the /opt/51d/ is the command to execute inside of the container.

adduser --disabled-password --gecos '' r
adduser r sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
su -m r -c /opt/51d/build

The script will be executed inside of the container.  To test the handler, we’ll need the vendor’s module installed.  And to install modules, we need to have root privileges.  We are going to use Apache::Test to test the handler but Apache::Test won’t let us start the httpd process as root.  The solution is to create a new user, r, and give him sudo capabilities.  With that in place, we hand off execution to the next process /opt/51d/build.

That all worked well on my local environment, but something interesting happened when I went to deploy this from Bamboo. The owner of the files in the container turned out to be the user that built the container.  I was the one building the container in my local environment, but I wasn’t the one building the container inside of Bamboo.  When the user ‘r’ attempted to create a file, it got a permission denied error because the directories are not owned by him.  I discovered this by having Bamboo list the files from inside the running container. They are owned by the mysterious user with UID:GID of 3366:777.

build   08-Apr-2015 09:28:35    /opt/51d:
build   08-Apr-2015 09:28:35    total 28
build   08-Apr-2015 09:28:35    drwxr-xr-x 7 3366 777 4096 Apr  8 16:28 51Degrees-PatternWrapper-Perl
build   08-Apr-2015 09:28:35    drwxr-xr-x 5 3366 777 4096 Apr  8 16:28 CDK-51DegreesFilter
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  530 Apr  8 16:28 build
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  120 Apr  8 16:28 build-docker
build   08-Apr-2015 09:28:35    drwxr-xr-x 2 3366 777 4096 Apr  8 16:28 docker
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  146 Apr  8 16:28
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  497 Apr  8 16:28

We can use this UID:GID information when creating our user. The stat command can be used to return the UID and GID of a file.  We’ll create a group associated with the group that owns the /opt/51d directory and then we’ll create our user with the UID and GID associated with the owner of the directory.  Our modified script is then:

addgroup --gid=$(stat -c %g /opt/51d) r
adduser --disabled-password --gecos '' --uid=$(stat -c %u /opt/51d) --gid=$(stat -c %g /opt/51d) r
adduser r sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
su -m r -c /opt/51d/build

And we can see that user ‘r’ is now the “owner” of the files.

build   08-Apr-2015 09:28:35    /opt/51d:
build   08-Apr-2015 09:28:35    total 28
build   08-Apr-2015 09:28:35    drwxr-xr-x 7 r r 4096 Apr  8 16:28 51Degrees-PatternWrapper-Perl
build   08-Apr-2015 09:28:35    drwxr-xr-x 5 r r 4096 Apr  8 16:28 CDK-51DegreesFilter
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  530 Apr  8 16:28 build
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  120 Apr  8 16:28 build-docker
build   08-Apr-2015 09:28:35    drwxr-xr-x 2 r r 4096 Apr  8 16:28 docker
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  146 Apr  8 16:28
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  497 Apr  8 16:28

With the user setup, hands off control to the build script to do the heavy lifting.  Here we setup our apache environment and start building the two Perl modules.  The () is a convient bash-ism that creates a sub-process, leaving us in the current directory when completed.  And the PERL_TEST_HARNESS_DUMP_TAP is an environment variable recognized by Tap::Formatter:Junit package.  Unit tests will live at the location specified by this variable.


source /etc/apache2/envvars
export APACHE_TEST_HTTPD=/usr/sbin/apache2
export PERL_TEST_HARNESS_DUMP_TAP=/opt/51d/CDK-51DegreesFilter/dist/results

(cd /opt/51d/51Degrees-PatternWrapper-Perl && \
        perl Makefile.PL && \
        make && \
        make dist && \
        sudo make install && \
        ../ FiftyOneDegrees-PatternV3-0.01.tar.gz)

(cd /opt/51d/CDK-51DegreesFilter && \
        perl Build.PL && \
        ./Build && \
        ./Build test && \
        ./Build dist && \
        ../ CDK-51DegreesFilter-0.01.tar.gz)

When the build script completes, we are done and the container is stopped and removed.  Because we have mounted the current directory inside of container, artifacts produced by the container are available after the build completes.  This is exactly the side effect we need to have.  We can publish the tests results produced by the build process as well as the RPMs.  And we have accomplished the goal of having an isolated build environment, D’oh!

Fun things learned along the way

Inside of the container, Apache::Test starts an httpd server on port 8529. It then tries to setup the mod_cgi library by binding a socket to a filehandle in the /opt/51d/CDK-51DegreesFilter/t/logs directory via this directive:

<IfModule mod_cgid.c>
    ScriptSock /opt/51d/CDK-51DegreesFilter/t/logs/cgisock

The httpd server had issues with this, not sure why, perhaps because Docker is binding the file system as well.  I resolved it by moving the ScriptSock location to /tmp/cgisock.  More details on this conundrum are available at stackoverflow where I asked and answered my own question,

Escaping Technical Debt

By Osman Shoukry (@oshoukry) & Kris Young (@thehybridform)

The Visit

On October 6th we had Michael Feathers (@mfeathers) author of Working Effectively With Legacy Code visit our facility.  The visit was two achieve two objectives.  The first was to give tech talks to our engineers about legacy code.  The second was to train selected key individuals in the organization.  Specifically, techniques and skills on how to deal with legacy code.

Mr. Feathers graciously agreed to give a recorded community talk about Escaping Technical Debt.


Key Takeaways

  • Tech debt

Technical debt is a metaphor for the amount of resistance in a system to change.  The larger the debt the higher the resistance.  When change is introduced, the time spent looking for where to make a change is one example of tech debt.  Another is when the system breaks in unexpected ways.

  • It takes a community to mitigate technical debt

Tech debt affects everybody, from engineers, to product owners to the CEO.  Mitigating tech debt requires everyone’s support and involvement.  As Engineers, we are responsible for how to mitigate technical debt.  The product owners should have input on tech debt mitigation effort.  The benefits of tech debt cleanup should be visible to everyone.  Don’t surprise your peers or management by the sudden change in productivity.  They will bring ideas to you to help pay down the debt in more enabling ways for the future… Involve them!

  • Don’t pay the dead

Code that doesn’t change, even if it is in production, doesn’t collect debt.  Dead debt is any part of the system that doesn’t change including bugs and poor design.  Many times engineers get wrapped up cleaning code that isn’t changing.  Resist the urge to refactor unchanging parts of the system.  If it isn’t changing it, it isn’t technical debt, it is dead debt, walk away.

  • Size your debt

Target the most complex changing code ordered by frequency of change.  Build a dashboard to show the amount of changing code by frequency.  These are the most expensive tech debt hot spots.  This should be the focus of the tech debt cleanup effort.

  • Seal the leak

Technical debt should be paid down immediately.  Simple code is easy to make more complex, and easy to make simple.  However, as the code becomes more complex, the balance tips in the favor of adding complexity than removing it.  No matter how complex the code is, it is always going to be easier to add complexity than remove it.  Inexperienced engineers, fail to see the initial complexity that is added to the system.  In turn, they follow the path of least resistance making the code more complex.

To seal the leak, first identify the most complex and frequently changing code.  Second, give explicit ownership for that code to the most seasoned engineers.  Third, let the seasoned engineers publish the eventual design objectives.  And finally, owners should review all changes to the code they own.

  • Slow down to go fast

Clean code takes time and disciplined effort.  Adequate time is needed to name classes, methods and variables.  Poorly named methods and classes will create confusion.  Confusion will lead to mixing roles and responsibilities.


Finally, low tech debt will yield high dividends with compound interest…

“Don’t do half assed, just do half”

Global Day Of Code Retreat 2013

By Osman Shoukry – Director, Software Engineering – Digital Advertising Platform @oshoukry.

Last November I attended the Software Craftsmanship North America (SCNA), the conference was very informative and I got introduced to some valuable techniques around developing craftsmanship within our company one of which is Code Retreat-ing.

What is Code Retreat?

Code retreat is a full day event where software engineers get together to deliberately practice their craft.  It is a forum to enable working outside of the normal comfort zone.  The reality is that most professional software engineers enter the professional world – wether self taught or having received formal education, without sufficient training into what is required to write great code.

Joining forces

Having returned from SCNA 2013, and gotten jazzed up with all that I saw, and excited to try out code retreats, I decided I’d organize one for my teams to try.  While doing the research on line to read more about it I stumbled on and saw that there is an international day of code retreat to be held on Dec 14th 2013.  I looked up Seattle events and found that Getty Images – a company in our building is sponsoring the event but it was all sold out.  I contacted the organizer (Andrew Parker @aparker42) and offered our Cobalt space.  We were all set and the event received additional sign up spots.

The Six Sessions

On the day of the event, Andrew and I decided to run the event in the Cobalt space bringing everyone together.  We had 42 crafts[men|women] that gave up their Saturday to attend this deliberate practice event.  We setup the tables to facilitate pairing with power and networking.  We started @8:30 with breakfast and socializing and then we got down to business.

photo 4

Session 1 – 9:15 am (Get familiar with the problem)

For the first session, we introduced the problem – Conway’s Game of Life, and let everyone work on it without any restrictions and constraints. We ran a count down timer with 45 min on the big screens.  With 5 minutes left, Andrew threw in a change to the problem, which was, if a cell was alive, then died then came back to life, it should now be considered a zombie and will never die again.  The purpose of this curve ball was to get everyone to think about how they would change their design.

After 45 min, it was “Delete your code, How did it go? and 5 min break”.  Many thought deleting was a very weird thing to do, however, there were a few that were relieved to see the code – aka the mess – get purged.

Some got quite far with the solution almost solving the problem fully, while others reported not much was accomplished because it took a while to understand how to approach the problem.

Everyone switched pairs and started again on a blank slate, people got to discover others who have languages they would like to learn to pair up with those individuals.  One engineer held an iPad up high as a sign, it read “I need Python”.

Session 2 – 10:15 am (Ping Pong TDD)

We noticed in session one there was a lot of domination of keyboard and mouse by one of the pairs, so we decided this would be a good next exercise.  The rule was to toggle back and forth, pair to write a test, the other makes it pass and adds a new test, returning back to the first pair.

This introduced a healthy dynamic and we saw a lot more collaboration, while walking around though, we noticed that some engineers kept the code from session 1.  This was interesting reinforcement that we – engineers, have a very hard time letting go of something that we worked so hard to put together.  And so we needed a new strategy.

At the end of this session, we requested that the right pair stand up when the left pair has successfully purged the code, and this practice carried on for all the remaining sessions.

We were planning to do a “Mute TDD ping pong” session after this one, where the pair aren’t allowed to verbally communicate but utilize only tests to drive the solution but opted against it.  We didn’t have enough mass in the group that were verse enough with TDD to make it effective.

Session 3 – 11:15 am (No Loops, No Conditionals & Methods < 4 lines)

This was probably one of the hardest sessions, many had a very hard time dealing with alive vs. dead cell state not utilizing boolean logic.  Not to mention lack of loops forces recursive solutions which aren’t readily available.  Again, there was different takes on this, some found the constraints insurmountable and others were able to make some progress, overall, it wasn’t easy, perhaps breaking the constraints apart into separate sessions next time may be more focused and provide a better learning experience.

Lunch Break 12:15 – 1:05

We had pre-ordered from Chipotle for most, and augmented the menu with sandwiches from Jimmy Johns.

Session 4 – 1:15 pm (Immutable)

For this session, everything had to be immutable, once you create an instance of something, you couldn’t modify it.  Most were able to cope with the constraint until it came to collections or array manipulation in languages like Java, how to add a new element since creation and appending must be on two steps.  We ended up modifying the restriction a bit for languages that make it unreasonable to allow adding a utility method that helps you with the appending collections / arrays as an exception to the rule.

As we were walking around we noticed two patterns:

Screen Shot 2013-12-18 at 1.34.44 PM


Screen Shot 2013-12-18 at 1.34.51 PM

Both of those looping techniques violate immutability because the loop variable is being mutated.  So the immutable restriction, meant for most no loops allowed unless you were creative enough to push the terminating condition to some method call return within the loop.  Feedback was received and welcomed by all, and alternate solutions materialized quickly.

Session 5 – 2:15 pm (No returns – Tell Don’t Ask)

This session is the exact opposite of the prior session, since all methods had to be declared with void return type, callbacks and other mechanism needed to be utilized for passing back values.  Of course there was the occasional “make it global variable” but all in all, the pairs found that the sessions provided a healthy and not frustrating challeng.  Some found it quite easy given their language of choice lent itself naturally to this style of programming (i.e. Java Script).  By this session, some folks were getting tired, we broke out some candy and snacks to help.  One pair was exhausted and decided to take a break this session.


Session 6 – 3:15 pm (Free form)

The final session, we gave the pairs the option to select any restriction to operate under and let them select.  A few were exhausted by then, but most persevered.

Goodies – 4:15 pm

We had two sets of things to give away that day.  Thanks to those sponsors for making everyone a winner.

  1. O’REILLY gave out one free ebook to everyone who attended a code retreat.
  2. C<>de School gave out “48 hours of Code School” to everyone who attended.
  3. Pragmatic Bookshelf gave out “25% off any published item” to everyone who attended.
  4. Solano Labs gave out “One month of free Solano Labs’ CI Service” to everyone who attended.
  5. PluralSight had some swag that we ruffled among the participants.


This code retreat session was quite valuable to everyone who participated, I think the biggest surprise to everyone who attended was the diversity of the possible solutions to such an easy to understand problem.

Craftsmanship is hard work, it takes a great deal of effort to communicate clearly the intent, and then proceed, and 45 min go by very quickly.

As a co-facilitator with Andrew Parker, I walked away understanding at a deeper level a few things.  I came face to face with understanding how truly amazing some of the engineers are and how desperate our craft is for forums to enable experimentation, quick learnings through failing safely.  Additionally, how much fun it is to be part of or facilitate a code retreat.

Many thanks to Andrew Parker @aparker42 for all his help organizing and facilitating the retreat, Michael Ibarra @bm2yogi for facilitating the initial event avenue – GettyImages.

%d bloggers like this: