Wednesday, 3 August 2016

Achieving goals using TeamCity

As a small company of 1 + sub contractors, I have very little room or patience for doing things I don't need to do. The concept of being an architecture astronaut is not new to me, as I have both designed and coded myself into a corner in the past so I am acutely aware of the YAGNI philosophy.

A very important class of tasks fall into the bucket of "things I need to do, but manually." This might include:

  1. Updating a sales database
  2. Generating development metrics to guage project health
  3. Monitoring license count/usage, bumping the count if needed or "upselling" the end user
  4. Announcing releases
These are all real tasks that I need to do every day/week/month. All of these are manual. Slowly, but surely, I am replacing them with automated scripts. That's not really too interesting.

The interesting part is that I'm using TeamCity to create the workflows to support all of these. At a quick glance:

TeamCity: not just for building software

Some projects/configurations are hidden, but this gives you the general idea. I not only use TeamCity for building and testing the software I write, but I use it to export sales, update mailing lists, generating development metrics and even automatically creating blog posts/emails announcing new releases.

Why would I do this?

Basically, I am buying myself time. By putting in a little bit of work ahead of time, I save myself man-months of work in the future.

What this really allows me to do, and is the ultimate goal, is focus on talking with customers. To create conversations with customers, you have to give them something they can talk about in the first place.

So every thought process is geared towards getting the customer communicating their pleasure, or displeasure with software they use.

Why is TeamCity the right tool to do this?

Well, for me it works because:
  1. I already use it and know it
  2. TeamCity has a very good concept of inputs and outputs: build artifacts. This is primarily what makes my non-build workflows possible
  3. My design process is agnostic of the tools/techniques I use, and so I can fit in whatever tool works best.
But before I choose the tool, I have a process for choosing an implementation. I've outlined the process below. I cannot take credit for this process as it was taught to me and I've honed it in the last little while.

The process

1. Choose a goal

Before I choose a tool to do the job, I look at the goal. While the high-level goal is obviously customer satisfaction (80% of people choose the "It's just perfect - thanks mom" option in recent surveys so I'm hitting that target pretty well), there are sub-goals that will help me get there. For an example, let's pick the "Notify customers of an update" goal.

2. Create a design

Next, I create a design to implement the goal. For example, here is the design I used to determine what happens when I commit a change. It's pretty standard though this evolved over the period of a few months. The purpose of creating this design is to let me go back to it and modify it when things change. Nothing is in my head, it's all here. This is a huge difference from only a couple of years ago where I would keep everything in my head.

3. Mark up the design where tools implement the required functionality

The next step is usually to print it out and circle places where I think tools will do the job I need. I simulated this with SnagIt as I only JUST recycled my actual notes for this particular diagram. Murphy's Law strikes again.

Why do this? Very simple: who wants to write a ton of code that isn't necessary? Outsource as much as possible to free (as in beer) software, commercial, whatever is needed to get you to your goal.

By doing things in this way, and keeping an eye on choosing things that let me do as little work as possible, I end up with a very lean design and most importantly, a lean implementation.

You can see the big arrow is the key piece here allowing me to outsource nearly 100% of the "notify users of an update" functionality. By linking in certain libraries on certain platforms and generating a very specific format of an update manifest, I can support updates on Windows, OSX and Linux.

In particular, Linux was super simple. Can you match the code below to the design above? It should be pretty straightforward.

cd $WLA_ROOT/build
export GNUPGHOME=$WLA_ROOT/dev/ci/keys/gpg
$WLA_ROOT/dev/scripts/run_gpg_command dpkg-sig -g " --yes" \
         --sign builder *.deb && \
    rm -rf repo && \
    mkdir repo && \
    cp *.deb repo && \

    cd repo && \
        (dpkg-scanpackages . /dev/null > Packages) && \
        (bzip2 -kf Packages) && \
        (apt-ftparchive release . > Release) && \
            ($WLA_ROOT/dev/scripts/run_gpg_command gpg --yes -abs -o Release.gpg Release) && \
            (gpg --export --armour > archive.key)

So... Why did you choose TeamCity when an actual workflow tool would do just fine?

Here is a version of the markup that makes it obvious where TeamCity can help: the package and manifest are treated as artifacts. The screenshot below, directly from TeamCity shows how treating the inputs and outputs as build artifacts allows the use of TeamCity as a tool for the rest of the process (after building and testing is complete):

Inputs for the "notify users of an update" box matching nearly 1:1 from design above. Linux has "15" because the "manifest" is a directory with a bunch of files in it so apt/yum  can do their job. Technically, I could zip that directory up and make it a single artifact.

What else do I get for free?

This is my favourite part of doing things like above: you get a ton for free. By integrating the everything into TeamCity, I get:
  1. Build history
  2. Artifact history
  3. Configuration management
  4. Project metrics (I specifically track LOC to make sure I'm not going too crazy):


I just wanted to record my thought process for posterity. Enjoy!