Tools of the Trade

Introducing: Jenkins Plugin Updater Command Line Tool

For those of you have been following along, you know that I’ve been working on a Continuous Integration series about working with Jenkins. One of the things that I really appreciate is that plugins are updated at a decent rate (I usually check for updates on Mondays).

However, it’s always a bit of a pain to manually update them and it’d be nice if I could automate the task.

To solve this problem, I wrote a little command line utility that will update installed plugins for Jenkins. I can then tie this into a scheduled task so now on Monday morning, the plugins can be updated before I come into work.

In terms of what my overall goal is, I’m working on a set of tools that can take a new machine install Jenkins, configure plugins, and set it up for .NET development. I’m one step close with this utility. I hope you all get some use from it and let me know what you think.

Tools of the Trade

Tools of the Trade – Continuous Integration (Part 2)

In the first post of the series, I talked about what continuous integration is and why you should use it. In this post, I’m going to start by describing what Jenkins is and the steps on how to set it up. Next, we’ll add plugins that will allow you to checkout a repository using Mercurial and to build the project using MSBuild. Finally, we’ll configure a means of notifying you if a build fails by using Gmail.

What Is Jenkins?

Jenkins is a stand-alone continuous integration server that listens to a repository for changes. Once Jenkins sees that a check-in has occurred, it begins to perform a series of steps (called a build). These steps can range from building the solution, running tests, deploying databases, copying artifacts, building installers, creating documentation, or deploying software. Jenkins (and it’s predecessor Hudson) have been around for quite a while now and is one of most commonly used CI servers around. One of the reasons I like Jenkins is that I can set it up on a virtual machine and have it running all the time.

How Do I Setup Jenkins?

The Jenkins server is pretty easy to setup. First, we will need to install the Java Runtime Environment (JRE). After installing the JRE, we will install the Jenkins tool. Once Jenkins is up and running we can proceed to install the plugins and start configuring the Jenkins instance.

Download and Install Java

The only dependency you need to run Jenkins is to install the Java JRE. You can download the latest run time from here. During the install, choose the defaults. After the JRE installs, you’re ready to install Jenkins.

Download and install Jenkins

After downloading and installing the JRE, you will need to install the Jenkins server by going to http://jenkins-ci.org/ and choosing the appropriate download option for your system. After downloading, extract the zip folder and run the setup application. After the install you will now have a Jenkins service running at http://localhost:8080.

Configuring Jenkins

Now that we have Jenkins up and running, it’s time to extend Jenkins by installing plugins that will allow us checkout out our source code and build the solution. These plugins make Jenkins very powerful since you can install which ones you need for your jobs. The following sections will detail how to install plugins in general and how to install the MSBuild and Mercurial plugins in detail.

Installing Plugins

As mentioned earlier, before we can create a job, we need to install the plugins required for said job. At the bare minimum for our example, you will need to install a plugin for your source control and a plugin for compilation.

In order to install a plugin, on the Jenkins machine, navigate to http://localhost:8080 and go to Manage Jenkins -> Manage Plugins (next to the green puzzle piece).

On this screen, you can see which plugins have updates, which plugins you have installed, and which plugins that are available for download. When adding/removing plugins, you will come to this area often.

Installing Mercurial

Now on the Plugins screen, navigate to the Available Plugins tab and using the filter box, search for the Mercurial plugin and select the option shown:

Installing MSBuild

After selecting Mercurial, it’s time to install MSBuild. In the filter box, type MSBuild and select the option below:

After selecting the MSBuild plugin, it’s time to install the plugins by clicking on the Download now and install after restart button. After clicking the button, click the check box and then Jenkins will restart.

After Jenkins restarts, it’s time to configure the plugins to be able to use them in a job.

Configuring System-Wide Properties of Jenkins

At this point, we have Jenkins up and running and our plugins installed. However, we need to configure these plugins so we can use them in a job. To configure plugins, we need to configure the system-wide settings of Jenkins. To get to this menu, navigate to http://localhost:8080 and click Manage Jenkins -> Configure System.

On this screen, every plugin installed with Jenkins is listed with some global settings for that plugin. In general, a plugin will need to know where the tool it uses resides at. For example, the Mercurial plugin needs to know where the mercurial executable is on the machine.

Configuring the Mercurial Plugin

Before you can configure the plugin, you will need to have Mercurial installed (Need to install Mercurial?, look no further). Not sure if Mercurial is installed? Open up a command prompt and enter “hg version”. If Mercurial has been installed, then you should see the version of Mercurial installed, otherwise, you probably don’t have it installed.

To configure the plugin, navigate to the Configure System screen and look for the Mercurial section. After finding the section, click the Mercurial Installations button.

After clicking this button, there will be some options to configure. For this setup, the most important ones are Name, Installation Directory, and Executable.

The Name field is what’s shown when selecting the source control for a build, so make sure it’s descriptive. I usually include the version number as part of the name.

The Installation Directory field is where the Mercurial executable is located. If you Mercurial is part of your path, then you don’t need to specify this, otherwise, just enter the directory that the executable is located.

The Executable field is where you specify what the name of your mercurial executable is. The majority of the time, hg.exe is the name you want, so shouldn’t have to change this.

Configuring MSBuild

Configuring the MSBuild plugin is similar to configuring the Mercurial plugin. As before, navigate to the Configure System screen and find the MSBuild section. From here, find the MSBuild section and click the MSBuild Installations button.

After clicking this button, there will be some options to configure. For this setup, the most important ones are Name, Path To MSBuild, and Default Parameters.

The Name field is what’s shown when selecting the compiler for a build, so make sure it’s descriptive. I usually include the version number as part of the name.

The Path To MSBuild field is where the MSBuild executable is located. By clicking the ? button, it will tell you what the default path is for MSBuild. Since the MSBuild is included in every Windows machine, this location should be the one to use. Make sure you add msbuild.exe to the end of the path (even though you will get a warning message, you need to provide the full path).

The Default Parameters field is where you can specify additional command switches to MSBuild. This can be helpful if you have a certain configuration you want used when building (like build in release mode). It’s not necessary to add any switches for MSBuild to work out of the box, so I’m omitting it here.

Configuring Email Notifications

At this point, we have the source control and compiler plugins configured, but the last big thing that we need to do is setup some way to be notified if a build fails. In this example, we’ll be setting up email notifications, however, there are plugins for many popular communication tools (Slack, HipChat, etc..).

To configure the email notifications, go to the System Configuration screen and find the Jenkins Location section. Here, specify the System Admin e-mail address for Jenkins (this does not have to be a valid email address, but it does need to be entered).

After entering an email for Jenkins, find the E-mail Notification section. This is where we will specify what mail server and other options to send out the notifications.

Since we’re using Gmail for our communications, we need to specify that in the SMTP server field and we need to click the Advanced… button to have some more configuration.

After clicking the button, entering the following values for each field (substituting your own Gmail address and password)

To ensure that the email notification is configured correctly, click the Test Configuration by sending test e-mail button and enter a valid email address.

When you click the button, you should get a success message. If, however, you get the following message, it’s probably due to Google locking down access to less secure 3rd party applications.

If that’s the case, you will need to tweak your Gmail settings (http://accounts.google.com/security) so that less secure applications can access your account (see below).

Summary

In this post, we installed Jenkins, a continuous integration server. Next, we installed the Mercurial and MSBuild plugins. After installing the plugins, we configured them to work with the installed tools. Finally, we configured Jenkins to send emails notifications through Gmail that allows Jenkins to alert us when a build fails.

In the final post for the series, we will be creating a simple job that will checkout a .NET solution, compile it, and deploying it to a different server.

Tools of the Trade

Tools of the Trade – Getting Started With Continuous Integration (Part 1)

In this post, I’m going to talk about a commonly used software practice, continuous integration, by first describing what it is and the problem it solves. Next, we’ll look at what the continuous integration process looks like and what some common steps are. Finally, I’ll outline some common arguments and counterpoints to them.

What is Continuous Integration?

First thing’s first. What do we mean when we say continuous integration? From Wikipedia

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. . . .The main aim of CI is to prevent integration problems, referred to as “integration hell” in early descriptions of XP (Extreme Programming).

Long story short, continuous integration is the act of merging changes (i.e. pull requests) into the main branch multiple times a day (instead of one big push daily) to minimize the effort needed to merge those changes. If you’ve worked on any project with multiple contributors, then you realize that merging changes can be a tough business, but with continuous integration, you can verify that your merge didn’t break anything.

Something that needs to be mentioned is that you will hear the terms continuous integration and continuous build used interchangeably. In the beginning, these terms meant different things, but now they have become synonymous. With that being said, my experience is that you will more likely hear the phrase “CI Server” more so than “Continuous Build Server”.

So What Does Continuous Integration Look Like?

To implement continuous integration, you will need a server that listens to your repository for changes. When changes have been detected, the server will then run a series of steps in order to ensure that the software still builds and passes tests. A sample workflow would look like this:

Checkout the latest copy of the source code
Compile the source code.
Run automated tests
Run deployment scripts (this could be creating an installer or by deploying a website)
Send an email notification with the completion status (success, or failure if one of the steps failed)

The beauty of this process is that you’re defining the steps that constitute a stable version of your code. If all of these steps pass, then you know that your changes/merges did not break functionality and you have a stable build to run tests on (i.e. the build is not broken).

On the other hand, if any of these steps fail, then you have a much smaller scope of changes to look at (the last check-in) and what step did they fail on? For example, if the automated tests failed, why did they fail and do they fail on your machine?

Why Should I Use It?

In this piece, we’ll explore possible reasons why you’re not using CI and provide counterexamples to each of these points.

I don’t have time to setup CI

For the most part, the only resources that a continuous build server takes is the initial configuration. After that, it’s pretty much “set it and forget it”. Even with no experience setting up the CI server, you can easily go from nothing to up and running in a couple of hours.

I’m a sole developer so I don’t have merge issues

Granted, the odds of having a merge issue are lower, I still find it very unlikely that you will never have merge issues. The main reason is that rarely will you have only one change going on at a time. It’s much more likely that you’ll be working on a new feature and still be doing bug fixes. If you follow best practices for Git or Mercurial by branching per change, you will eventually have merge issues.

However, let’s pretend that you never have merge issues, how do you guarantee that the changes you’ve made haven’t broken anything? By running the code on your own machine (i.e. it works on my machine, but not in production)? By deploying to QA? None of these solutions really work.

It takes too long to maintain the server and/or the job

For this argument, let’s approach it two ways. First, if maintaining the server is taking too much of your time (or you don’t have the IT resources), go ahead and choose a cloud solution. The rates are reasonable, (for example Shippable offers a great package for $12 a year) and they handle the updating of the server for you. Second, once the job is setup, it will need little tweaking. Maybe to add another step, but for the most part, you’re build process won’t change a whole lot. And even if it does, it’s as simple as adding an additional step or removing a step.

I don’t care about catching merge issues, that’s what QA is for

Unfortunately, I’ve gotten into many arguments with developers who don’t use continuous integration and why they don’t use it. My argument is essentially that a developer’s job to either save the company money (by solving the company’s problems) or to make the company money (by creating a service/product).

Given this, it’s in our best sense to find and fix issues when they’re the cheapest and the earlier issues are found, the cheaper they are to fix (source). By the time QA has found an issue, it’s much more costly to fix the bug than if it was found at the development stage.

Wrapping Things Up

In this post, I introduced continuous integration and the problem it was supposed to solve. In addition, we looked at what continuous integration looks like by explaining the different build steps. Finally, we wrapped up with some common arguments against using CI and how to counter them.

Next time, we’ll explore setting up a Jenkins CI server and configuring plugins.