At the day job, I’m still working with Jenkins and our build system. I’ve written about some things I learned at the onset, and now I’d like to add some things I’ve learned since then.

Plugins

Be mindful of them. There’s a great many out there that can help you solve things, but they seem to also bring much risk. For example, Jenkins was going belly-up at least a couple of times a day. I’d look in the system console and see some plugin that I wasn’t even using (just had installed) was somehow causing memory failures, throwing an exception, and Jenkins went stupid and had to be restarted. Once you figure out what plugins you need, uninstall every other plugin. Keep your system as minimal and tight as possible to avoid introducing risk or uncertainty.

Backups

I had to write a lot of custom scripts (some bash, a lot of ruby). These files I can keep in our version control system. But Jenkins files themselves are harder to do. Make sure you have a backup strategy. Make sure you run it often while you work. Working on a Mac and using Time Machine is useful… make a change, screw up, easy to revert.

Updates

Don’t apply an update unless the changelog shows it directly affects you. For example, the git plugin was just updated to fix a regression that prevented using env parameters in the branch name — I need that, so I upgraded that plugin. This goes back to the plugin issues I mentioned above… and if you screw up, it goes back to my backup issue I mention above.

Planning

Do your best to plan your jobs and pipeline. Figure out the abstract ideal that you want. Then figure what Jenkins can offer to get you there. You might find plugins, but you might also have to write your own scripts and other management.

Then, don’t be afraid to revise and revise again, even starting over if you have to.

DIY

I wanted to use a lot of plugins, because they’re supposed to make it easy, right? Well, it depends. The Xcode plugin is supposed to make it easier to do xcodebuild-based, builds. And in theory it does, but our needs are different and so I just have to get dirty with our own scripts.

The email-ext plugin is really cool, but I couldn’t get it to understand and bring in a bunch of env vars and do other bits of logic processing that we needed. So again, scripting to the rescue. Net::SMTP to the rescue.

CodeSign problems

I noticed from time to time that I’d start a build and eventually xcodebuild or ‘xcrun PackageApplication’ would fail:

“There are no valid public/private key pairs in the default keychain”

It seems this would happen after rebooting the machine (and Jenkins started up via the launchd stuff). I could manually use launchctl to unload then (re)load and things might be back to normal, but that’s annoying.

I found this post at stackoverflow.com, and adding the “SessionCreate = true” key/value to the launchd plist seems to do the trick.

But then further down things failed out. Despite the codesign command line tool being granted special Keychain access, it still hated life:

<path to build output app>: User interaction is not allowed.

Uh…

So I found this and added a:

security unlock-keychain -p "" login.keychain

in the custom ruby script before invoking xcodebuild and PackageApplication.

BTW, I don’t have the passwords encoded into the script. Remember the use of the .netrc file? Use that.

Speaking of .netrc… it seems curl doesn’t like a .netrc where newlines separate entries… it wants everything on one line.

Passing Data to Downstream Jobs

Using Jenkins’ parameterized build mechanism, I was able to pass parameters around. I’d have the main build job, which would then use the promoted builds plugin to allow me to move the build through a pipeline, like to promote to QA, then promote to the App Store, or whatever. But there’s a lot of information from the main build job that I want the downstream jobs to have so the jobs can be properly named, emails can be properly formatted, version control tags set up, whatever. I found the easist way to do this was at the end of the main build job to use a little ruby scripting, create a Hash with all the things I cared to preserve, use YAML to preserve it to file, make sure the build job archived and fingerprinted that .yml file, and moved that “config.yml” file along with the rest of my archives and promotions. Then it’s simple enough to load and look up key/values out of the config.yml file in the later job scripts.

Tagging Git

One of the downstream jobs is a “deploy to the Apple app store” job. Certainly when we do that we want to tag our github-hosted repository with info about the build so we can know what was deployed. Trouble is, at that phase of the build pipeline we don’t have source code. So we can’t just “git tag” then “git push”. What to do? Use the github v3 API. In that config.yml file I was able to preserve the SHA hash of the code we used, so that’s all we need.

At first I thought to use the Tags API, but as I played with that it didn’t work. Even if I could create a tag, it wouldn’t show up in the Tags area of my git GUI app SourceTree. In fact, it started to give errors about refs. So I started to play with the References API and tried things like:

$ curl -XPOST --netrc https://api.github.com/repos/:user/:repos/git/refs \
-d '{"ref":"refs/tags/MyTestTag","sha":"be43262431c7a4b9db67a23d37f51e7901b9845c"}'

and lo… that seemed to work.

Is that enough? Is that correct? I don’t know. I’m not an expert on the low-level plumbing of git. I have contacted github support, but as of this writing have yet to hear back.

The Journey Continues

That’s all that I have for now.

It’s a lot of work to set up a build system, but it’s rather satisfying to do it. I still have a long ways to go. Right now we just have the basics for building and deploying to help our general workflow. It still needs lots of work for validations, testing, continuous integration, and other good stuff like that. One step at a time.