Who broke the build?

You'll never break the build again!

Warning to regular readers – this post is “Geek Factor 10”

At PaperCut we use continuous integration to monitor the quality of our codebase. It doesn’t happen often, but occasionally someone makes a mistake and the build gets broken. It’s important this is detected and rectified early so other members of the dev team are not inconvenienced.

Traditional Lava Lamps, and of course email, are the tools of choice to notify a team that the build is broken. However, lava lamps, just like pink curtains and shag pile, don’t really fit into the culture here at PaperCut. We thought we could do better, and so a new open source project called Retaliation was born!

Retaliation is a Jenkins CI build monitor that automatically coordinates a foam missile counter-attack against the developer who breaks the build. It does this by playing a pre-programmed control sequence to a USB Foam Missile Launcher to target the offending code monkey. Check out the video to see Tom take one in the back of the head all because of a missing semicolon!

At a deeper level Retaliation is more than just a “simple python script”. It’s a radical rethink into how to manage software development teams and the software development lifecycle. It works on a deep psychological level to offer vast productivity improvements. The primal threat of mutually assured destruction lurking in every coder’s psyche ensures that even your sloppiest developers will never forget to check in that missing file again!

Like all good apps Retaliation has testimonials that sing its praise…

Retaliation brought us the productivity improvement pair programming promised but could never deliver! We’ve seen a 13.37% decrease in build breakage since its implementation.
Will, Chief Code Hacker

Honestly, would you work in a dev team with a lava lamp build notifier? What next? Nyan Cat mouse mats? Real coders work under the threat of Retaliation!
Matt, Coding Machine

Does what it says on the box. I’ve seen improvements in my team and we haven’t even installed it yet! Just the threat has kicked my team’s coding into line!
Tom, Head Code Captain

It’s great to see that Chris’ skills in reverse engineering printer binary protocols has not gone to waste. He’s been able to apply this skill to a new field in reverse engineering Foam Missile Launcher system USB protocols! Coding print management software may not be the most glamorous job, however it just got that bit more sexy now that Retaliation is involved!


Subscribe to new posts:

This entry was posted in General. Bookmark the permalink.
  • rossm

    Awesome, I was impressed at the speed and accuracy (or Tom has a big head?)
    Perhaps time to ditch the print management software and focus on Retaliation.

  • Chris

    @rossm Tom did have a big head, but it shrinks each time he breaks the build :-) Unfortunately for Tom he’s almost the closes to the missile launcher so it’s hard to miss.

  • Ricardo

    We got one of these at the Santiago Stock Exchange dev team… it also has a laser aiming point that is REALLY scary :D

  • http://blogs.windwardreports.com/davidt David Thielen

    I think this is fundamentally a horrible idea. You can either punish failure or you can reward success. You can’t do both because constantly striving for success means there will be a lot of failures along the way.

    Yes, you can strive for success elsewhere while saying a build should always compile when it checks in. Very fair. The problem is when a broken build is handled with a swift hit in the head. That’s a heavy punishment for failure and that will impact how willing people are to possibly fail elsewhere.

  • Viktor

    I’m with David. It’s a horrible idea. The look on Chris’ face says all.

  • Pedro Mendes

    Sweet! :)

  • Missile Commander

    David: we disagree. Missiles launched!

  • Pingback: Twitted by johnlkinsella

  • Nerf Man

    @David: Punishment?? Real punishment would include a financial or career impact. Getting shot at by a Nerf gun is not punishment, it is about as gentle reminder as possible that you just did something that impacted the progress of the team.

    I would hate to work in such a politically correct workplace where mistakes were ignored. There are consequences to breaking a build.

  • http://mig5.net mig5

    That is awesome. And thanks for opensourcing the code :D

  • Chris

    @David I this your sentiments would be true if we were a Highbrow development team. We are however quite relaxed and this is all done in good fun. A broken build, because it’s not yet release software, is quite inconsequently and hence not really negative. It’s a fun way of dealing with what can be a monotonous process – fixing the build! We’re probably not going to have this active every day for the next 10 years, however in the meantime it’s all fun. No one is taking it seriously and it’s proven to be a good way of energising our team at “random” intervals!

  • Jake

    I think this is very cool! I also don’t see how its a punishment. It’s fun!

    Wow. Punishment? Really?

  • Infinity

    Definitely fun. People who don’t see that may have had their sense of humour crushed by the corporate machine. :)

    The next step is to support multiple launchers, so that it still works in offices that aren’t open plan, or have high partitions.

    “Bravo launcher, Bravo launcher, we have a code Red Build. Your coordinates are cubicle Delta-Oh-Nine. Bravo launcher you are clear to fire…”

    • Andrew Wolfe

      Also please note – remote offices worldwide can have Retaliation in every office area.

    • Andrew Wolfe

      P.S. I think the voice announcement would be another good feature of Retaliation.

      • Chris

        Agree. Maybe random. Some shots can be announced so everyone turns around and looks while others can be stealth. The mix and match may make things interesting!

  • Nick

    Ricardo: I’d be scared of your laser aiming, if only because looking directly into it can damage your retina. Breaking the build shouldn’t be a risk of blindness.

    This missile launcher doesn’t look to be a squishy soft nerf ball. It’s probably relatively dense foam that, in the remote chance of direct contact with someone’s eye as they hear the sound of it firing and look up, could easily cause damage. I hope your company health plan includes generous coverage of workplace-related eye trauma.

  • Nathan

    This is awesome, but the biggest problem is that I’d spend too much time deliberately breaking the build on someone else’s behalf, which would quickly degenerate further to an all out build-breaking war. Maybe it’d be better to have it so that whenever you fix a build, or make a successful check-in, you get to target the missile launcher at any of your co-workers!

  • http://madebynathan.com/ Nathan B
  • Joshuah Kuttenkuler

    A nerf launcher ? This thing needs to be loaded with boots, and only fires if a the build that fails was supposed to be the release build! Last minute changes are not for civilized development!

    You delayed the release date to include the, apparently, broken, foo.h?
    BOOT TO THE HEAD!

  • Pete

    I like this. But where I work, we aren’t in a cube farm. I think this would need to incorporate a Roomba as the platform so it could cruise down the hall to inflict the punishment.

    pk

    • http://www.papercut.com/about/#tom Tom

      That sounds like a great idea! Or maybe one of those RC tanks? The Roomba’s obstacle-avoiding AI would definitely help out with navigation though.

  • Mike

    What you need is an array of launchers to coordinate with each other to attack the offending developer. Macross Nerf Missile Massacre anyone?

  • Chris

    @Infinity and @Mike. Retaliation does have “cluster array support”. If you have a divided office it’s possible to have multiple launches each with their own programmed set of coordinates. All will be notified and only the one with the matching coordinate set will fire! I did the initial cluster server support in our print control software so I thought it was only appropriate that I also implement this in Retaliation :-)

  • http://blog.abmundi.com Fabio Fabbrucci

    Amazing! You are my superheroes!

  • Pingback: MegaSoul · Who broke the build? – PaperCut Blog / News

  • Giulix

    I agree: retaliation for broken builds is not good for the morale. How about launching coke cans and cheeseburgers at the whole team for every SUCCESSFUL build? That would be a fine incentive (don’t forget a helmet for Tom, as coke cans can be pretty work-disruptive at that range)!

    • OM

      Coke and Burger? I thought we said “no punishment”…

  • Pingback: GCU Dancer on the Midway - Link blog: atheism, rationality, latex, integration

  • http://www.hanoulle.be YvesHanoulle

    I also think it is a bad idea.
    for 2 reasons.
    1) yes you give the message that failing is not an option. It might look funny”, but the medium is the message.
    2) I would go even further. Having a failing build is OK. Having a failing build that is NOT quickly fixed is where the problem is.

    The easiest way to not break the build is to not check anything in. Now you have a serious problem. And one that might be harder to see.

    If this is not happening, at least people will take longer before checking in.

    I wrote about almost the opposite idea:
    Offering a croissants after a cruisant build
    http://www.hanoulle.be/2010/06/croissant-build/
    it also has it’s problems, still its more positive.

    n

    • http://www.papercut.com/about/#tom Tom

      Nice idea. In our team, the difference between failed and successful builds is that the foam missiles are a joke. The doughnuts for the team after a successful release are real. And tasty.

      We did have a suggestion for a missile countdown. E.g. the launcher locks on and (audibly?) threatens to shoot unless you fix the build in T minus 5 minutes.

  • Gareth

    I think this is a really bad idea. The whole point of continuous integration is to detect errors, thus it could be said that “the build is there to be broken”.

    I can see that it might be a bit of fun and meant in jest but I’ve worked in companies where there’s something like this starts off as a joke that quickly evolves into a public character assassination and the little shared joke turns into a big joke at the expense of an individual, and a blame culture is slowly being cultivated.

    I absolutely hate the attitude “You broke the build, you didn’t do your job properly. Fix what you broke right now and don’t waste any more of the team’s time.”

    My attitude is “Great, our CI approach is working – the build server is doing its job. We’d better fix what we broke.”

    I would much rather be in a team where success is engendered rather than small mistakes being publicly drawn attention to.

  • Chris

    @YvesHanoulle @Gareth You may a number of vary fair points. I agree that in many environments the culture would not be right for something like Retaliation. It’s also not something that’s appropriate long term. All dev teams need to mix things up a little to keep it interesting, develop a culture of challenge ideas/norms. Doing something a little left-of-center and funny like Retaliation is a good way to foster this.

    If your team is of a flat structure and has a great working relationship then Retaliation may bring a bit of fun for a while. If the environment is “corporate” then Lava Lamps and a weekly management report may be the go!

  • http://www.darkfunction.com Sam

    This is really cool.

    There are also a lot of humourless people posting here

  • Chris

    @Sam +1 :-)

  • efrohnen

    This is whimpy. Lights, sirens and a paintball to that back of the head would be better.

  • 0x4765654b

    We use animatronic toys, dancing Elmos and the worst build toy is a “Potty Time Elmo” which is reserved for checking in bad code and going on vacation.

  • http://www.barcodebeasties.com Richard Green

    I don’t understand how he missed a semicolon …. surely his IDE would have flagged it? And then to still check it in without even running it once ????!! UNFORGIVEABLE …… I would like to have seen all 4 fire if that was the case !

  • http://www.hanoulle.be YvesHanoulle

    @Sam: so from the moment someone like me says there are reasons not to do this, I’m humourless ? That sounds almost like a personal attack. Oh wait that was a joke too…

    Yes I can see the fun of it. And I know from experience that it can backfire.
    Good part is that it seems to have grown out of the team.
    I’m afraid that if this code is used in other teams where it will be used as a joke by SM, PO, team lead or someone else not part of the team that it will backfire.

    It would be nice to see stats of the number of daily checkins before this was used and the now.
    Yes, the best way to have less aims at breaking the build would be check in often. Unfortuntaly most developers think it’s the opposite.

    y

  • justin

    Nice, except that it missed him.

  • Nicky

    How about launching some missles at the managers who didn’t give enough time to the developers to do a good job? Or at the executives of the company that get an obscene salary and bonuses, while the ‘little people’ slowly kill themselves by working overtime.

  • Justin

    My only question is how no one ever thought of this brilliant idea before?

    And some people clearly have their pants on too tight.

  • Brendan

    Some people respond to punishment and some people respond to reward. I am pretty sure it’s been deemed that positive reinforcement is much better than punishment when it comes to productivity. How about making something that gives me an extra quarter times the number of lines changed within the companies source code to the developer’s paycheck? That would get me excited as a developer, especially since some days I commit up to 1000 lines of new code.

    If this were SERIOUSLY in place (i.e. it actually hurt), I’d never want to commit in order to avoid the retaliation. That’s just me, though, because I hate punishment and respond negatively to it, often times continuing the bad behavior to the next degree. It’s no wonder I was in the principal’s office every day as a child because I refused to take the punishment. However, this looks like a funny joke and no one’s getting hurt so I probably would be laughing my ass off every time, especially because I know a certain someone who has a habit of breaking the build, committing, having it still broken, committing again, having it still broken, etc….hehe

    P.S. half the time I break the build it’s because multiple project managers or executives are breathing down my neck for multiple features at the same time and try to commit the code to make them happy and go away. It’s not my fault they got the scheduling wrong. I tell them time and time again you can’t definitively estimate how long it takes to complete a feature.

  • Rich

    @Gareth: I don’t know how your system is set up. But where I work, “breaking the build” and “causing test failures” are two different things. If your continuous integration system doesn’t distinguish between the two, it seems to me that it’s a rather blunt instrument.

    Breaking the build in our environment is either caused by (a) checking in an incomplete change (which is caused by a combination of limited tools and limited attentiveness)… or (b) the developer is letting the integration system catch mistakes that should have been detected by doing a compile and running unit tests before inflicting the code on the rest of the team.

    The first is an occasional “oops” that a good developer will quickly apologize for and rectify.

    The second, however, is simply anti-social behavior. If this is a pattern, I can practically guarantee that it is costing the company in money and/or morale…and the consequence to the repeat offender ought to be more than a Nerf dart.

  • Pingback: Who broke the build? « Hornet Dear Bernard

  • Ron

    Years ago at Lotus, we had the Book Of Shame for those who unfortunately broke the build. A digital picture of the offender was recorded, photoshopped, and added to the book. Your edited picture was also on the splash screen of the next successful build. Good humor and also very successful at reminding people how important it was to make solid commits.

  • Chris

    @Ron That’s a different way of handling the problem…. the “mug shot”. I like it! Humor, if done right is a great way of dealing with these types of situations.

  • http://jamiealexandre.com/ Jamie

    Awesome! I think you can simplify your “jenkins_target_user(user)” function, however. How about something like this:


    def jenkins_target_user(user):
    user = user.lower()
    if user in COMMAND_SETS:
    # We have a command set that targets our user so got for it!
    run_command_set(COMMAND_SETS[user])
    else:
    print "WARNING: No target command set defined for user %s" % user

    or even better:


    def jenkins_target_user(user):
    try:
    run_command_set(COMMAND_SETS[user.lower()])
    except KeyError:
    print "WARNING: No target command set defined for user %s" % user

  • http://www.unsuccessfulartist.com/ sean ruiz

    Yeah, that’s pretty good. If you really want to motivate people you should add a little sewing needle, tipped with depleted uranium, to the end of ze’ missle.

  • Chris

    @Jamie Agree. As long as the COMMAND_SETs only contain all lowercase usernames this would work. A fair ask, or earlier code could force it to lower case. I’ll make the change.

  • Jan

    Hi!

    Did anyone get it to work on Ubuntu? I get an “usb.core.USBError: Resource busy” exception when trying to call “python retaliation.py ‘chris’ “. The driver of the Missile Launcher seems to be USBHID…

    thanks in advance and regards

    • Chris

      @Jan It should work unmodified assuming libusb and modern python is installed. Try running with elevated permissions (i.e. root) to rule that out.

  • Jimbob

    There aren’t enough foam missiles in the world for my workplace…

  • Peter

    Hook it up to a Glock and watch as no one dares make a mistake again ;)

  • Andrew Wolfe

    I would like to suggest you create a ‘reward’ alternative named “Pavlov” to trigger this device when a programmer fixes one or more busted unit tests:

    http://hackaday.com/2011/01/07/internet-enabled-drip-coffee/

  • Pingback: Foam Missile Launcher shoots the Coder who breaks the Build - Nerdcore

  • Hamilton

    Funny :) You could leverage this into your hiring policy also – if you show someone the missiles and they say “blah blah negative reinforcement” then you know that they belong somewhere else much more serious, preferably far far away.

    • Chris

      Agree. With this, mixed in with interview questions about coffee brewing we’re bound to find out who are the really geek programmers :-)

  • Henrique

    Not being able to utilize live ammo doesn’t generate the tension levels necessary in an organization to enforce the following of better development practices and shipping of quality software. The correct utilization of lethal force is the only way to instill fear and discipline, therefore achieving optimal results.

    I suggest extending the API to control the Mk 49 GMLS, which is a proven system in service for the past decade. It supports connection via bluetooth, and also via a proprietary iPhone app once you authorize it with your Facebook account.

    http://en.wikipedia.org/wiki/File:RIM-116_Rolling_Airframe_Missile_Launcher_3.jpg

    (PS: Awesome idea BTW )

  • Michael

    Hi, great idea.
    Do you know whether the binary protocol is the same as that used by the Storm “O.I.C” dart launcher? (The model with an integrated webcam.)

    I am planning on integrating your CI dart launching code into the build system for my current company. Since we have a couple of offices, the webcam functionality might be useful.

  • http://www.hanoulle.be YvesHanoulle

    So the system is now in place for at least 3 months.
    As I said in an earlier post, it would be nice to look at the statistics to see how this system has an effect on the number of bad checkins. And how this has an effect on the nr of checkins. (Do people checkin less often or not?)

    • Chris

      No really notable change. Other things (like sprints/releases) have a larger affect. Only one developer has been caught deliberately breaking the build to liven up the day :-)

  • Pingback: Fanatic » Blog Archive » After the ci traffic lights and lava lamps….

  • Pingback: html5shiv and Serving Content From Code Repositories | Zoompf

  • http://dont-ask-such-questions.com K

    Funny, but at a deeper level – sad. Just another way to get the slaves to work harder. :)
    I was just thinking about applying for a leading job here. But after seeing this… I am off.
    Good luck with the builds!

    • Chris

      You’re correct. It’s all a joke and in good fun. Our next project is to make customer submitted automatic crash reports send a 240v shock to each member of the dev team ;-)