Cruisecontrol is an example of a continuous integration tool. In theory, these are amazingly useful development tools. The idea is to have a tool with the sole job of getting the newest code, compile it, test it, and give some results. This way whenever a change is made, within a short amount of time, we know if it broke something else, even if it wasn't intended to. This is great for those Windows developers stomping all over the Linux guys, or the Asapi guys breaking the UI, again.
The way Cruisecontrol is currently shipped, it is ideal for one machine, one project, one repository. Unfortunately when you have 10 different targets you want to run, differentiate between build failed and tests failed, compile on different machines, architectures, and operating systems... etc. This is where Cruisecontrol starts to slow down. It's still top notch, and modifying it is a cinch, but all these modifications can add up.
So far Cruisecontrol here consists of 4 machines, 2 Windows build machines, 1 Linux build machine, and a Windows web server. The build machines are all accessing an existing Perforce repository, and they FTP the necessary results to the web server for display. So far, this is out of the box behavior. Cruisecontrol can automatically check Perforce for changes, sync the local client, and there are build in FTP classes for sending logs and build status files.
Trying to get Cruisecontrol to play nice with existing build and test products was the first challenge. When running the build system from the command line, alot of output is generated, and almost none of the useful stuff is distinguishable. Cruisecontrol does the nice thing and parses all of the warnings and errors out of the logs for you. The build scripts used had a lot of unnecessary warning level messages, so the first step was to change those to info level messages. Next, the web interface was just poor looking. I mean, it's great to have it out of the box, but everything just ran together. So with a little added style and a few changes to the xslts, the errors and warnings all stood out by target, file, everything.
Next were the tests. Needless to say, of course we don't use JUnit here, the default test suite that Cruisecontrol supports. So the test results aren't going to match up. Luckily there was another xslt that transformed our results into JUnit style xml. Not all of the data was Cruisecontrol expects was present, and Cruisecontrol obviously had a bunch of contingency cases built in when they designed their xslts to get the test results web ready. Since our company uses our results, I scrapped the Cruisecontrol xslts and wrote a simple transform that does the trick. If anything new comes up, which I doubt, I'll worry about that then.
Now we need to deal with our 45+ minute build times. It would be nice to know what part of the build we are on, just as a general estimate on how much further we need to go. It's also nice to know if maybe the build crashed and needs to be restarted. There is a very handy plugin already provided that overrides the default ant logger. All it does is override the "target started" event to append the target name to the build status file. This way whenever a target is started, we can see the time and name of that target on the project web screen. Well, we could if this was uploaded to the web server.
Like I mentioned, our configuration is unique in its needs with separate build machines from web servers. The remote web server can't (yet) use the build machine's JMX console to manage the builds. It also can't read the minds of the build machines to know of there has been any updates. The build machines have to send everything to the web server in order for it to post the results.
Cruisecontrol was nice because it offered FTP support for the built in reporting functionality, but this doesn't automatically extend to the upgraded logger mentioned above. It's also another pickle because technically the FTP classes extend to be Cruisecontrol plugins, but the logger is an Ant plugin. This is where Frankenstein becomes my good friend. I took a direct copy of the logger extended class and called it an FTP class. There already existed an "AbstractFTPClass" in Cruisecontrol. Only problem is my one class couldn't extend from the logger class AND the ftp class, too bad. So I ended up having to write a GenericFTPClass that fakes all of the parts that are needed. Put a generic into my new logger class, and we're golden. I'm downplaying all that of course, but you get the idea.
Next is to possibly add some features to the new FTP class to slow down the transfers a bit. On every new target, there is an FTP send. I can imagine this traffic skyrocketing with multiple builds, etc. So maybe adding an "updatenum" or "updatetime" that will specify a number of updates or seconds before FTPing. The next thing is to also get this better incorporated into the web interface. Our typical build has well over 300 targets before it's done, more if running tests. Having all of these displayed at once is WAY too vertical.
Lastly, there's the cool things that other houses do with CITs. Whenever a build breaks here, (almost) everyone gets an e-mail if they are on the changelist for that broken build. If you're unfortunate enough to have submitted something after it broke, you're getting an e-mail too. Soon everyone, every change, until a change fixes the build. I've seen online different lights that can be triggered, alarms, etc. Lava lamps even that not only tell you the status of the build, but how long it's been there. We've even talked about hooking up a text reader that will periodically announce the names of the people on the changelist and politely remind them to fix the build.
Hopefully this little (really?) blurb helps everyone understand how useful these continuous integration tools (cit) are.
