As teams and code bases grow what are some of the common problems encountered with build automation? How do you overcome these problems to keep CI and CD pipelines flowing?
Mark Shapiro, Segue’s Senior Architect for Application Development, joined an online panel on the subject as a part of Continuous Discussions (#c9d9). This community initiative, sponsored by Electric Cloud, powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test, and deployment processes. Continuous Discussions is a series of community panels about Agile, Continuous Delivery, and Devops.
Below are a few soundbites from Mark’s contribution to the panel.
What Do Build Bottlenecks Mean For Your Pipeline?
“Most of the bottlenecks that we encounter tend to be where we can’t automate and have to have some kind of manual intervention. Manuel intervention means you have to have the person available to actually do the thing you need done, and do it correctly. A lot of times it’s a scheduling issue. We’re a smaller company so sometimes we’ll have particular resources shared across a number of projects and so if I create a build and pass it onto QC (quality control) for manual testing and QC is not available at the time because they are working on something else- that’s a bottleneck and that prevents me from getting that software out to the customer, the end user, and wherever else it needs to go. Automation really is the key to eliminating those bottlenecks because the less you have to rely on a person to do something, the less the likelihood of there being a bottleneck. Obviously there are still physical resource bottlenecks. For example, if you only have one build server and can only be running one build at a time and you have got a lot of builds going, you’re going to have a pretty big queue of builds waiting to go. If you’re a smaller company and you’re only doing one or two systems at a time or building one or two at a time, that’s not as much as an issue.
We don’t usually run into situations where we have multiple long builds that we are trying to run at the same time. We have a couple of different build systems depending on what platform we’re building for, so that wouldn’t happen unless we were building two very large systems for the same platform at the same time. Even then our systems can handle multiple concurring builds-it just slows things down a little bit- but there we’re talking minutes, not days.
Sometimes build automation isn’t about getting things out to the customer faster; sometimes it’s just about getting it out to the customer better. Knowing what’s in a build and knowing that you click this button and all four of those components are going to be built, sometimes that is worth more.“
What Are Some of the Common Problems with Build Automation?
Third Party Dependencies
“Third party dependencies can sometimes cause problems with build automation. I think that it’s amusing that we build systems with third party dependencies and in the old days we would download the libraries and include them in version control or put them on a server or something and now we have new, centralized repositories for these libraries. We have NuGet, we have Maven repositories, and countless other systems. Using those systems introduces another third party dependency and if you’re using one of these centralized internet repositories and your connection goes down or that repository goes down, now your build process is broken and you’re actually in a much, much more difficult position when it comes to fixing it because now you have to completely change out this dependency for some other one. However, they certainly do simplify the third party dependency management when it comes to the build as long as you’ve got a good connection and the repository is actually available.
There are also runtime third party dependencies. We don’t run into problems with those now as much as we used to. In the old days, when older third party libraries included in the build tended to be more complicated to include, in some cases we had to actually install a program on the build server that had those libraries available. Those days are long gone and we don’t have to deal with that anymore. All of the library vendors have more or less come on board and have made using their libraries easy by supporting independent library repositories. Even if it’s a commercial library you can still download the library for build purposes. In terms of integrating with third party systems at run time, that’s really not a build issue- that’s much further down in the pipeline- but there you have the whole gamut of issues.
Right now, one of our systems, we’re waiting on our customer’s vendor of another system to put a patch in place so that we can build a capability that the customer wants us to build but it depends on that other system. So we’re hamstrung when it comes to moving forward with some things because of that.”
Informal and Formal Code Reviews
“If code reviews are a prerequisite for getting code checked in and included in a build, they’re a bottleneck in your build pipeline. If instead they are a thing that happens in parallel to the build pipeline, where the purpose is not ensure that you’ve written it correctly or written the correct code, but simply to add a second check on quality- and if there is a problem with the code written then you address that as a change in another build cycle- then they are not so much of a bottleneck.“
Source Code and Build Version Management
“These two, in my opinion, tie very closely together. Part of build version management is knowing what code went into the build so source code control, version control, and getting the code from version control in order to do the build- whether it’s the subversion tag or something in git or whatever version control system you’re using- if you can tie that code back to the build, then your build version management gets much easier. Of course getting the code for the build, at least for any kind of a formal build that’s going to move through your pipeline, getting that from version control is absolutely critical. Otherwise, you have no repeatability and really no idea what code what went into it. If your developers are kicking off the official builds from their development machine, from the code that they have checked out, you might have code that gets into a build that goes into production and never gets into version control. Then the next time you do a build now you’ve broken something that was previously working in production.”
Need Help? Contact us