There are a number of challenges non-Agile organizations face when attempting to adopt Agile development practices and the Scrum methodology. In this article we will discuss three of the more common ones.
Problem 1: Scrum fails to get traction or is a distraction from the real work of the project.
In order to be effective, a Scrum Master and as many team members as possible must have start-to-finish experience with team projects of enough duration to have had scheduling delays, non-project distractions, and requirements drift, among other things. Six months is usually long enough to have experienced at least some of these issues, but a longer duration has a way of compounding them and challenging the team even more. This experience has a way of making the value and purpose of agile practices vividly clear. It helps to have worked on a number of waterfall life-cycle projects and to have been frustrated by the impedance mismatch between how they were managed and how developers actually work.
Without that experience, at least for the Scrum Master, CSM training will not have sufficient relevance, and the Scrum Master will probably not be able to guide the team through the many day-to-day decisions that will have to be made. Scrum and agile are practice frameworks, and the unique details of each project must be considered carefully. Experience is keenly important here.
This may well be the source of complaints from experienced developers that Scrum and agile are ineffective. If your project involves a team consisting of subject matter experts, product owner, developers, and project manager, then Scrum provides the avenues of collaboration sorely needed for project success, and should be seriously considered for the benefit of the whole team and overall project goals.
Problem 2: Developers accustomed to working autonomously may find that Scrum is unnecessary and slows them down.
There is no question that Scrum adds some overhead to the development process, as compared to a development process with no formal methodology. By design, Scrum is a management tool for agile projects; intended to give management a meaningful view of the health of the project, and the ability to make management decisions about how to proceed. This, unfortunately, entails some amount of overhead. But, arguably, Scrum, done right, provides more realistic information about the project than traditional tools, helps the team self-manage, and incurs less overhead than traditional tools would incur.
Some projects are better suited for a smaller number of developers working autonomously. Personal Kanban might be a more useful project management tool for these projects. However, when you need to scale up to a team of developers and product owner(s), you need to emphasize collaboration among the team members, and Scrum is an excellent solution for that situation.
Whether to use a collaborative approach like Scrum, or a more individual-based approach, should be based on the nature of the project.
On one hand, if you are building a new product in a new product space with no user base against which to measure it, you might consider a less collaborative approach to development, at least early in the project, to give the inventors a broad range of freedom. If you have a few visionaries, they may be all you need to steer the project.(You might also consider a Lean marketing approach to learn what might win the market).
On the other hand, if you are developing a project based at least in part on any existing solution, especially if subject matter experts already exist, then a collaborative approach like Scrum is the right way to go. You may also want to consider Scrum or another agile approach if the number of people needing to communicate regularly exceeds 3 or 4.
Problem 3: Some development efforts don’t easily fit into a time-boxed sprint. Therefore, Scrum doesn’t work for me.
This is a real problem. Several kinds of development resist being meaningfully squeezed into standard size sprints. Here’s a partial list:
- New system architecture
- New complex user interface design
- Database ETL requiring extract, cleanse, transform, stage, and present data
Some of these may take several tries to get something that even works, let alone the best solution. They all have trouble conforming to a sprint-sized effort.
If time-boxed sprints are one of the best ideas of agile, why does it appear that I am suggesting that we need to make exceptions sometimes? The goal of a sprint is to ensure that all of the sprint’s backlog items are completed, tested, and working, and that the sprint delivers its designed functionality, however small, to the end user within the fixed duration allocated to each sprint. Delivering a sprint on time requires good planning, good discipline to stay focused, and good teamwork. One of the problems is with the phrase “to the end user”. If the end user is defined as one of the consumers of the application, there are some development tasks that normally take longer than a single sprint. But wait! There are things you can do to make them work within an agile framework. Let us start with 3 problem areas:
New System Architecture often involves many different hardware components, many existing software applications, and different layers of an organization’s IT and administrative staff. Hardware must be purchased, installed, and made to work. Third-party and in-house applications must be made available for access by the to-be-built application. Security must be implemented according to the organization’s current infrastructure. Permissions must be planned and granted for the development team and at least a small SME test team. All of this crosses organizational org-chart boundaries and requires administrative approval. Delays can be caused by hardware not being ordered or not arriving when expected, IT and administrative delays, low support staffing, and many other things not directly under the control of project planners. While virtualization and cloud resources help lessen the burden of setting up the initial operating and development environments, it is easy to see how implementing new system architecture is hard to predict and is dependent on many external factors which do not lend to fitting it into sprints.
Another example is Complex UI Design, which can take many tries to get right, involving both development team and SMEs, and often requires many trials and many errors, involving the creation of a large number of style sheets, mockups, wireframes, graphics, and other design assets. All of this effort does not lend itself to right-sizing for sprints.
The third common problem area is Database ETL, which often requires many layers of work, including extract, cleanse, one or many data transformations, staging, and finally presentation of the data according to requirements. Presentation is the first time that the end user actually sees the output of this work. Again, a big bite for a single sprint.
Each of these scenarios presents a challenge to fitting into a standard sprint. However, there are several ways that even the largest of tasks can be broken down into sprint-sized chunks:
- Loosening the definition of “end user”: Open the definition of end user to mean something other than a person using the application. The end user could be the next hardware layer (even if no human sees the interface). It could be the business layer interface which is one step away from the presentation layer that the end user sees. It could be the cleaned extracted data ready for transformation, something a real end user never sees. By breaking large tasks down into layers, you can split those layers between sprints for more manageable delivery.
- Narrowing the “river” of the sprint: Instead of delivering the entire environment right away, focus on delivering the pieces of the environment that will be needed earliest in the development cycle: if a server (virtual or otherwise) will be supporting capabilities you will not be developing for several months, provisioning that server can be tackled in a later sprint. Instead of tackling ETL for multiple data sources from disparate organizations, focus on one data source at a time. If a data source is sufficiently large, one sprint might focus on extraction & cleansing, the next sprint on transformation, and a third sprint on loading. For additional database-relevant ideas, see Agile Data Warehousing Project Management by Ralph Hughes (Morgan Kaufmann, 2013).
- Using the idea of a special “Sprint 0”: Sprint 0, which may be shorter or longer than a standard sprint, focuses on all of the tasks necessary to launch a project before any “productive” work can be done. Carefully allowing a sprint to take longer than the normal time-boxed iteration is acceptable as long as it does not become a habit. Allowing some slack when there is inherent uncertainty is a good thing.
Allow us to underscore the value of using the time-boxed sprint:
Parkinson’s Law describes the phenomenon that “Work expands so as to fill the time available for its completion.” Often working on a very specific solution, developers will suddenly come up with a much more flexible generalized solution that would take a bit longer but would be so much more capable. Sometimes, however, the more generalized solution is just not needed. When you are working under a finite time-boxed constraint, you tend to avoid the not strictly necessary embellishments. The discipline of a time constraint can be an amazingly effective productivity tool.
Consider also that breaking larger efforts into smaller time chunks has been proven to be much more likely to succeed. You are less likely to get waylaid, you are more likely to stay focused, and the frequent sanity check keeps you closer to your chosen track. These are among the reasons that agile works.
If you find yourself resisting the adoption of an agile approach, consider whether the issue is primarily a problem with agile itself, or the execution of agile that you are experiencing. Here, I have addressed the top 3 complaints that we hear. Like anything new, agile methods and Scrum will cause their share of growing pains. But the track records of Agile and Scrum are good, and these approaches will almost always reward the teams that embrace them.
For additional ideas in an area relatively new to agile – database development– see Agile Data Warehousing Project Management by Ralph Hughes (Morgan Kaufmann, 2013).