Articles and White Papers

A Quick Primer on Agility and Scrum

By Dr. Martin Schedlbauer, CBAP, OCUP

Too commonly, projects are started with the best of intentions, plenty of process structure, lots of templates and required documentation, and yet they still fail. Surveys of our clients have shown that lack of user participation in requirements gathering, too little time devoted to really understanding the needs of the business, unrealistic expectations, inadequate feedback from actual users, and poor project management and estimation are among the root causes of challenged projects.

Moreover, there is mounting evidence that waterfall-style “big-bang” approaches are inefficient and prone to failure. Agile methods are an alternative to sequential solution development. Agile methods are based on:

• Just-in-time development
• Lean requirements analysis
• Business value generation
• Product orientation

Download Catalog

Why is Agility Desirable?

While rigorous processes are sometimes necessary, they generally introduce significant inefficiencies:

• Large amounts of documentation that is often unnecessary and frequently out-of-date
• Increased process rigor to deal with risks of distributed teams and unavailable stakeholders
• Inflexible processes when requirements invariably change
• Long delays before a usable product is available for review by stakeholders

Traditional Vs. Iterative

The traditional approach to project management is to move through the various phases of a project sequentially. Progression through the different stages is structured, which means that the next phase is not started until the prior phase is complete and has been validated. For example, design is not started until all of the analysis has been done. This approach is commonly called the “waterfall approach” because one only goes forward, but does not revisit a prior phase. While this sounds like the right thing to do in theory, in practice this presents a problem since requirements are often not fully understood until later in the development cycle. There invariably are changes to requirements and the waterfall process is not suited to projects where the requirements are not well understood up front.

Figure 1. Classic Waterfall Approach to System Development

In contrast to the waterfall model, the iterative approach defines a solution in stages or iterations. It is an incremental strategy where each iteration produces a more refined product. The development team continuously plans, design, codes, tests, and so forth. Each iteration lasts a fixed amount of time and produces something of value at the end — ideally a demonstrable product.

Agile methods rely on iterative project management and do not assume that all requirements must be defined before development begins. Instead, they allow new requirements to be discovered and addressed as they arise. Therefore, new requirements are added to the project “backlog” and are then addressed during an upcoming iteration when the stakeholders decide that the new requirements is more important than all of the other requirements discovered previously.

Figure 2. Agile Approach to System Development

Benefits of Agile Methods

The agile approach mitigates risk by addressing high priority requirements early in the development process and providing customers with the opportunity to provide frequent feedback by producing tangible results very early in the project.

Applicability of Agile Methods

Ideally, agile methods provide the best results when the development team is fairly small and generally has no more than 8-10 team members. Larger projects are often divided into multiple agile teams. In addition, development team are most productive when they are co-located, which means that they work together very closely, ideally in the same room or general area to maximize collaboration and communication.Generally, agile approaches do not work well when teams are distributed, large, or have team members who are novices.

Agile Methods in Use

Scrum and Scrum/XP hybrids are the most commonly used agile project management methodologies.

VersionOne: Agile Development: A Manager's Roadmap to Success, 2008.

Scrum

Scrum, a particular and very commonly applied agile method, follows a specific framework. Scrum was developed by Jeff Sutherland and Ken Schwaber in the mid 1990’s for rapid client/server and web application development.

Figure 3. Scrum Methodology Overview

There are some key Scrum terms, so here’s a Scrum terminology primer:

• Iterations are called sprints are generally last generally 30 days, although smaller sprint lengths are not unusual on shorter projects
• Scrum is a rugby term
• Scrum master takes the role of team manager; there generally is no project manager in the traditional sense
• Product owner is the champion for the project representing the needs of the stakeholders
• Other terms, such as retrospectives, reviews, and backlog will be explained later

Scrum team members are dedicated to the team, but may be part time. The team works together to achieve the sprint goals. There are no distinct roles—everyone helps out to achieve the goals. Volunteerism and self organization are key principles to instill ownership and commitment.

Key Practice: Collaboration

Agile teams value collaboration over needless documentation. Daily communication amongst team members is critical (called the daily scrum in Scrum terminology). The product owner and domain experts are consulted when needed during an iteration (sprint in Scrum terminology). Teams often work in an open area to foster collaboration and communication. Team members help each other; overloaded team members transfer work to underutilized members.

Key Practice: Daily Scrum

The daily scrum is a 15-minute (time-boxed), stand-up meeting scheduled early in the day and attended by all team members. During this meeting, the team shares progress made during the last 24 hours and raises any impediments to success. Each team member then explains what they plan to do today and any obstacles that stand in their way. Commitments are made to peers not the scrum master.

The meeting moves at a brisk pace; issues are not resolved, only raised. Observation by guests is encouraged so that they can see what the team is doing. This establishes trust between the business and the project team.

Key Practice: Sprint Review

The spring review is a four-hour meeting at the end of each sprint in which the team demonstrates what it has accomplished. Demonstration should be live and show working software, not slides. To discourage slide demos, the preparation for the sprint review should be limited to a couple of hours so that the team focuses on the essentials.

The review meeting is facilitated by the team, not the scrum master. This will put pressure on the team during the sprint to perform and be quality focused; after all, they will have to stand in front of the product owner and the other stakeholders and demonstrate what they’ve done over the past few weeks.

Everyone is encouraged to attend and provide feedback including bugs, new features, enhancements, etc. Feedback is added to the product backlog as new user stories.

Key Practice: Sprint Retrospective

At the end of the sprint review, before planning the next sprint, the team meets to review its own processes and methods. The sprint retrospective is inward looking. It is a 30-minute brainstorming session in which the team determines:

• What we should continue to do
• What we should stop doing

Managers and observers may not attend. It is reflective, looks inward and is intended to be honest so that process improvements result.

Key Practice: Frequent Testing

To assure a continuous working product, testing is done during every sprint. To do this efficiently, automated regression testing tools are essential. Each team member first tests their own code and then the QA role tests the product before the sprint review. Continuous integration is emphasized and acceptance tests should be written before coding starts. This forces a full and accurate definition of the exact product requirements.

Key Practice: Time-boxing

Each iteration (sprint) is time boxed, which means it is held to a particular length of time. The typical length of an iteration is 15 to 30 calendar days and the iteration length remains the same throughout the project. The length can be extended on large projects, but the risk of building the wrong product may increase as reviews are done less frequently.

No extensions are allowed, but reduced scope is accepted. A fixed iteration length allows us to establish the actual working capacity of the team. With a fixed iteration length, there is a tendency for project managers to demand that everything that was agreed upon for the iteration be completed no matter how much effort it takes; working nights and weekends is often required—which is not good! The team must work at a sustainable pace; otherwise you will never know the exact capacity of the team other than its peak capacity. And that pace is impossible to sustain through the entire project unless you want team members to burn out and eventually leave.

All tasks must be time boxed—sprint reviews, daily scrums, design meetings, sprint retrospectives. A scrum team sets a deadline for everything it does. This practice focuses the team and forces them to forgo less important work.

User Stories as Requirements

User stories define requirements from a user’s perspective and are deliberately written without much detail. Details are added during conversations with the user, product owner, and subject matter experts once the user story has been selected for implementation during an iteration.

Example user story:

As a cashier, I want to be able to enter a customer’s order so that I can track and review it.¹

Managing User Stories

User stories are often tracked using simple methods such as hand-written index cards, but software tracking systems are also available. User stories should be visible to all team members and stakeholders; it is common to pin them up on the wall in the team’s war room.

User stories are found during initial story writing workshops as well as during development and during end-of-iteration reviews. In fact, new stories can be added to the “backlog” at any time.

Story Card Layout

User stories are commonly written on index cards. The story statement is on the front along with relevant details and the story’s “size”. The back of the card lists the minimum done criteria whose implementation would be considered a successful implementation of the story by the product owner.

User stories intentionally do not capture a lot of details because a story may never get implemented if it is not of high enough priority, and the story details may change by the time implementation starts.

Since we don't capture much up front, this assumes that business users are engaged during implementation since details are uncovered in conversations.

Figure 4. User Story Cards

Story Sizing Guidelines

Smaller stories with a clearly defined goal are better. Large stories that are high level and vague are called epics. Epics should be decomposed into several smaller stories. Ideally, a story should be implementable in a single iteration. Stories should be written at a high level; details should be omitted until implementation starts.

Anyone can contribute a story:

• Product owner
• Users, analysts, and SMEs
• Developers and testers

Epics: Large Stories

Large user stories are called epics. They contain more than one idea or feature and commonly come from managers. Users tend to give “click-level” requirements and features. Epics need to be split into smaller and more tractable user stories so that they lend themselves to being implemented in a single iteration.

Themes: Related Stories

A theme is a collection of user stories related by a common subject area. Iterations often address stories from one theme as they tend to exhibit more interdependencies and also rely on the same infrastructure code.

Story Writing Workshops

Story writing workshops are a common approach for finding the initial user stories. Additional stories will be discovered by the developers and business analysts during iteration implementation as well as by stakeholders and the product owners during iteration reviews. New stories can be contributed by anyone at any time. New stories are simply added to the backlog of stories and will be addressed in a subsequent sprint if the story’s priority is considered high enough. Stories are captured but not discussed in detail yet.

Participants in a story writing workshop include anyone committed to the project and must minimally include the product owner, users, developers, business analyst, and testers.

Story Writing Practices

Here are some suggestions for finding and writing good user stories:

• Identify different user roles
• Brainstorm user stories for each identified user role
• Build a low fidelity throw-away prototype to help users visualize what they are looking for
• Write a story for each needed feature; break epics up if possible but do not spend too much time on this activity yet

Role of the Product Owner

The product owner is the key stakeholder — he or she knows what the product must deliver. This role should be fulfilled by someone from the business, but could also be a business analyst as a product owner proxy. The product owner must be present to answer questions from the development team during the project. If a product owner or a SME is not available, the business analyst is expected to bridge the gap. This may require additional documentation to reduce the risk of the project because of the frequent unavailability of the product owner.

Change Management

Any new or changed requirements are simply added to the product backlog at any time. The iteration (sprint) goals however remain untouched. The team manager (scrum master) protects the team from changing sprint goals, but the organization can abort a sprint at any time to refocus the team on a new, higher priority set of goals. The product backlog is intentionally dynamic: there are no frozen requirements.

Product Backlog

The product backlog is a collection of estimated user stories that have not yet been implemented. The backlog also contains constraints and bugs. The total effort for the project is the sum of all story points. Story points are assigned by the implementation team, and are a measure of the story’s overall complexity and required level of effort. They are generally not person hours, but rather a number between 1 and 10, where 1 is an easy to implement story while 10 is a very difficult story requiring lots of work. Note that agile teams do not attempt to estimate the effort required for a story up front. Instead the project schedule is derived through observation.

Schedule Estimation

Scheduling in agile projects is empirical, i.e., it is based on actual measurements rather than models or guesses. A common method for deriving schedules is the following:

• Observe the actual number of story points implemented during a sprint: this is the teams rate of productivity or velocity
• Divide total number of unimplemented story points by the velocity to get number of sprints needed to implement the entire backlog
• Multiply number of sprints by the sprint length to get the number of calendar days needed to complete the project’s scope

How Does Velocity Change?

Velocity decreases when:

• People are removed from the team
• People have external assignments
• Sprints are frequently aborted
• Product owner is unavailable
• Scrum master does not remove obstacles

Velocity increases when:

• Infrastructure has been substantially built out
• Team becomes more familiar with domain
• Team has competent members

Agile Success Criteria

Agile methods are successful when the right conditions are in place. Open communication and transparency are two of the most critical success criteria. For example, anyone can come to daily scrums and sprint review sessions. The backlog is openly viewable.

It is also very important that there is a certain level of commitment by the organization to let the team work and not fear “punishment” when an iteration had to reduce scope because the team committed to too many features.

Summary

For software projects to be successful, they must value transparency and frequent communication. Agile methods such as Scrum emphasize lean requirement management, but are highly structured. Scrum emphasizes co-location, strong collaboration, dedicated teams, transparency, frequent reviews, and engaged stakeholders. Documentation is only created when it helps decrease the risk of the project and the risk of miscommunication; otherwise, verbal communication is preferred over written communication. The focus is on building business value and solutions that address actual stakeholder concerns.

¹ This is the format originally suggested by Mike Cohn.