Introduction

Modern application development is rarely a solitary effort. Designing, coding, testing, and packaging applications requires a cross-functional team involving designers, developers, testers and graphic artists working collaboratively and each owning tasks dependent on other members of the team. This is made more difficult by the fact that requirements are more often than not unknown or incomplete. Combined these make the process of developing applications that meet user expectations, are stable, and are delivered within time and budget constraints a highly complex enterprise.

Since the early 1970’s various methodologies were created to overcome these difficulties and over time the most popular has been the waterfall methodology. This methodology proceeds through the following phases in sequence to create a deliverable application.

• Requirements definition
• Design
• Development
• Quality assurance
• Maintenance

The problem with waterfall is that each phase must be completed fully before progressing to the next phase. However, the design and development process is not sequential. New requirements can surface at any point necessitating changes to design, which in turn results in new development activities, testing, and documentation. Arguably, the issue with waterfall is it is designed to resist change once requirements have been set. Most waterfall methodologies require detailed work orders and multi-level approvals before any change can be implemented.

Since the 1980’s the size and complexity of applications have increased the impact that resistance to change has on a project. On one hand, this can result in applications which don’t fully meet user needs. On the other, it can adversely impact the stability and quality of the application. In either case, the result is applications that don’t contain features and functionality necessary to accomplish their intended goal, as well as applications that are severely flawed due to quality issues.

A New Approach

As application size and complexity grew during the 1980’s and 1990’s many thought leaders began experimenting with development methodologies in an attempt to overcome the issues experienced with waterfall methodologies. Despite different approaches, these were lightweight and embraced, rather than resisted, change. A few of the “new” methodologies are:

• Agile Unified Process
• Crystal Clear
• Extreme Programming
• Kanban
• Scrum

Collectively, these are commonly referred to as Agile methodologies and are to varying degrees reflect the principles defined by the Agile Manifesto:

1. Customer satisfaction by early and continuous delivery of valuable software
2. Welcome changing requirements, even in late development
3. Working software is delivered frequently (weeks rather than months)
4. Close, daily cooperation between business people and developers
5. Projects are built around motivated individuals, who should be trusted
6. Face-to-face conversation is the best form of communication (co-location)
7. Working software is the principal measure of progress
8. Sustainable development, able to maintain a constant pace
9. Continuous attention to technical excellence and good design
10. Simplicity — the art of maximizing the amount of work not done — is essential
11. Best architectures, requirements, and designs emerge from self-organizing teams
12. Regularly, the team reflects on how to become more effective, and adjusts accordingly

The main difference between waterfall-style project management and agile projects is waterfall fixes the scope of a project to improve the accuracy of resource and time estimates, while agile estimates scope and fixes resources and time. This reversal is necessary since is it is not possible to identify all requirements at the beginning of a project with anything approaching 100% accuracy; any attempt to do so will lead to issues in stability and quality. Agile methodologies address this by adopting an iterative approach to development with each iteration becoming a foundation for its successors.

Figure 1 show the relationship between the traditional “iron triangle” of waterfall and that of agile.

Figure 1 — “Iron Triangle” — Waterfall vs. Agile

What is Scrum?

Scrum is based on the principles of transparency to all stakeholders along with continuous inspection and adaptation to changing conditions. These result in a methodology embracing change and promoting an environment where all members of the project team share an equal voice regarding how the application will deliver value to its users.

Figure 2 shows the roles, information, and processes that make up the Scrum framework.

Figure 2 — The Scrum Framewok

Scrum includes the same types of activities as the waterfall methodology, but rather than implementing them as sequential phases, they are encapsulated into multiple iterations, or sprints to create a working application. In this way, Scrum builds the application incrementally, with each increment adding and improving features and functionality created by its predecessors.

Another integral component of Scrum is the different types of feedback loops used to ensure that,

• Every member of the team understands their role and is aware of issues and tasks across the team.
• Issues are quickly identified and corrected.
• End users and customers see the team’s results quickly and provide timely feedback
• End users and customers refine existing requirements and provide new ones as the project progresses.
• Sponsors see the team’s progress as it takes place to eliminate wasteful activities such as formal and often repetitive status reports that detract from the time spent on actually developing a product.

Pieces & Parts

Using Scrum requires people specific Scrum-roles, information describing the features and functions of the application, information needed to ensure that the workings of the project is transparent to everyone on the team, and processes that govern the operation of the project team.

People & Roles

There are three types of roles internal to a Scrum team and one that’s external. Internal roles are:

Product Owner

The Product Owner is the primary interface between the team and the Business Owner, and is responsible for maintaining and prioritizing the backlog of tasks to be performed. This is performed transparently and collaboratively with the Development team. The Product Owner is also the arbiter of questions concerning backlog items and to make sure that the team understands the purpose and value the application has to to the customer. The Product Owner is also keeps the backlog current as new requirements surface and existing requirements are refined.

Scrum Master

The primary goal of the Scrum Master is keep the Development team productive. The Scrum Master must lead by example and act as a servant-leader for the team, coaching them to make sure they understand and follow the Scrum process and that they operate in a collaborative and professional manner.

From the perspective of the Product Owner the Scrum Master represents the Development team ensures that their issues, concerns, and roadblocks are taken into consideration by the Product Owner.

Development Team

The Development Team is responsible for delivering the product. The team is typically cross-functional in terms of expertise and experience and the team as a whole is accountable for the delivery and quality of the application.

Development teams are also self-organizing. This means that the team, not the Product Owner or Scrum Master, are responsible for defining rules dictacting how they work together. The Product Owner is responsible for defining the backlog of what needs to be accomplished, but the Development Team is responsible for defining how it is completed.

There are no titles or sub-teams within the Development team. Members of the team may have different skills, specializations, and experience, but they all have a voice in the operation of the project and all contribute to the creation of the application.

The Scrum Master and the Development Team are collectively referred to as the Scrum Team.

The external role is that of the Business Owner. Although this is sometimes omitted in descriptions of the Scrum framework, but it is nonetheless an important one. The Customer is the group or organization who is sponsoring the project and who will ultimately use it. The Business Owner represents the customer and is the source for application requirements.

Information

Product Backlog

The main source of information about the project is the Product Backlog, which defines requirements the application must meet in order to be successful. Requirements are expressed as user stories of the format:

“As a: <role> I want to: <function-description> So I can: <value-statement>”

User stories (or just “stories”) are short statements identifying the type of person, the functionality they need to perform in their own terms, and the value they expect to achieve from it. These are intentionally expressed in very personal terms and are kept to what would fit on an index card. Keeping stories short is important because it ensures the requirements are fine-grained.

Sagas, Epics, & User Stories

It is often helpful to categorize stories as sagas, epics, and stories to reflect differences in their scope and lifespan.

Sagas

Sagas are “big picture” requirements that are extremely broad in their scope. An example of a saga might be:

“As a: end-user I want to: use an application that has exceptional performance and responds to my requests within a matter of seconds So I can: quickly complete my work and stay focused on my immediate task.”

This type of requirement is very broad in its scope and sets an expectation for all functions in the application. As such, every feature and unit of functionality must take it into consideration. This means that multiple epics and stories will be based on this type of saga. Sagas typically have a long life and while they must be adhered to they are not fully completed until all lower level epics and stories rooted in them are finished.

Epics

The next type of story is an epic. Like a saga, epics have a lifespan of multiple sprints and cannot be considered complete until all stories based on them are also complete. However, epics have a narrower focus than a saga and although still broad in nature, they can be completed within a few sprints. For example,

“As a: content-contributor I want to: categorize the content I create So I can: ensure that readers can easily locate it”.

This type of requirement will result in multiple stories defining how content categories are defined and maintained, how they are associated with content, and how search functions utilize them.

Stories

The user story is the most granular description of functionality. Stories describe deliverables that can be completed in a single sprint. An example is:

“As a: experienced-end-user I want to: Save my work using a command-key sequence So I can: Quickly save my work without multiple clicks”

It’s important to note that there are many details the end user may not be aware of or even care about when it comes to specifying a requirement. It’s up to the Product Owner and Development team to refine stories so they are actionable. In many cases this means that the sagas, epics, and stories provided by the customer must be decomposed into more discrete and therefore actionable units of work.

There are a number of tools that can help with the capture and management of stories including Jira and Trello. However, stories can just as easily be maintained using a manual method such as index cards.

Metrics

Does the product meet the needs and expectations of the end user? This question is the ultimate measure of success for any project. But metrics are also useful to the team in gauging their progress and efficiency. Two simple, yet effective, metrics are the burndown chart and velocity.

Both are based on story points, which is a measure of relative effort or difficulty required to complete a given story. Part of the backlog grooming process is for the Scrum Team to review user stories and estimate the number of story points required for each one. There are many different methods that can be used for this and the one chosen by the Scrum Team isn’t as important as the need to be consistent when estimating story points.

The burndown chart is used to provide a graphical view of the number of stories in the backlog that have been completed against the total number remaining across sprints.

Figure 3 — Burn-Down Chart

Velocity measures the average rate that stories are completed across sprints. The basic method is to divide the number of story points completed by the total number in the product backlog. Over time velocity is a measure of the work that can be expected to be completed in a sprint and it is used to ensure that the team doesn’t overcommit the number of story points to be completed in a given sprint.

Teams often find a high degree of variance in velocity during the early part of a project, but it tends to smooth out as the team gains experience with the application and as a team. It is important to note that this doesn’t mean that the team stops “stretching” themselves. It’s the responsibility of the team to continuously strive to seek out efficiencies in the development process that will allow velocity to increase as long as quality is not compromised.

Processes

The main processes contained in Scrum are the sprint, sprint planning, the daily Scrum meeting, the sprint review, and the sprint retrospective.

The Sprint

Sprints are application development cycles lasting from one to four weeks. Sprint length is fixed across the life of the project and is chosen by the team. A fixed number of user stories are assigned to each sprint. This is not to say that stories cannot be added to the sprint. Just that they can’t be added if doing so exceeds the capacity of the team to create, test, and deploy a working application by the end of the sprint. It is quite often the case that teams under estimate the number of stories that can be completed and have to add stories to fill out the remainder of the sprint.

As previously mentioned sprints build upon one another. With each sprint the usability and value of the product increases as features are added and refined. Even though sprints result in a deployable application users may choose not to use it in production until it reaches a certain minimum threshold of feature and functionality. However, having a working application increment helps to demonstrate progress and solicit customer feedback at the end of each sprint.

Sprint Planning

At the start of each sprint the Product Owner defines a goal for the sprint and identifies which stories need to be completed to achieve it. The Development Team selects stories from this pool, reviews them, and commits as a team to their completion. This includes considering both the individual and collective complexity of each story using story points. One outcome of the review may be that the goal needs to be decomposed across multiple sprints if it exceeds the capacity of the Development Team

The team also plans for how they will work together to complete the sprint. This may include discussions around risks and contingencies, test plans, etc. An option available to the Development Team is to pair members of the team to work on certain stories. For example, it may be advantageous for a frontend developer and a backend developer to pair with one another to ensure that API’s are well established.

Daily Scrum Meeting

During the course of a sprint the Scrum Team gathers daily for a 15-minute meeting to report progress and roadblocks. This meeting is referred to as the Daily Scrum and its purpose is for each member of the team to review what was accomplished towards the sprint goal since the last meeting, what is to be accomplished by the next meeting, and to identify any obstacles. The goal of this meeting is to make sure that there is transparency across the team to both successes and roadblocks.

The Daily Scrum is not intended to solve issues or problems during the meeting, but to expose them to the entire team and for teammates to schedule time to work on them. This allows the proper resources to be brought to bear so the time of other teammates is not wasted. Desired behaviour on the part of the team is for individuals to volunteer to help rather than resources being explicitly assigned by the Product Owner or Scrum Master.

Sprint Review

At the end of each sprint a Sprint Review is conducted to demonstrate the working product to the customer. This is an “all hands” meeting attended by customer representatives, the Product Owner, and the Scrum Team. This meeting is an opportunity to get direct customer feedback on the state of the product, discuss issues and possible changes or new features, and to decide on what to do next.

The Sprint Review allows the Development Team to receive unfiltered feedback directly from the customer and involves both the customer and the Development Team in identifying next steps. In this respect the Sprint Review provides timely and relevant input to the next Sprint Planning session.

Sprint Retrospective

The Sprint Retrospective is conducted by and for the Scrum Team to promote continuous improvement. It is the primary means to the team uses to improve their work and to drive value not just for the customer, but also for themselves in the form of a more integrated and smoothly operating team.

This is an inward look by the team at how they performed during the last sprint and identify changes for the next sprint. This isn’t just about technology and tools, but also about procedures, interactions between people and roles, and successes and failures. The goal is to improve by implementing “midstream” corrections at the point they are needed.

Adaptations

Just as applications have changed over the decades so has the nature of teams responsible for its design and development. Where teams were once co-located in the same geographic region, time zone, and even office building they are now distributed across multiple geographies, time zones, and many work from their residences rather than a traditional corporate office location. Whereas in the past team members shared the same language and cultural backgrounds they are now separated by both language and culture.

Changes in the nature of development teams presents challenges to organizations seeking to use the Scrum framework. Overcoming these challenges requires that it be adapted to the team rather than the other way around. For example,

• Meetings must be scheduled for times most convenient for all members of the team. At the start of the project suitable meeting times compatible with team members time zones should defined and agreed upon by all. Web sites like Meeting Planner can be used to find acceptable meeting times.
• National and religious holidays in the team members geographies should also be identified at the beginning of the project and made available to all team members. In addition, these should be considered in sprint planning since they reduce the amount of time allotted to the sprint.
• Cultural and language differences between team members directly impact the effectiveness of communication between members of the team and are among the most difficult to address. In particular, gestures and mannerisms have different meanings from culture to culture. For example, shaking one’s head up and down in some areas of the world doesn’t necessarily mean agreement. All team members must be attuned to these differences throughout the life of the project and It may be beneficial to start the project with a workshop to identify and implement mitigations to address these considerations.
• Communication, both written and spoken, should be expressed in the simplest possible manner. Complex phrasing and complicated words should be avoided as much as possible. Team members having a high proficiency in foreign languages should be called upon for advice in how to best communicate with those who do not have similar skills.

The Daily Scrum is the primary vehicle for sharing information across the Scrum Team. When team members are separated by geography using video conferencing is a good means of conducting the meeting. While conference calls are also effective it is no substitute for team members being able to see one another since gestures and mannerisms make communicating more effective.

In projects where team members are employed or contracted to the same organization it is almost always possible to find a suitable time for this meeting, even across large distances. However, when team members are separated by organizational affiliation as well as geography it might not be feasible to find time for a daily meeting, even one as brief as the Daily Scrum. Although not optimal, one solution to this problem is to schedule this meeting as frequently as possible and supplement it with written communication. Tools such as Slack allow team members to post the information they would normally share in the Daily Scrum to be read by other members of the team when the are online.

Common Misconceptions

Even though this framework has been in use for over 20 years there are still a number of misconceptions surrounding it. Some of these are:

Scrum is applicable to all projects.

Incorrect. Waterfall methodology is a better match for projects required to meet highly stringent regulatory requirements which require that development pass through proscribed stage gates. Organizations must also be culturally suited to adopting Scrum since doing so requires not just acceptance by developers, but also by customers and managers whose experience is often limited to waterfall-style project management.

Scrum has no time or budget constraints.

Incorrect. In agile methodologies like Scrum resources and time are both fixed, while scope is not. Fixing resources and time sets an upper limit on the number of sprints in the project which means the project must deliver a minimum viable product that at least meets, and hopefully exceeds a threshold of user requirements by the end of the last sprint.

When human capital is constrained projects can save money by combining the Product Owner and Scrum Master roles.

Incorrect. These roles have different responsibilities and within the context of Scrum, act as a check-and-balance system against one another. Combining these roles eliminates that balance which in turn affects the Development Team.

The most important role in Scrum is that of the Scrum Master.

Incorrect. All roles in Scrum have a distinct purpose and set of responsibilities. They all play an equal part in the success of the project.

There is no need to conduct detailed design or documentation for Scrum projects.

Incorrect. The point of Scrum is to perform tasks such as design and documentation incrementally and at the right time. To be successful every project must have a vision, an overall design, and an architecture. Initially these can be laid out very broadly and then elaborated on in each sprint.

Conclusion

Scrum is an effective framework that is simple to understand, but often challenging to implement due to its radical difference from traditional methodologies and the fact that it imposes a set of cultural values on the organizations that adopt it. Cultural values like self organizing teams which may be foreign to some organizations.

The intent of this article has been to provide an introduction to Scrum, not to explore all of its nuances. As such there are some concepts such as the “definition of done” and the details of estimating story points which will require additional reading and study. It’s up to you, the reader, to take a deeper dive!

Additional Resources

The Agile Manifesto

The Scrum Guide

https://www.scrum.org