OTN Interview with Bob Galen: Agile and Scrum

by Marina Gil-Santamaria

Published June 2008

OTN spoke with Bob Galen about performance testing -- If your organization is looking into implementing agile/Scrum, or you just want to learn more about these development techniques to keep up to date with what is happening in the industry, this interview with Bob will give you a great insight into it.

OTN: Bob, you have a wide background on software development, project management, process management/improvement, testing and QA, so what drove you to testing on the first place? What do you find interesting about this field?

Galen- Throughout parts of my early career I found myself occasionally leading and/or building QA teams as part of my development role. I found QA to be a weakness of mine, so I worked hard to get training and gather insights into the role of testers, since I felt I needed them to effectively lead and understand their role and the respective challenges.

A few years ago I happened to be out of work and a QA management role surfaced. I took that position, which was the first time I had only a "QA hat". Boy what a difference! I found that being a development leader is much different and in many ways easier than being in QA. For example, I don't believe the business understands nor appreciates the QA role as well as the development one. Also, I found that doing a solid testing job is at least as hard as equivalent development efforts. IMHO, it is a lot harder. In QA, having sufficient resources seems to be an ever present challenge – so balancing risk against coverage is a very dynamic proposition. You're also challenged to balance product testing against process improvement and automation development—all with the same resources.

In my latest role at Channel Advisor, I have moved back to a pure development leadership role. However, my exposure to QA has helped me immensely in understanding the balance required for producing high quality software. I don't think I'll ever be quite the "same" developer

 OTN: Bob, you are working on a new Software Planning and Estimation book, so what do you think is the best approach to tackle estimating testing time and QA resources when planning within a Waterfall-type of project? Do you have some quick guidelines, for example if development will take X man hours, testing should be a Y percentage of development time, or this is not feasible in your mind?

Galen- In the new book I'm dedicating a chapter or two towards QA estimation & planning. Although the books' focus is more for the Software Project Manager, my view is that PM's need to have a more solid understanding of QA work in order to more effectively partner in the overall planning. I think that at a very high level one can view the ratio of developers to testers, and developer work production, as some sort of a very high level predictor of testing resource requirements and project portfolio workflow planning. I think the value of this is more at a strategic level, and targeted towards team capacity planning. I don't think I would ever use such practices to plan "actual" testing projects / work. It is just too abstract and inaccurate.

When it comes to estimating & planning real test work, instead of using some sort of ratio for guessing, I prefer to get the teams (Development, Test, Operations, etc.) together and use collaborative planning techniques to pull together the right testing plan that accommodates all of the variables facing the team. To use other algorithmic or historical data based approaches is a fool's errand in my mind.

The Agile techniques recommend these techniques as well. For a given project, you engage your team in brainstorming tasks, estimating task duration, discussing tasks sequencing and aligning important milestones for integration. Instead of this coming from a spreadsheet or other tool, it comes from the team—creating much more buy-in to the results. As a part of the estimation brainstorming, the team itself normalizes the estimates to values that include different values and points of view. There's an Agile technique called Planning Poker that I have found to be quite useful in these sessions.

OTN: You just briefly touched on this, but what are the main differences that you see when estimations are being done within a project developed using agile and Scrum methodologies?

Galen- The Agile Methodologies are not plan-driven. Instead, you drive work from a simple list (a Backlog) and timebox the work—delivering & inspecting "chunks" of functionality quite frequently in weeks not months. You measure iteration over iteration throughput (Velocity), and use this capacity measure to predict how much effort the team can produce in future iterations.

I think of this approach as being reality-based and true capacity driven. Instead of trying to accurately predict the future, which is inherently difficult due to software project complexity, you measure output and extrapolate it against your overall backlog of features (and other work) to predict release timing.

In practice, I have found velocity-driven planning to be a much more humane way to manage software projects. It also makes your capacity completely visible to stakeholders. Instead of a "plan, then demand, then accept disappointment" model —they are fully aware of the teams true capacity, and are part of adjustment (usually scope) decision-making

Estimation in this case uses the same "units" as you measure velocity. For example, you estimate story complexity in perhaps Story Points, which include development and testing time. You then measure how many points the team can deliver over 1 or more iterations, and this becomes your velocity. You can also measure velocity in hours. Mike Cohn has written a wonderful book on Agile Estimation & Planning that explores these concepts much further.

OTN: How does testing works within a typical agile team? At which point of the sprint cycle do you think that QA teams should think about functional testing, regression testing, performance testing?

Galen- To begin with, our traditional testing approaches immediately change in an agile context. Requirements are usually short and fuzzy things called stories, that aren't well defined until right before or even during implementation. Since most testers are accustomed to having well defined, or heavier weight requirements, this causes their planning and process norms to break down.

The typical phasing of testing is also disrupted. Instead of testing in large waves, for example running pervasive regression tests, agile testers need to run their test phases in much thinner, iterative slices as appropriate for the features being delivered within each iteration.

There is a normal skew within agile iterations for tester focus. The first priority is normally focusing on unit and feature acceptance testing. Once features are stable, you turn next towards partial integration and focused regression testing. Usually, you run out of time to do more pervasive testing within the iteration. As each iteration completes, you end up building up test debt surrounding things that have not been tested.

One Agile technique used to eventually mitigate this risk is running what's called stabilization or hardening iteration(s) after a number of development iterations. These iterations are much more focused towards formal testing than towards development. This is when you focus on system wide integration, full regression, and performance testing. It sounds waterfall-esque, and to a degree it is. However, it many domains it is virtually impossible to complete all requisite testing within the context of a development iteration.

OTN: If we think about a sprint backlog and the process of creating user stories, do you have some quick guidelines about estimating testing time?

Galen- First of all, each story should be estimated from a single effort perspective, so development time and testing time should be combined into one estimate. Sure you can call out each separate effort estimate, but from a work (story) getting done perspective, all work needs to be completed before you get credit for completing the story. Many agile teams use the notion of Earned Value in their tracking, which only allows the team to take "credit" for completing a feature when it is completely (done) with all acceptance tests running successfully.

A big part of testing time is associated with the number and level of acceptance tests that need to be written and automated for each story. So a large part of testing effort estimation is driven by that.

As I have already mentioned, much of agile planning revolves around the team brainstorming task cards, estimating them, and then planning the work via iterations. So the testers simply participate as part of the team as they form a solid agile release plan. A really important part of the role testers bring to this estimation process is beyond the individual story. It is at the level of release planning!

Agile teams quite often become very feature focused in their delivery and they "forget" that they're delivering a product. Testers need to bring insights on integration phasing, where performance tests need to be run, where regression opportunities exist, how to mitigate regulatory requirements, how to fit automation work into the iterations, etc. into the planning. This higher level view to workflow is something of great value for the team and a unique perspective that test has.

OTN: In your opinion, what are the main differences that a QA Engineer/tester professional should see on his/her role when working on an agile/Scrum environment versus a more traditional type of project? Do you have any recommendations for folks transitioning into agile testing roles?

Galen- I have written a few Scrum articles for Software Testing & Performance magazine (June 2007 and June 2007) and each of them contain hints surrounding this transition. I have quite a lot of passion around it because I see so many traditional testers struggling to survive and thrive in agile teams. Some don't make the adjustment to agile techniques and literally get "ejected" from them. Still others change too much, throwing away their traditional testing techniques and losing their rigor and thoroughness. That doesn't work well either. I think the key is to realize the key differences between agile and traditional techniques, but adjust carefully and retain those tried and true approaches that work.

Here are three key areas that I immediately think of as central to Agility:

(1) High levels of collaboration are required— One of the hallmarks of agility is collaborating across the team. Testers are sort of a glue layer that facilitates this. From a requirements perspective, they should work with the agile customer to define requirements (stories), acceptance test cases, and then running the tests. You could make the case that they move towards a more traditional Business Analyst role. Beyond that, they are also working with the developers to implement, test, and accept features. They very much facilitate the definition, construction and verification of each feature being implemented per iteration. At the same time, they're also responsible for their more typical, for example regression and end-to-end testing.

(2) Technology comfort, software development skills, and even soft skills—Quite often agile testers pair with developers during feature development. Often they are focused towards verifying the operation of a feature and running various sorts of tests.  You have to be comfortable with your product and development technologies to do this well. While this might not always mean writing code, you need to bring some value to the table as a development peer—even if that is relentless listening and open-minded learning from them.

Beyond the technical aspects, testers in an agile environment need to be good communicators. This isn't solely focused on bugs, but includes all aspects of getting the project done and sharing insights on just how well the code is evolving and features are being implemented. Consider this the Voice of the Customer and the Compass for the Project personas, which are crucial for agile teams. Finally, there needs to be a transition from "Quality Cop" to one of "Quality Partner".

(3) Comfort & trust with Just-in-Time delivery—Everything in agile is built incrementally. Part of it is driven by the Lean principle of reducing and avoiding waste. Waste in this sense is anything that the team produces that isn't truly used. We've all probably written a 100+ page test plan because the process or our boss says to write one. Even though we know nobody is actually going to read or use it. From a Lean perspective, this effort was waste.

Agile/Lean tries to build things in smaller chunks—so perhaps 10 pages at a time. You iterate until nobody is reading it or using it, then you stop. This perspective permeates coding, requirements, planning, testing—literally all aspects of agile teams. I think it is very hard for testers to comfortably grasp it because they think it leads to shoddy work.

A final note about testers transitioning to agile teams. Look for a team that truly "gets" the agile quality proposition. That is—it isn't QA's job to gain application quality, instead it is the entire teams' job! Look for developers who understand unit testing and how to develop solid code. Who are passionate about quality and help achieve it by working with QA and the rest of the team to deliver excellent code. Look for a sane pace, mutual respect, and congruent organizational leadership. This is a good Agile environment and one where testing can be absolutely amazing!

OTN: Thank you very much for your time, Bob. I know that you are very interested on software process improvement, so let's finish with another quick tip. For OTN readers looking for ways to improve their QA processes, what do you think are some of the first things that they should evaluate?

Galen- Well, keeping in line with the agile theme. Always collaborate effectively as a team; make your work very visible —to everyone; work on the most important things first—always; and avoid as much waste as you possibly can. Multitasking is a waste element that I didn't mention above, but it is quite insidious in effect.

In my experience it is very common for testers to multitask across testing projects and efforts. However did you know that a task switch can often cost you 20% of your time between just 2 efforts? It is true! And it gets worst as you add more projects. By simply focusing more in your personal and team testing; you can get a very significant time return.

Many thanks Marina for asking me to contribute. It has been my pleasure!