OTN Interview with Paul Gerrard: Risk-based Testing & Industry Trends

by Marina Gil-Santamaria

Published June 2008

OTN met Paul Gerrard to talk about Project Intelligence, Risk-based E-business testing as well as trends emerging within the testing industry. Paul is an acknowledged author, trainer and consultant in software development and test strategy & tools, and a popular keynote speaker on the project management, assurance, testing and QA conference circuit. He has received public recognition as the "Winner of Best Presentation Award", EuroSTAR 1995, and "Winner of Best Presentation of 2003 Award,BCS SIGIST".

OTN: Paul, please tell us a little about you… What drove you to testing in the first place? Did you start there your professional career? If not, where did you transition from?

Gerrard- I studied engineering at university and my first job was with a consulting engineering firm. On my first, or maybe second day, the managing partner said, "Paul we have no work for you right now, but we've just had a computer delivered. None of us have time to figure it out. Here are the manuals - get yourself up to speed. It will be a useful exercise for you." Well, before I knew it, I was the office computer expert of course. The computer was an AM Jacquard minicomputer - heard of those? Probably not! It used a dialect of basic. I wrote some nifty little record keeping and graph plotting programs but everyone knew real engineers used Digital VAXes or Cray supercomputers for real engineering work. Time passed, and I got the opportunity to use a VAX and Fortran at a small bureau facility. Wow! What a step up. I was using maps of Africa and a digitizer to capture valley contours to calculate the water volume of prospective hydroelectric schemes, I refined a poorly-written boundary-element analysis program and wrote the plotting routines for calculating/visualising stresses in the rock in hydroelectric expansion chambers. So many arrows it looked like a John Wayne Western. I was having fun. Doing really exciting stuff - in 1980. I recall telling my boss - hey when I log out at the end of the day, the VAX tells me I spent £2.03 - about 4 dollars in my all day session. "Paul", my boss says - there's a bug in their logout/accounting routine. You are spending £200 a day. Wow! I was a lot more careful then - using the VAX computer was costing ten times more than my salary! Well that was it, I was hooked - no more engineering for me. I moved through a couple more jobs and ended up an IT guy. I spent maybe ten years writing code then managing small development teams - all on VAX. I loved VMS and the flexibility it offered. My team did prototyping, incremental development, automated testing using Dec Test Manager. We also used the Dec VAXset tool kit to do code management, source code analysis, performance profiling. We had a daily batch process doing automated builds and running tests. Our builds and tests sent us emails every night. It seemed like the obvious thing to do in 1988. Little did I know, we were doing stuff that might be called advanced or even Agile fifteen years later. Then, when I was rather fed up of working for development managers, I worked for a marketing department. I was asked to plan the testing of what would now be called a 'data warehouse'. Knowing not much about testing (I was a developer, after all) I bought the only book on testing I could find - Boris Beizer's red book, "Software Testing techniques". I read the first few chapters and scanned the rest of the book in a weekend. What a revelation. I wrote a test strategy. The manager liked it. That was it. Suddenly, I was the testing guru of the department.

 OTN: OTN: In your opinion, do you see any clear trends emerging within the testing industry? Also, do you see big differences when looking at the situation in let's say EMEA, Americas, and Asia Pacific?

Gerrard- Looking back twenty or more years, the testing landscape is transformed. I don't recall there being any testing companies of note in the 80s. Back then hardly anyone even called themselves a tester (at least in the companies I worked for). Testing was a development activity wasn't it? There are several dimensions to the testing industry. I'm working on a talk called Past, Present and Future of Software Testing. I can't pretend I can predict the future but it's a bit of fun speculating. The seven areas I'm focusing on are:

(1)Attitudes - these have changed dramatically. Testing is now a recognised discipline, whereas it used to be a necessary evil.

(2)Process and Techniques - in some respects, these have changed very little. The V-model is still the main test approach preached by the certification schemes in Europe even though there are more flexible alternatives. The test design techniques of most value are still boundary values and equivalence partitions. Not much has changed.

(3)Improvement - again, the process improvement models seem popular in Europe - TPI and TMMi and less so in the rest of the world. To me, these are helpful as cross-checks, but shouldn't be used to drive an improvement programme. 'Problems of test' are most often caused by issues upstream, organisational and cultural problems. I favour a holistic approach - in fact, I always ask for my 'test improvement' assignments to be a called software process reviews. It is foolish to focus all attention on test.

(4)Tools - although most focus of attention is on test execution tools, in principle these are not advancing much. Essentially the vendors are working hard to keep pace with developer technological changes only. The tools are not really improving the test design productivity of testers. Having said that, it now seems that test frameworks have matured to the point that the major vendors are taking them seriously. Also, there are signs that the test management tools are advancing beyond basic record keeping tools and are offering much improved business goal, risk and coverage based analysis and reporting. About time too!

(5)Certification and Training - the single biggest change in the industry affecting practitioners is the rapid take up of certification. In the UK and Europe the ISEB and ISTQB schemes have more than 40-50,000 successful candidates. Certification regularly appears in job ads as both a desirable and a mandatory requirement. Now, there is quite a polarised discussion as to the value of certification. I think it has value, specifically in helping to encourage consistency in terminology, test case design and notions of coverage and structured process. But I also think the syllabuses in use are too theoretical, ideal world and waterfall based. As a proportion of what testers need to know, I'd put certification content at between 10% and 15% of the total - which isn't that much, really.

(6)Outsourcing - there's no doubt that outsourcing has become an everyday activity, especially in larger companies. I would say that outsourcing has moved from the hiring of specialists in non-functional testing and test management to much larger scale outsourcing of teams - sometime 100+ strong. Historically, this aspect of the business has been straightforward body-shopping. But body-shopping is not a testing business - it's somewhat like a factory process. Recently, there are signs (at least in the UK) that managed services are gaining ground. Whereas body shopping does not advance the industry, managed testing service companies have to offer more sophisticated value-add services and through longer-term relationships with their clients. Managed service companies will become the engine-room of progress in the testing industry as they gain influence and market power.

(7)People, skills and the job market - there's no doubt that the job market has matured to the degree that a significant percentage of jobs in the IT industry are now for career testers. Although the majority of acceptance testing is still performed by end-users seconded from their business, there does seem to be a trend to develop permanent acceptance teams from internal resource, but also to bring in new blood and ideas from the testing community. As far as specialists are concerned (e.g. automation specialists and non-functional testers) there is a worrying theme of hiring tool experts rather than testing experts who have tools skills. But this is part of the market growing up perhaps. As far as the UK is concerned, the test Management Community is thriving. I host the UK Test Management Forum (uktmf.com). This is a rapidly expanding networking group that meets quarterly. In January 2008 we'll run the second annual Test Management Summit. I believe the TMF community will also have significant influence over the UK testing market in the coming years.

OTN: Changing the topic a little. Paul, you devised a very interesting concept of "Project Intelligence", a corresponding framework and a course on this topic. How would you define it, and also can you share with OTN a little about how it "started"?

Gerrard- It all started with a keynote for Eurostar in 2002. The theme of the conference was ”The Value of Testing”. At first, I thought this was a terrible theme - but it really made me think hard about what exactly the value of testing is. This was a tough one - it really focused my mind and it eventually turned out to be a great stimulus. As a tester, as a provider - who are we to put a value on our product- we're not independent, are we? So I thought it might help if we think like marketers for a moment and view testing as a service or a product. Would that help? Maybe if we think a little more like marketers we can enhance not only the value of our testing, but also of our customers' perception of it? To cut a long story short, I came to the conclusion that the only value of testing is in the information we provide to stakeholders, management, developers, authors. Essentially, testers measure achievement. It's OK for project managers to monitor time and cost and activity status. But time and cost are inputs, not outputs and activity is definitely not the same as achievement. The real measurement of achievement is successful completion of the reviews or test of the products of those activities. Products can be documents or code or systems. But I thought straight away that the disciplines of testing can also be applied to testing services provided by integrated people, processes and software. And then it occurred to me that we could apply the same measurement approach to all business benefits. Stakeholders want to achieve business goals - software is just an enabler - why don't we offer to measure benefits too? Now, all this was happening round about the same time as was writing the Risk-Based E-Business Testing book with my friend Neil Thompson. How does risk fit in here? And how about the traditional coverage-based approaches? It was getting messy. But then it became clear. We should define a strategy for collecting information to demonstrate achievement and address risk and with sufficient coverage to be confident in our decision making. So why not propose a test strategy that aims to measure successful achievement of all technical and business goals?

OTN: Related to this, I know that you recently had a presentation on Test Methods and Tools for ERP implementations at TAIC PART 2007. How do you apply Project Intelligence to ERP testing, and specifically to SAP environments? Also, what are your thoughts regarding ERP testing and automation in general?

Gerrard- TAIC PART is an interesting and unusual kind of conference organized by the academic community in the UK. TAIC PART stands for Testing in Academia and Industry: Practice and Research Techniques.

OTN: Paul, you have written a book on Risk-Based E-business Testing, how would you define risk-based testing? What are some of differences that you see between risk-based testing/test planning when working with e-businesses or with other applications/environments?

Gerrard- The best way to answer this is to reproduce a section from the book. In it, I tried to set out the scope of a risk-based tester in the form of a Manifesto. But, taken out of context, it doesn't quite explain the complimentary roles of coverage- and risk-based testing. Essentially, requirements or coverage-based testing generates a set of evenly distributed tests across those requirements. Risk-based testing compliments those tests by identifying where failures would be of greatest concern, and adding tests to enable a refined risk-assessment to be performed. The Testers' Manifesto

(1)To assess software product risks. The tester needs to identify and assess, through consultation, the major product risks of concern and propose tests to address those risks.

(2)To estimate overall test effort. Based on the nature and scope of the proposed tests and using their experience, the tester needs to estimate how expensive and time consuming the testing will be.

(3)To gain consensus on the amount of testing. To achieve, through consensus, the right coverage, balance and emphasis of testing.

(4)To acquire budget to do enough testing. To convince management to allocate enough budget to conduct the testing to address the risks of concern.

(5)To plan, implement and execute tests. To specify and run tests that address the risks of most concern.

(6)To provide information for a risk-based decision on release. Perhaps the most important task of all, as information is the major deliverable of all testing.

OTN: Paul, a common question that a lot of OTN members have is: what is the typical ratio of developers to QA Engineers, and how do you go about estimating testing time versus development time?

Gerrard- Like most of your questions, this is a difficult question to answer!. The question is usually asked by test practitioners who feel they are "outnumbered" and they want some argument for increasing test resource. Even if the need is to optimize the 'ratio' there obviously isn't enough information in the question to provide a recommendation. What are the development and testing tasks? Estimate against those and the ratio comes from that. The question is premature, without understanding what developers and testers do. Potentially, testers can be a small minority - if developers take on most of the test activity. In a small agile project with well-disciplined developers, it's possible that end users do much of the testing, and testers aren't required at all. In other environments, such as a maintenance environment with little test automation, testers would probably be in the majority. There are too many variables I'm afraid to give a meaningful answer to this question.

OTN: Thank you very much for your time Paul! I would like to focus a little in the testing industry in Europe/UK to wrap this up. What are some of the recommendations that you have for folks starting a career in testing/QA there? Also, do you have any tips that you could share with some of our "newbie" OTN member to help them prepare for growth and success?

Gerrard- I would advise all testers, experienced and newcomers to recognize that the testing discipline, unlike development, is not technology-focused. It is an information business, a people business and a business business. What do I mean by that? The purpose of testing is to collect, analyse and disseminate information. Testers do not write code, they do not put bugs in and do not take bugs out. In this respect, we do not improve quality or add value to the products we test. We are however responsible for providing the most valuable information required by developers (to fix bugs), project management (to understand achievement and product-risk) and stakeholders to recognise business value. In this one respect, testing is all powerful - it is the single source of knowledge of achievement in software projects. Testing is a people business. By this, I mean that most system and acceptance testers need excellent interpersonal skills, particularly communication skills. Testers need at one time or another to communicate with, negotiate with, influence and advise end users, developers, technical/environmental support, project management, business and stakeholder management, internal and external auditors, outsource companies, customers, client services, and product managers. Interpersonal skills are, in that kind of environment, more important than technical skills. A business business? Ultimately, testing exists to detect bugs to protect end users, but we should look to a higher goal: to inform business user and management decision making. The stakeholders who must decide to release, accept, delay, re-think or re-plan software projects are critically dependent on the information produced by testers. In this respect, testers are the best friend of business and stakeholders. Obvious, really - isn't it? So I have these suggestions for all testers:

(1)Understand the value of the information you provide to your projects. Focus your attention on learning what information your stakeholders need and in what form, and pursue a test strategy designed to optimize that information service.

(2)Recognize the importance of interpersonal skills. Hone them, practice them, evaluate and refine them. Seek out good communications skills training and enroll.

(3)Acquire friends in your business. Ask them what their fears are for success and what information they need to make their acceptance decisions with confidence. If you work for a product company, your strongest advocates will be in Product Management or Customer Services.

(4)Find a coach. This might be your boss, but could be a colleague or even a stakeholder. Seek out an adviser and leader who will give you a lead, push you and encourage you. In particular, find a coach that will advocate your role as an information provider and decision enabler.

Marina, you are most welcome!.