OTN Interview with Jon Bach: Exploratory Testing

by Marina Gil-Santamaria

Published June 2008

OTN spoke with Jon Bach, to continue our discussions around Exploratory Testing and Session-Based Test Management. By reading this paper you will get to know a little bit better Jon and his perspective on various topics, as well and the philosophy behind Exploratory Testing & Rapid Software Testing

OTN: Jon, can you please share with OTN a little about yourself. I believe that you have a degree in journalism, right?

Jon- Yes, I have a BA in Journalism from the University of Maine. I am not using it, so I wish I could sell it on eBay, but it did turn out to be relevant to my testing career.

 OTN: So, how did you get from a career in journalism to testing?

Jon- After college, I worked at Borders Bookstore in Tacoma, where I became friends with the guy who ran the Computer book section. When he left the company, he encouraged me to join him at a small start-up company that provided an MLS (Multiple Listing Service) for realtors. It was just a data processing job, but it was double the pay. More importantly, it was fun because I found all kinds of bugs. The programmer (who was the company president) found my reports helpful, and I quickly become fond of testing as treasure hunting. Meanwhile, 12 blocks away, my brother James worked at a software testing company called ST Labs. Suddenly we had a lot in common. I no longer saw him as just some super genius computer whiz way above anything I could understand, but as a very down-to-earth practitioner of something in which I was now getting very interested. I never bothered to learn that he was an expert in the field until I started working with him as he developed new ideas for testing techniques. Each day after work, I would walk the 12 blocks down to his office where he helped me cultivate my skill in return for me asking dumb questions about his testing theories. In a few months, I knew enough to be dangerous, and got a job as an entry level tester for Microsoft. Twelve years later, I am not quite James' equal, but I do consider him a colleague.

OTN: How would you describe your testing philosophy?

Jon- I am glad you said testing and not QA, because I can not Assure Quality. But testing is an art and a craft like journalism, so it is not much of a stretch from my degree. Both involve the pursuit of truth, asking questions, consulting sources, and communicating a story to an audience. Without a journalistic ethic, though, testing becomes a mindless confirmatory exercise rather than the rich set of thinking skills it needs in order to be done well.

OTN: You and your brother James developed Session-based testing in 2000. What is the relationship between Session-based Testing and Exploratory Testing?

Jon- Session-Based Test Management is our suggestion to make exploratory testing manageable and measurable in ways of service to project stakeholders. It is a framework to guide exploratory testing using the concept of a session. A session is a unit of time roughly 45 minutes to 3 hours long where the tester is guided in their exploration by a charter or mission statement. When the tester feels they have fulfilled the mission, they write a report with their notes, bugs and issues which is then debriefed by a test lead. The biggest notion my brother and I are trying to advertise with this method is that exploratory testing can be structured and purposeful in a way that leads to improving testing skill. It is a way to make exploratory testing accountable for those who want more autonomy and freedom from their supervisors, and for projects that notoriously find their best bugs *without* the use of test cases, but need to discover why.

OTN: And between Rapid Software Testing and Exploratory Testing?

Jon- Rapid Software Testing is my brother's creation -exploratory testing combined with heuristic testing and risk-based testing. If exploratory testing is an approach using a set of techniques (domain, stress, flow, etc), than Rapid Testing's focus is on the tester -how they think, what they report, what they're influenced by- which is usually things like tight time constraints, vague or missing documentation, and scant resources. Despite these obstacles, it is possible to do expose some important issues quickly. Rapid Software Testing is a way to frame answers to those kinds of problems with focus on epistemology- or the study of how it is we know what we know. It is an exploratory mindset using an exploratory skill set.

OTN: What about development and QA relationship…do you have any tips on how to improve those for our OTN members?

Jon- At the Conference for the Association for Software Testing (CAST) last week, there was a tester competition. Thirteen small teams competed for $2500 in prize money. The winners happened to use a strategy called "co-location". They sat close to the developer, who happened to be a judge for the competition. They even called themselves the "Hey, David!" team because the developer (David Gilbert from SiriusSQA) was sitting just a few feet away. They were asking him a lot of questions and he gave them a lot of information in quick exchanges. That is the relationship I like to see on projects. When my brother and I developed the open source scan tool to parse session files to use with the Session-Based Test Management approach, it was the same way. We sat next to each other – he wrote code and threw it to me on floppies and I would test it as fast as he could provide it. The feedback loop was very tight. He fixed things on the fly from my verbal bug reports and I tested his new stuff as soon as he told me about it. Very rarely do programmers appreciate that kind of service. They frown and ask us "Who would do that?", "Are you serious?", and say "It does not happen on my machine". But I understand completely. I have been a programmer on two small projects and had a love-hate relationship with testers. I hated when they said vague things like "This does not work" and reported things that were not bugs because they did not take time to read the release notes. But I loved when they found dumb mistakes I made, and when they found problems that would frustrate my users. I loved when they asked for more information or suggested a way I could make the thing more testable. It goes back to my days as a college newspaper reporter- there were writers and there were editors. Each has a different agenda. One is to create, the other is to critique. But the overall mission is the same: great content for the customer. I loved when editors saved me from typos or factual errors -things that would embarrass me or undermine my credibility. As a tester, I try to be the editor for the project programmers. My aim is to be their personal Secret Service, working undercover. No one has to know they made mistakes or did not think thoroughly through a design. Just treat me with respect and I will return the favor.

OTN: From an organizational perspective, do you think that QA should be an independent group or report back into dev?

Jon- It doesn't matter to me whether I report to Programming or if I'm independent, but I think your question is about impartiality. I consider myself like a CIA agent. I am placed in the field to gather data toward a mission. The government can ignore the intelligence I gather, they can act on it, they can mull it over, weigh it in context, whatever. The focus is not on what they do with it, but with what my value is to them. If I report into Dev and they ignore my bugs because they're too embarrassed to acknowledge them, they do that at their own risk. Have I ever been told not to file bugs? Yes. Did I do it anyway? Yes, but I assigned the bug to me just to be sure it got fixed before release -but that hits at ethics, not service, necessarily. If my boss is a programmer, it comes down to ethics and if I am going to be asked to ignore testing of certain features or cut my testing short because I was told to. But I will always be willing to risk the assumption that developers want to ship good code -that they, like me, want to be useful and provide value. I do work for an outsource (onshore) test lab in Seattle, so the programmers are always physically separate from me, but my lab has been hired by Development departments and Testing departments. My role has been the same on each - find good bugs (and good information) quickly, and be accountable to my work.

OTN: In your mind, what is the perfect development to QA ratio?

Jon- "Perfect" for me depends on the context. On small projects, I like 1-to-1 because I like to get to know the programmer without other distractions. On large projects, I like one or more testers per programmer because that increases the chance that they are getting a variety of bugs for their code assignment. Plus, it allows me to be triggered or inspired by other testers' ideas.

OTN: What are some of the most challenging projects that you have worked on, and what did you learn from them? Out of curiosity, anything that you would have done differently in your testing practices or past projects knowing what you know today?

Jon- On a project I just finished, I was tester and programmer –what our industry would call a Software Design Engineer in Test. It was hard to switch hats, but it had to be done. Certain things I'd miss as a programmer, I'd find as a tester. Things I'd miss as a tester, I'd find in my code. It was a very schizophrenic experience for me. I made a lot of dumb mistakes because of the context-switching. But the most important thing I learned was around the notion of the cost to fix a bug. We have all heard that the longer you wait before fixing a bug, the more costly. Even though I had an experience with that on this project, I also had an experience which NEGATED that. For example, I found a bug in my code that I let linger for weeks because it was really tough to fix. I put it off for lack of brain cycles and time and energy. It had to do with building an Excel macro that scored a certain array of cells depending on the value of those cells. The bug was that my macro did not weigh the score correctly. I knew why, but the fix was intense. Weeks went by and the time came for me to make a call on what to do. It would easily take a week to sort out. Coincidentally, that same day, I got an email from the project planners saying that the array of cells in question had been ripped out of the design! Had they delayed sending me that message for a week, I would have spent all that time coding a fix (or a workaround) for nothing! So sometimes, the more you wait, the LESS expensive the fix. No one ever talks about that because it's too controversial, but I have seen that context on more than a few projects. Another example was a project I was on where the programmer let one of my bugs languish for 213 days. When the triage committee asked him what the deal was, he said to be patient. The next week, Microsoft released an SDK which had the module he needed that would fix the problem in 10 seconds. Had he spent time fixing it up front, it would have been a waste of time and resources. Sometimes technology comes up with a fix for us, saving us time and money if we just wait a little while. Do I recommend always waiting to fix a bug? No, but I am saying it depends on your *context* and I encourage more people to find examples that challenge blindly accepted notions like "the sooner you fix a bug, the less expensive it is."