OTN Interview with Karen N. Johnson: Performance Testing

by Marina Gil-Santamaria

Published June 2008

OTN met Karen at STP Conference in Boston, and ask her some questions related to performance testing and her session there. If you are looking to learn more about performance testing, such as tips, or skills needed to tackle projects in this space, or if you want to read about more general topics such as improving relationships with your development counterparts, please read on

OTN: Karen, can you please tell us a little about yourself�what drove you to testing? Did you start there your professional career

Johnson- Like Jon Bach, I have a Journalism degree. I truly enjoy writing, and I'm an introvert by nature, so the combination was a natural fit to become a technical writer. When I graduated from college in Boston, the computer field was booming and I started working as a technical writer. I learned about computer systems and was content to be left alone to write. After about seven years of tech writing, I found I was drawn to breaking the software and was growing tired of writing tech documents. I made the switch over to testing fifteen years ago. I can't believe it's been this long!. Technology continues to change and I've made several job changes constantly seeking new technology as well as intentionally seeking fields I've not worked in before, such as moving from banking to manufacturing software, and onto contact management, e-commerce and several others. The variety keeps my work fresh. I remember my first days of testing thinking: this is amazing, I get to sit in the back and break software! As the years pass, I'm far less introverted and I have returned to writing, although now I'm writing articles about software testing and hopefully teaching other testers in the process.

  OTN: What recommendations do you have for newbie OTN members that are looking to start a professional career in testing?

Johnson- I really think that there is no magic formula that you follow, or book that you read, and voilà!, you are all set. That being said, I think that there are great testing books out there that will give you a great foundation to start with, such as "Testing Computer Software" or "Lessons Learned in Software Testing". I think taking an honest assessment of your skills is a good start to setting what you need to learn. You should ponder areas such as: what do I like about testing, where are my weakness, where are my strengths, where do I want my professional career to advance to?. Once you do some thinking about these, you can set milestones and targets for yourself, and put work towards learning that will help you to achieve your goals. For example, I have always considered my weakness to be around math and statistics, so I self-train on this space taking evening courses on these topics, and reading a lot on the subject.

OTN: Let's talk about performance testing, Karen, you just had a one hour session on STP Conference on this topic. In your opinion, what kind of specialized skills are required for performance testers that are different from standard QA / functional testers?

Johnson- This is an excellent question because I think there are very different skills needed for performance testing versus functional testing. I believe performance testing is one area that highlights the value of having a multi-disciplinary skill set. A mathematical background or knowledge, statistical knowledge, graphical presentation skills are some of the skills needed. Strong analytical skills are essential to discern information out of the volumes of test results often generated from performance testing is needed too. I'm not saying these skills aren't applicable in other areas of testing, but these skills might be needed less based on the type of software being testing. In performance testing there's almost a guarantee that these skills will be needed.

OTN: Karen, what do you mean by graphical presentation skills?

Johnson- I'm referencing our skills in Excel or Power Point or other tools we may use to show and explain the data we've discovered from testing. Especially in performance testing, the information we learn can be shared graphically and probably is best shared through graphs and charts. You might not expect graphical skills to be needed in testing, but performance testing is one area where graphical presentation is often needed. Being able to show information graphically can be valuable. I recommend studying the work of Edward Tufte and Bill Jelen.

OTN: How do you typically prepare to tackle a new performance testing project in terms of tools, resources, estimation of testing time, etc?

Johnson- This is another question that if I answer too briefly or too prescriptively, what I say might not help someone. I think each project is different and so planning needs to reflect the project specifically, so there is no cookie cutter answer. Available resources, budget and time will impact the decisions that get made. One thing that I do when planning performance testing consistently across all my projects is to make a point to walk around or schedule time with the key project stakeholders and assess their expectations. If I need to adjust either my planning or provide people with a reality of check of what we're going to be able to do, I start with each person one at a time. After I've finished this legwork, I write a test strategy and make sure that what will be executed is openly discussed.

OTN: Do you have any tips for our OTN members about data analysis, and how to assess and focus only on performance testing relevant data?

Johnson- Performance analysis was the topic of my presentation and a topic I have tremendous interest in. I don't think this is a question that can be answered briefly without the risk of omitting information, but I can outline three (of several) things to think about. One is to group similar tests together before analyzing information. I know this may sound fundamental, but in performance testing we execute many tests, and we often alter the criteria and variables of the tests (criteria such as ramp up time, think time, etc.). We need to be careful and segregate like tests, so we can begin conducting an accurate analysis. Another step is to review each test execution to determine whether the test was "clean", or there were errors or problems during execution which need more investigation before a test is included in the set of tests used for analysis. And a third concept I can outline is, you should consider performing a short statistical view of the test to conclude whether it should be included or excluded. Tests with wild standard deviations or outliers might be tests that should be held aside for a different analysis, but possibly not included in performance transaction timing analysis

OTN: Changing the topic a little, there is always room to improve development and QA relationship in any organization. Do you have any quick tips for our OTN members on how to interact better with their development counterparts?

Johnson-The relationships we build with development is crucial. One way to improve this liaison might be to remember that despite our sheer joy in finding defects, what we are often finding are someone's mistakes or oversights, and no one needs their mistakes rubbed in or pointed out in a mean-spirited way. As a tester or as someone who leads a team of testers, I realize as I find mistakes that the next mistake I find could be my own. Testers make mistakes too. So I believe strongly in reporting defects in a respectful manner which includes not only the text that gets entered in the defect tracking system but how we discuss defects in team meetings or the hallway. While I might be thrilled to break the software, when I report the defect, I report with respect and sincerity. Other relationships that are of great value to testers are our DBA's, network specialists and customer support. These are teams or individuals that can help testers with technical knowledge and ideas about where we can continue hunting for bugs.

OTN: Thank you very much for your time, Karen. Let's end with a quick recommendation. For OTN members that are looking to grow their team, what do you think are the most important skills or personality characteristics that they should be looking for in new team members during their hiring process?

Johnson-Curiosity, I can't stress this enough!. Testers need to be curious people, constantly wondering things such as what if I do this, or what if fill-in-the-blank here. Technical skills can be taught if someone has the aptitude and motivation. But behind all good testers, there is a burning curiosity. I think this is why testers enjoy TV shows such as the CSI series. We like to solve mysteries, we like the investigations, the fact-finding and the details. Coincidentally I think having a curious mind keeps us young and engaged with life as well.

Many thanks Marina for asking me to contribute. It's been my pleasure.