Collecting User Data in Oracle’s Usability Labs
Author: Jatin Thaker, Senior Manager, Usability Labs – Oracle Applications User Experience
Through interactive experiences with our usability professionals, Oracle obtains a clearer picture of any issues with a given design, and we use this information to improve and validate the design. This process is repeated throughout the lifetime of a product.
These steps are part of the Oracle user-centered design process depicted in the schematic to the right.
Engaging the User
Oracle maintains participant databases of individuals who are willing to take part in usability activities. These people have job experience and other qualifications identical to current and future users of Oracle products. Our current databases contain nearly 5,000 test candidates globally. These candidates are screened for eligibility on a per-test basis and sign non-disclosure agreements before they start the tests.
Recently, while the user experience team was reviewing designs for the next release of an applications suite, we received an e-mail from a sales consultant expressing frustration that a customer was having with an existing user interface.
We invited the customer to provide feedback on existing products and help to foster ideas for future designs. The customer agreed to send several participants to take part in a number of product design and development activities, including customer advisory boards, focus groups, and usability sessions.
Setting Up the Environment
We arranged an office-style usability lab for the customer feedback session. Here, we could conduct usability evaluations on software, perform guided walk-throughs with the customer, and test paper prototypes of Oracle’s products.
The usability session required a live environment that would not change between participants. We set up the hardware and software and trained the usability engineer on the process of refreshing the environment, so that each participant would see identical information. Maintaining an identical environment for each participant helps to ensure that we gather consistent and accurate data.
The Oracle Usability Labs provide state-of-the-art technology for observing, measuring, and recording usability data, both locally and remotely. The labs were founded in 1994, and there are 18 usability labs located in seven locations worldwide: Redwood Shores, Calif.; Pleasanton, Calif.; Denver, Colo.; Burlington, Mass.; Reading, in the United Kingdom; Hyderabad, India; and Bangalore, India.
Conducting the Usability Session
The Oracle user-centered design process supports international standards in usability methods, such as ISO/IEC 25062, which is the Common Industry Format (CIF) for reporting usability test results. Oracle was heavily involved in development of the standard with the National Institute for Standards and Technology (NIST), Boeing, and other Oracle customers and vendors. These standards recommend collection of data pertinent to user effectiveness, efficiency, and satisfaction with the product, in task scenarios similar to how they will actually work with the product. Customer involvement is often critical in producing real-world use cases that allow the valid testing of software under development.
Prior to conducting the usability test with the customer, we conducted a pilot test with typical users and a benchmark session with a product expert, to estimate the time it would take to complete each task, the task difficulty, and other details. Participants were informed that the sessions would be recorded and viewed through a one-way mirror and also viewed remotely using the Web Conferencing application in Oracle Collaboration Suite.
The participants then tested Oracle’s user interface and provided invaluable information about their experience with our product. A usability engineer logged this information using in-house data logging devices that help Oracle compile data more efficiently and accurately.
The sessions were observed live in the lab, as well as remotely, by usability engineers, designers, and developers. The sessions were also digitally recorded as high-quality MPEG4 files for distribution to UX team members who could not attend the sessions. The MPEG4 files were also used to review the sessions and note any additional observations that may have been missed during the live test.
Providing the Results
A summary of the results of the sessions were compiled and provided to the development team, which implemented some key changes in the software based on the results of the usability sessions. These changes will be available in the next release of the software.
A detailed report was created and published, providing an in-depth analysis of our findings. A version of the test reports in the NIST CIF format was available to the customer under a non-disclosure agreement.
The customer was thrilled to be engaged in the development process and was excited to see such involvement pay off in increased productivity after Oracle made improvements in the user interface.
For information on the Oracle Usability Labs, contact email@example.com.