Oracle Usable Apps | Applications User Experience Simplicity, mobility, extensibility
   
 
Customer Role: How User Experience Research Is Conducted
 
Lessons from Vegas

Gathering Customer Feedback at a User Conference

Author: Teena Singh, Principal Product Manager, Oracle Applications User Experience
Co-Author: Sean Rice, Principal Usability Engineer, Oracle Applications User Experience
Revised: Mar. 18, 2009
First published: Nov. 30, 2007




What is Oracle Doing to Make Sure Its New Products Will Work for Customers?

Teena Singh
Teena Singh

Recently, Oracle’s HCM UX team, or Human Capital Management user experience team, conducted a set of Customer Feedback Sessions on Fusion prototypes at a user conference in Las Vegas. It was an immense undertaking, and well worth the effort. The team gathered critical feedback and, be assured, the information gleaned in Vegas did not stay in Vegas. The valuable feedback the team received is being fastidiously incorporated into the next iterations of Fusion application designs.

Going to Neutral Ground

The UX team felt that gathering feedback at a user conference was an ideal opportunity to meet customers on neutral ground, take advantage of a constellation of important profiles, create a buzz about Oracle’s latest design innovations, and learn firsthand what human resource professionals want in their next release from Oracle Applications. 

Customer Feedback

Oracle’s UX team has established a number of Oracle-specific methods for gathering critical customer feedback. The particular Oracle methodology selected for this user conference in Las Vegas was the Customer Feedback Session.

In a nutshell, Customer Feedback Sessions are the equivalent of a Pepsi taste test. Each session is a usability activity in which a customer who fits the targeted user profile completes tasks from a prototype by clicking through it and speaking aloud. 

Customer Feedback Sessions are flexible usability activities that can be conducted outside of a usability lab. Not much equipment is required, just a laptop, monitor, and mouse. Feedback obtained during these sessions helps a designer learn how to improve the prototype design in the next iterations. The designer seeks answers to the following questions:

  • Can the user navigate naturally through the prototype?
  • Does the user understand how to start and complete tasks?
  • Does the user understand the terminology used in the prototype?
  • Can the user discover features?
  • Does the user interpret features in the way intended?
  • Does the prototype give users the content they need on a page?
  • What does the user like about the prototype? What are the user’s main points?
  • What is the user’s overall impression of the layout?

Organizing the Team

The HCM UX team started organizing the Customer Feedback Sessions months before the conference. A number of i’s needed to be dotted and t’s crossed to guarantee that the lab would be functional by the time the team got to the conference.

First and foremost, we needed space in which to conduct the tests. Conference staff members provided a room just a few minutes’ walk from the main conference room.

The next step was to choose the right prototypes to test. The UX team worked with HCM Senior Management to determine which prototypes required customer feedback. Several prototypes directed at human resources needs were selected to ensure comprehensive feedback on the design of Fusion applications.

Next, members from the product teams joined the project. For each Customer Feedback Session, a moderator was needed as well as a note-taker to record what we discovered. The UX team planned a number of comprehensive training sessions to make sure that each person involved understood the testing methodology.

The HCM UX team provided the following comprehensive workshops to product managers:

Scenario and Task-Writing Workshop – This workshop explained how to write tasks that a customer would complete during the feedback sessions. The UX team described how to write realistic, goal-oriented scenarios that encouraged participants to navigate through a prototype. 

Moderating and Note-Taking Skills Workshop – This workshop focused on how to conduct Customer Feedback Sessions. The UX team outlined to moderators and note-takers how to actively listen to customer comments and watch how they click, regardless of whether they succeed or fail, when they click through a prototype. Moderators learned how to deal with a frustrated participant or prod a customer for more information about a specific task through various role-playing exercises.

Doing a Test Run

After the training, moderators put their training into action during a pilot session. All team members, including a User Experience buddy, who is a member of the same usability team, attended the pilot session. After each pilot session, the execution of the session was discussed by the team and revisions were made as required. The User Experience buddy provided constructive feedback about moderating skills.

Planning for the Conference's Customer Feedback Sessions

gif
Aylin Uysal and Teena Singh, Oracle Applications User Experience

The layout of a temporary usability lab is pictured above.

Setting up a temporary usability lab at a conference center requires planning and coordination. The focus was not on logistics, but on watching, listening, and learning from customers as they navigated through the prototypes.

The UX team needed to transform a stark conference space into a temporary usability lab. Team members worked with Oracle marketing staff to collect banners, tablecloths, and posters. The team also worked with conference organizers to secure furniture, monitors, keyboards, and mouse devices for each feedback station. Refreshments were coordinated with the hotel services.

A barren hotel banquet room soon became a functioning lab, where customers were comfortable sharing feedback. The diagram and photograph below offer a look at how the lab was organized. 

Recruiting Customers for Feedback

The most critical planning piece was to recruit customers. After all, they were the reason we were organizing these opportunities for feedback. The UX team sent e-mails to conference attendees asking whether they were interested in providing feedback. After putting together a list of interested conference attendees, we scheduled their sessions and explained how we would gather feedback.

Time for the Sessions

Customers checking in for their feedback session were whisked away from the hustle and bustle of the casino to the quiet waiting area in our usability lab. To ensure that each customer was relaxed and in the right frame of mind to provide reliable feedback, we offered refreshments.    

The moderator then led the customer to a designated testing station, where the customer was introduced to the product team and provided with an overview of how the Customer Feedback Session would be run. The moderator typically said:

gif
Photographs by Teena Singh, Oracle Applications User Experience

The waiting area in the center of the usability lab in Las Vegas provides customers with a comfortable place to sit.

"Please understand that we are NOT TESTING YOU, or your abilities. We are testing the user interface, and if anything is unclear, difficult, or frustrating, then we have some work to do to make it more clear and intuitive.”

The moderator then asked whether the customer had questions or comments.  When the recording started, the moderator asked the participant to read the first task aloud. Now was the time for the product team to watch, listen, and learn from the customer. The customer was interrupted only if the product team wanted to gather more feedback on the customer’s impression of the design, or the customer was encouraged to speak aloud. To promote dialogue, the moderator often asked:

  • "What are you thinking here?"   
  • "What are you trying to do? "

The sessions were one hour long. After completion, some final questions were asked of the participant:  

  • "Can you tell me your top likes and dislikes of the designs you saw today?" 
  • "How does this compare to what you are currently using?"

After the feedback was obtained, the product team members thanked the customers and gave them a gift before beginning again with the next customer. Typically, eight to 10 customers provided feedback on a particular prototype.  

gif
Customers offer feedback on Oracle applications prototypes during a conference in Las Vegas recently.

Documenting Usability Findings

A few days after the conference, the product teams began to outline their findings. Two weeks after the conference, an official usability report was completed. After all of the usability lab reports were submitted, the HCM UX team prepared a complete report detailing the issues and design recommendations. The high-level issues found during the feedback sessions were sent to senior management. 

What Customers Thought

Many customers thanked the usability lab team members and said they were pleased with the experience of walking through a Fusion prototype. Almost every customer also filled out a brief survey, and every person who filled out the survey said they enjoyed the feedback experience. Even more impressive, each participant said that he or she would like to be involved in another Customer Feedback Session. Here are some of the comments we received:

"I liked being able to influence items that will affect my company in the future. I liked being able to see that Fusion will work for us when we switch.”

"Giving feedback is good; seeing the prototype in action is outstanding.”

"I feel like our needs are being listened to -- we're being included in the solution.”

"I like the face-to-face feedback session rather than on the phone.”

"Great to see how new products evolve and ability of users to give feedback.”

"Have more of these!”  

The product team and the UX team both found the experience invaluable. So much relevant feedback was provided from customers that many product teams needed to take a step back and rethink their designs.      

Summary

Conducting Customer Feedback Sessions at the user conference in Las Vegas required a great deal of planning, coordination, and precision in execution. However, the overall gain for all parties involved is immense. Just a few of the benefits are:

  • Hands-on opportunity for customers to interact with designs.
  • Face-to-face experience instead of remote or phone sessions.
  • Great occasion to build a design relationship with customers.
  • Fantastic opportunity to introduce the user-centered design process.  
  • Efficient and cost-effective method of gathering customer feedback data:
  • No additional travel cost for customers.
  • Only an hour of time invested by the customer.
  • Opportunity to collect feedback on multiple prototypes at once.
  • Opportunity to reach out to users from different locations such as Europe, the United States, Asia, and more.

Breaking Vegas Tradition

When it comes to Oracle Customer Feedback Sessions, the details of a busy week in Vegas are not going to be swept under the carpet, as they might be after some bachelor or bachelorette party. In this case, what happened in Vegas has been chronicled and documented, and will be thoroughly exploited.

The feedback from the usability sessions held at the conference was recorded, and detailed notes were taken on each customer’s interpretation of the design. The UX team reviewed the feedback from each customer and compared it against that of others. The UX team is now working diligently on the next iteration of Fusion prototypes that incorporates this valuable customer feedback.

The HCM UX team is setting up shop at the next HCM user conference, and is ready to plan usability sessions at major user conferences in both the United States and the United Kingdom.If you would like to learn more, please check out how you can get involved
Oracle.com  |  About Oracle  |  Careers  |  Contact Us  |  Legal Notices  |  Terms of Use  |  Your Privacy Rights