User Test

Number of Participants	Facilitators	Category	Duration	Level of Diffuculty
min. 5	At least 1 Facilitator	Prototype & Test Phase	3-4h	Moderate

Description

What is user-test?

User-test method has been known as a beneficial usability method which involves users of products or interfaces directly. The main purpose of the method is to collect valuable data regarding how real users react, feel and behave while using a certain interface. Additionally, further problems might be discovered during a user-testing workshop. Various supplementary methods available, which can be employed in a user test. These methods can be used separately, however, it is recommended to combine some of them in one user-test process to get better outcome and higher quality of test analysis. Some common supplementary methods are:
Think-Aloud Method -Users are asked to verbalize their thoughts during the test, so that how users interpret or understand the interface can be discovered.

• Questionnaires and Interviews – Users’ satisfaction with and preference for the interface can be assessed and evaluated.
• Eye Tracking – The observation of exact user flow on the interface can be recorded using a special eye-tracking device, which is attached in the monitor.

When to use user-test?

User-test can be used at the later stages of the design-thinking process, namely Prototype and Testing. For Prototype, the design team can test the first versions or functionalities of the product within the team or with a small group outside of the team to identify the best solutions for the problems defined in the earlier stages. This is a good way to investigate how well the design solution fits to users’ needs and integrate users’ feedback in the design as much as possible. In the Testing phase, the complete product is available and tested by users. Usually the testing phase is an iterative process, in which user-testing plays an important role. This process will help the team to discover further needs or problems regarding the available products. Additionally it is also helpful to further discover users’ behavior, thought and feeling towards the product. Changes or further development can still be made in accordance to the analysis of the user-test if necessary at this stage.

Materials

• Post-Ist
• Flipchart Paper
• Markers
• Voice / video recorder
• Facility ready for user-test workshop: mobile devices, computer and in many cases stable internet connection is required for the test.

Preparation

General Preparation

Workshop instructions to be cleared and easy to understand
The tasks for the user to do during the test to be printed out or to be displayed on the screen. In case users are divided into groups and each group has a specific task, the tasks need to be displayed at the group-work stations.
All electronic devices and internet connections to be tested before the workshop to avoid technical errors
The preparation varies according to the supplementary methods that are applied in the user-test process, which are explained briefly below. These supplementary methods are the most common and can be used in combination to provide better outcomes. In that case, the preparation needs to cover the requirement of all methods employed in the process.

Supplementeay Method 1: Thinking Aloud

• Voice / video recorder
• One notetake/facilitator to be assigned for every tester.

Supplementary Method 2: Questionnaires & Interviews

• Choose the applicable methods: Structured, semi-structured, open interview or questionnaires or a combination of both.
• Build an interview outline based on the goals of the user-test
• In case of using questionnaires, we can conduct our own questionnaire based on the goals of the user-test or use the available usability questionnaires. Some common satisfaction/usability questionnaires:
  • System Usability Scale (SUS)
  • Questionnaire for User Interaction Satisfaction (QUIS)
  • Computer System Usability Questionnaire (CSUQ)
  • End-User Computing Satisfaction Questionnaire (EUCS)
  • Website Analysis and MeasureMent Inventory WAMMI)
• Before the test, questionnaires and interviews outline should be printed out.
• In case of conducting interview, a voice / video recorder is recommended

Supplementary Method 3: Eye-Tracking

• Eye-tracking equipment to be built into the computer monitor.

Step-by-Step Instructions

1. Build a prototype or a product to test
   1. Users should be able to test on real interfaces or products. Therefore, to conduct a user-test, it is important that the conceptualization of the product is done, main functionalities are already available in the prototype and ready to test.
2. Determine objectives
   1. Specific objectives of the user-test should be clearly defined. What the team want to achieve in the user-test. This step is a foundation for the up-coming steps in the process.
3. Determine participants
   1. The group of participants in the test need to correspond to the real users of the interface or product. Based on the objectives, the users can be further divided into subgroups, to discover different aspects of different groups of users.
   2. The number of participants should also be defined in this step, based on objective, the characteristics of the products and the supplementary methods that are applied in the process. Mostly with 5 users, the result is good and reliable. Some suggestions for some specific supplementary methods are:
      1. Quantitative studies: at least 20 to get statistically significant numbers
      2. Eye-tracking: at least 39 users to get stable heatmaps
4. Choosing the tasks for users to perform at the workshop
   1. It is recommended to define scenarios and the tasks that users need to perform are included in the scenarios. The tasks should represent the objectives of the User-Test that was defined in the earlier step.
   2. The scenarios and the tasks should be short, to the point, clear and easy to understand, so that users will not face any confusion during the workshop.
5. Document the test (at the workshop)
6. At the workshop, these information should be explained:
   1. The space set-up, e.g. group working station, camera setup, voice recorder
   2. Introduce the team who will support users during workshop, e.g. facilitator, co-facilitators, notetake
   3. Introduce the product that is evaluated
   4. Explain the whole process of evaluation, e.g. which supplementary methods are used at the workshop, in what purpose the data will be used for, whether or not there are questionnaires to send out after the workshop.
   5. Users will be divided into groups. Each group may perform different or the same tasks. at least one co-facilitator to be assigned to support every group.
   6. The test should be documented with the agreement from users.
7. Data analysis
   1. All data needs to be analysed using a certain method or tool, such as thematic analysis, Excel sheet. In this step, the team should be able to define some trends, patterns of problems or opinions, and some more problems might appear in the analysis.
   2. The team should also try to find out the reasons for the main findings of the analysis. This would help the team to integrate the users’ feedback into the design and improve the user experience.

Remarks, Tips, Limitations

Because many supplementary testing methods are available, which can be applied in the user-test, this is a flexible usability method. The test can be made comprehensively by employing multiple supplementary methods at once.
Which supplementary methods to be used and how many users to be selected should be decided based on several constraints, such as the timing, resources, budget, characteristics of the product and scope of the project.
Additionally this method is helpful to investigate further user’s needs, which have not been defined in earlier stages of the design-thinking process. A well run user-test would support the design team to provide better user experience.
Each supplementary method has its own advantages and disadvantages. Therefore, as mentioned above, to conduct a comprehensive use-test it is recommended to use these methods in combination and method users should not rely on only one single supplementary testing method.

• Thinking-Aloud
• Questionnaires & Interviews
• Eye-Tracking

References

Hasan, L. 2014. THE USEFULNESS OF USER TESTING METHODS IN IDENTIFYING PROBLEMS ON UNIVERSITY WEBSITES. JISTEM – Journal of Information Systems and Technology Management (Online) . 11, 2 (Aug. 2014), 229-256. DOI: https://doi.org/10.4301/10.4301%2FS1807-17752014000200002 .
Nielsen, J. 2012. “How Many Test Users in a Usability Study?”. Nielsen Norman Group. 2020. https://www.nngroup.com/articles/how-many-test-users/
Brooke, S. 2018. “How to Conduct a Usability Test in Six Steps from Start to Finish.” UX Planet. 2020 https://uxplanet.org/how-to-conduct-a-usability-test-in-six-steps-from-start-to-finish-4082e8d57858

Contributed by Nguyen Viet Thinh Pham.