Preparing a Usability test

Our usability test is now planned, the script has been created and tested, and we have started to recruit participants. At this time, French chefs religiously begin the mise en place – they methodically clean, cut, weigh and organise all of the recipe’s ingredients before the pan goes to the stove. We as well, before moving on to the testing sessions with users, must carefully take care of all logistics. Much like in cooking, this step aims to:

1. make us more efficient;
2. help us to foresee any problems.

And, similar to cuisine, in usability tests this is also the most tedious (although critical) phase:

• to ensure that the testing sessions run smoothly;
• and that we get to the end of the day with many, and above all reliable and relevant findings about what is wrong with our product or service (and clues about how we can improve it).

Preparing a usability test may entail two or three precautions which, taken together, may take less than half an hour, or may take several days of meticulous chores – depending on the complexity and sophistication of our test. A test that may be confused with a simple, lively conversation in the corner café with a childhood friend. Or that, on the other end of the spectrum, may involve a complex, confidential financial system, laboratory testing, cameras, the collection of metrics and a translation into English, in real time, for a customer participating in the test from the armchair of their office in New York. 

Whether a remote test, laboratory study or guerilla test, we can group the preparations that we have become accustomed to doing here at Tangível over the years into eight major concerns:

1. Creating, revising and distributing documents;
2. Dealing with compensation;
3. Ensuring the conditions of the test location;
4. Preparing the testing equipment;
5. Setting up the communication, recording and observation system;
6. Guiding the people involved in the test;
7. Setting up the system to be tested;
8. Having water, coffee and snacks.

1. Creating, revising and distributing documents

In simple tests, we are normally talking about the test script – which itemises the tasks to be performed by the user. Sometimes, we also have a final questionnaire, such as the system usability scale, to gauge the satisfaction and perceived utility of the product, service or process tested.

However, more elaborate tests may also include the following.

• A moderator script – contains the initial explanation to the user, steps to be remembered (e.g. stop recording) and questions for the user at the end of the session.
• The tasks – a card (e.g. A3 paper) for each task in the script, which is given to the participant to be read aloud before starting the task.
• Non-disclosure agreement – common in projects that are innovative from the standpoint of business competition.
• Confirmation of receipt of compensation – to ensure we have not forgotten to compensate participants, and that participants do not forget that they have been compensated ;).
• Declaration of consent – authorisation to record and collect personal data in accordance with the GDPR. As Davis Travis explains, resist the temptation to include the non-disclosure agreement and confirmation of receipt of compensation in this document. 
• Perception of success and difficulty – two metrics of usability: at the end of each task, the participant evaluates the difficulty experienced in performing the task and indicates if they believe that the task was completed.
• Observer script – useful for the standard collection of metrics, such as the number of mistakes made, the success or failure in performing the task and the user’s difficulty while performing it.

In some tests, there may also be physical or digital artifacts used in performing the tasks, which we must also safeguard. For example: a task may involve consulting a hardcopy invoice, copying an IBAN from a bank passbook or reading a letter received in the mail. Using artifacts can help us create more realistic scenarios; it is different for the user to make up any old nine-digit number for a field requesting their Citizen’s Card than to go to their wallet, take out the card and enter the actual numbers while trying to read them (they are definitely not the most legible numbers on the planet). 

Some recommendations:

• Always ask a co-worker to review – the standard copy/paste of old projects is great for speed, but bad for typos.
• It is preferable to have printed documents. They are always much easier to consult and read, both for us and for participants.
• Always print extra copies.
• Print double copies of declarations of consent and other documents where the participant retains one copy.
• Print tasks and post-task questionnaires, one per page, ideally on A3 paper.
• Use large font, especially for the script and tasks.
• Avoid printing on both sides, and number and staple pages.
• Make a cover for subsequent filing, for example, of declarations of consent.

To prepare some of these documents, a good starting point are the templates (in English) available at usability.gov.

Remote tests require several adjustments to the documents. For example: instead of being printed, tasks can be in Google Docs – one per document – and, during the test, we can send the link to the respective document/task to the user, for instance via chat.

Finally, tests for certain groups such as minors, people with limited visibility, people with weak literacy, etc. require extra care as regards documentation. Since this is not really my thing, I refer simply to:

• the need for parental consent in tests with children and adolescents,
• and that best practices for consent from visually impaired people include documents in digital format, optimised to be read in screen readers, digital signatures, impartial witnesses who can assist in the process, audio recording of consent, etc.

2. Dealing with compensation

It goes without saying that we should always compensate someone for their time spent with us, oftentimes trying to do unusual things and answering questions with no apparent meaning.

As such, we should take care of compensation in a timely manner, whether in the form of traditional gift cards or, possibly, a fine bottle of red wine.

Tips for gift cards

• Choose cards with a wide range of options for use, such as Sonae Group cards.
• Request an individual receipt for each card at the time of purchase, to serve as proof.
• Attach a contact card to the card’s envelope: it may be useful to the participant if problems arise at the time of using the card.

Supplementing the gift card with a small piece of chocolate or other souvenir is also good. At Tangível, our tradition is to give away a mug: “I participated in a usability test.”

Depending on the type of project and who the participants are, compensation can vary.

• For family and friends, things that we know are valued by the participant, such as a cookbook, for example, work better than gift cards (and are less expensive).
• For people with highly reputable professions in society, such as physicians or lecturers, gift cards usually don’t go over so well…. Choose something more refined and exclusive that demonstrates care, such as a fine wine, a new Fitbit or other modern gadget, or a voucher for a romantic hotel.
• If the compensation can be tied to the study project, even better! If we are testing a ticket purchasing system, give away tickets to a football game. In the case of an e-commerce website, give a voucher for products from the website. Be creative.
• In short or guerrilla tests, consider compensating people with a snack, a round of beers, lunch, etc.
• Just like in chapter 1, if the participants are visually impaired, for example, a physical gift card may not be the most accessible type of compensation. If we are testing with children, we should offer an incentive to their parents, and another to the children who will be surprising us over the next half hour.    

We want people to feel respected, to be motivated, but not feel “bought” or influenced, so that they remain ready and willing, but not anxious, to participate in future studies.

3. Ensure the conditions of the test location

Whether in a café or a meeting room… look for a place that is quiet, comfortable and well lit, with a stable Internet connection and power sources. As a precaution, always have an outlet strip in your bag and a good Internet tariff in your mobile phone.

Some precautions to consider for meeting rooms:

• If a reservation is needed for the room, reserve extra time: one half hour before and after each session.
• If there is more than one test on the same day, reserve the room for the entire day: for complex usability tests, keep the rooms from being used by other people between testing sessions.
• Notify co-workers in advance so that the tests are not interrupted. A sign along the lines of – “Tests in progress. Please do not enter!” – hanging on the doorknob of the meeting room also helps.

Internet: make sure that all participants have access.

To facilitate connecting to the Internet, have very clear, visible instructions on how people can connect at the location.

Tests at customer facilities require extra care. Sometimes, there are strict criteria for Internet access or time-consuming procedures for installing recording software, or the interface only works on the intranet using a company desktop or an employee login, etc.

It is essential to visit the testing location the day before in order to:

• Bring the compensation, equipment, documents, etc.;
• Do a pilot test in the room;
• Get everything ready for the first test.

Prepare the room the day before. Example: put the test computer on the table (to the right of the moderator is usually best), together with a testing script, declaration of consent, water and compensation.

If the test is on a laptop, always bring a mouse for the user. If it is on a mobile phone, leave the phone’s document camera set up for recording.

Mark, for instance using tape, the location where the user should leave the mobile phone standing while performing the tasks from the script.

If there is a separate location for observers, try to create a layout in the testing room to allow easy observation. Just like the testing room, do not forget to prepare the observation room as well.

 

4. Testing equipment

In a word: guarantee the availability and operability of the computers, mobile phones and other devices that will be used in the test.

When using our own personal equipment or test machines from our work location:

• Choose everyday equipment – avoid top-of-the-line devices such as the latest iPhone or 28-inch full HD monitors;
• Avoid the temptation of using a beloved MAC, by choosing machines with operating systems familiar to participants – in Portugal, this typically means Windows and Android devices; 
• Reserve equipment in advance;
• Check for necessary updates;
• Make sure they have the right software;
• Clear sessions, cookies, histories, files and inappropriate photographs from the work environment, downloads folder, etc.

Since the people in this case are not working from their own device, if they need to log in for the test, notify them in advance that they will need to know or bring their respective access data.

Ideally, however, the test should be done on the participant’s device. In such case, you should know about their equipment and its properties beforehand:

1. to determine whether the test can be executed (e.g. if we are testing contactless payments, it is good to know whether the mobile phone has NFC);
2. and to ensure we are not doing all of the tests only in iOS, for example.

5. Setting up the communication, recording and observation system

Generally, the audio, the participant’s face and the screen or prototype where the interaction occurs are recorded. For this purpose, Morae has been the preferred software here at Tangível (although we are thinking about changing and are open to suggestions), both for tests on PCs and on smartphones.

For tests on smartphones, we record the interaction with an IPEVO document camera and the face and voice of the participant using a laptop with Morae.

To send images and sound to the observation room, or to remote observers, we have used the popular platform Zoom, although a simple Skype call with screen sharing works for simpler scenarios.

There are other alternatives, some more and some less sophisticated. In any case, the important part of this point 5 is the idea that setting up and configuring recording software and hardware are tasks that must be done in advance, which can take time. And, if there is something critical which must be tested meticulously, it is this.

In remote testing, on the day before the test, it is important to do a video call with each of the participants, for example to check the Internet quality for screen sharing + video call. These and other guidelines on how to prepare and conduct remote tests are detailed in this full article of NNg.  

6. Guide people participating in the sessions

There should be a Murphy’s Law for the number of people at an event and the likelihood of something going wrong… and, since a round of usability tests can bring many people together, it is important for all of them to be on the same page to avoid problems involving communication and expectations.

In the days preceding the test, it makes a difference to contact participants:

• to remind them of the date, time and duration;
• to tell them that the session will be recorded, if applicable;
• to give them a phone number in the case of unforeseen circumstances;
• to notify them again that they will need to bring or have with them the following things for the study, e.g.: a personal bill or a health insurance card with login information to access the customer area.
• to tell them how to get to the location, enter the building and who to ask for in the case of in-person tests;
• to emphasise the importance, in a remote test, of being in a quiet location with good lighting, where they will not be interrupted, preferably with a headset including a microphone (e.g. mobile phone headset), and a good Internet connection. 

Customers (programming, product, marketing, business, compliance, etc.) are always welcome. However, they should be brought up-to-date beforehand on how the session will proceed and what is expected of them. Having a printed guide with best observation practices to assist them during the sessions also helps. If they are attending remotely, remind them that they must absolutely have their microphone turned off during the entire session (using video call software is best, since it allows us to do this on our end).

We must also ensure that our team’s moderators and observers study the script and interface in detail before the sessions. And, to assist during the sessions, it is good for us to print a testing schedule, enhanced with the most relevant data of each participant.

Without dwelling on the subject, there are also many other people who can be involved in a test, and who will need guidance prior to the sessions. Translators, who are common in international projects, are one example. Other examples are the owners and employees of the café, library staff, etc. in guerrilla usability testing.

7. Setting up the system to be tested

The tasks under this point are highly dependent upon the nature of the test (e.g. task simulation, Wizard of Oz, an actual complete purchase) and the system in question (e.g. a prototype on paper, a desktop app in production).

In this phase, for example, the following things are normally needed:

• adjusting mockups to the tasks of the script and, if possible, to the participant in question (small details like the name can already help);
• placing prototypes in the testing machines and installing applications in the equipment of the participant.

However, over the years, we have also had to do things such as: create audio clips to simulate answers from virtual assistants and program scripts to load the test configurations into the database of the application.

8. Have water, coffee and snacks available for people.

Here is a fun study: there are two groups of people, and each group has a task to do together. Before getting to work, in one of the groups, the participants are invited to have tea, passing the cups among themselves. Which group do you think will have the most cooperative people?

In practical terms for the world of usability tests, be sure to have the following:

• Water in the meeting room. Think aloud protocol: both the user and the moderator will be doing a lot of speaking.
• The option for participants to have a coffee and, if possible, tea. If it is a new location, bring a coffee machine and coffee capsules, just to be sure. Few people actually have coffee, but we have had participants who have asked for it, even when we did not offer.
• Fruit and snacks in case people get hungry and need to recharge their batteries. Sweets work well. At Tangível, palmiers are the tradition, but these days, more vegan, light and gluten-free options should not be overlooked. A small bowl of mixed fruit is always easy and goes over well.

Coffee, fruit and snacks in a corner of the usability study room.

In summary…

Create documents. Take care with the meeting room. Arrange for compensation. Test equipment. Record and observe. Set up the interface. Guide people. Food and beverages.

To remember all of the above points of this (potentially large) set of tasks, there is nothing better than making a checklist. Even better: adapting the checklist from a past usability study.

Preparing usability tests is an invisible, boring and inglorious activity. But disregarding it is opening the doors to the possibility, in the middle of the session, out of nowhere, that a task may become completely infeasible – this is speaking from the experience of moderating tests that were prepared off-the-cuff. In the wise words of H. K. Williams:

“If you fail to prepare, you are preparing to fail.”

A careful preparation results in more motivated participants, more focused teams, more engaged customers and more relevant and detailed findings. The reward is worth more than the effort.

Good luck testing!

P.S.: This is a summary of what has been, essentially, “learning by doing” here at the company. Much of what I have written here was learned with my friend Filipe. There will certainly be some missing information. Different things, for example, in remote testing; successful improvisations in guerrilla testing. Feel free to share with us what you do while preparing for your usability test.