User Interviews & Testing

How to conduct user interviews and tests online.

Since 2020 we have been conducting remote, online, user tests and interviews. We have found a pretty flexible way to perform that testing. Below you can find a suggestion of how to conduct those user interviews and tests, but make sure you adjust the tools according to your needs!

Template

98KB

User Testing Template.xlsx

Prepare & Execute

Scope & Overview [scope-interview]

scope Decide with your team members what you are going to test and why. This will dictate what questions you'll ask, what type of test you'll conduct and who you will contact.

• What

  • Which assumptions do you know you have, and which ones do you want to test?

  • What answers are you looking for?

  • What do you know you don't know yet?

• Who

  • Who can answer these questions for you? Who will work with the application? Who do you need to convince to get the entire team to work with your app?

• How

  • How you're going to test is going to depend on what you are testing. If you are testing raw ideas, you'll probably share wireframes (via screen sharing for example). If you are testing an application, you'll need a dev environment ready and you'll probably want people to share their screen with you.

• Why

  • What do you want to do with these answers afterwards? Do you want to make sure you know what new features to develop, or which ones to improve? Or would you want to set up webinars afterwards to get people to use it? Many options.

Tab one: Scope & Overview

Overview

inhoud test Define the structure of your meeting.

• Will it have an interview part? Will you send the questions up front?

• Will you test with wireframes, mockups, a prototype or something else?

• Which parts are there to your test?

• Conclusions

  • For drawing conclusions overall, after each session, we already add conclusion tabs to the template, per interview- and test section. These conclusion tabs you can already prepare up front; because they help you define up front what you want to know. Sometimes you need multiple questions to gather one insight.

Timings

timing Allocate time to the different parts of your test and interview. Allocate the most time to what is important. Make sure you have some time to run out; you don't want your meeting to be stuffed to the brim. Things will go wrong.

No clue how to time it? Do an internal pre-test with some colleagues to give yourself a better idea.

Attendees [deelnemers-locatie]

Interviewees

name, type of participant , organisation , type of organisation, contact person , tag

Get an overview of who you need to talk to to get the relevant information for your project – and what steps you can undertake if you cannot find those people right away.

Usually the product manager sets up the meetings via personal contacts, or partnerships that are relevant to the product or via the client (VVSG – Vereniging van Vlaamse Steden en Gemeenten, Inter, Klankbordgroep Gent).

Fill out their name, type of participant (think of which type of people you need to see to get a good grasp on your answers, broad and diverse enough), organisation and type of organisation(also relevant for your end results) and who the contact person is (probably your product manager; this is helpful during the introduction or when someone does not show up). The tag can help you to anonymise information later on, but is optional.

Planning the meetings

date, time, location

• Before anyone starts contacting interviewees, make sure you block out enough placeholder time ahead with the appointed team members; so the product manager can align agenda's with the interviewees.

• Make sure you have a bit more time allocated to the meeting than you planned for in the interview / test (because things will go wrong).

• Location: make sure everyone knows up front where the meeting will take place. Add them to the document to make sure you can switch easily.

Interviewers & Notes

interviewer, notulist, specialist

• Appoint an interviewer, and someone that takes notes (notulist). If they switch, it'll be different every session; just make sure everyone knows up front.

• Sometimes it's helpful to have a specialist on board (like a jurist) to answer specific questions about the topic you're dealing with. Try answering these questions at the end of the session, because it's easy to get lost in details and go off-topic.

More than 2 people can be overwhelming to the test subject, so find out what's the best for the outcome.

Consider turning off camera's for people that are not taking the interview, but do involve everyone in the introduction (smile & wave) and when saying goodbye.

Setting the Scene [introductie]

The goal of the introduction is to make people feel at home and set expectations.

Introduce everyone

The interviewer takes te lead and lets everyone (including the notetaker, specialist, ...) introduce themselves briefly. Mention the contact person (see attendees) if needed to make sure everyone knows why they are here.

A virtual round of the table is hard; you don't know who's next to you. The interviewer calls out everyone one by one to introduce themselves.

Set up expectations

The interviewer goes over the setup.

• What can they expect, what do you expect

  • Explain what is going to happen and why; what you want people to focus on.

  • Explain the structure of the meeting before they start – so they know what's coming.

  • GDPR: Explain what happens to the notes, and what their rights are.

• Sometimes interviewees think they have to elaborate, do things differently than usual or "do a good job" (most of us want to!) . Try to debunk that.

  • If something goes wrong, that's excellent.

    • Make sure everyone knows it's okay to make mistakes, be confused and have questions – and that it is valuable information. We are testing the application (or service or idea or...), not their skillset.

  • Honesty is key to get value out of these sessions.

    • The interviewees should know that they can't insult (or compliment!) anyone in the room.

  • Their lead

    • The interviewees don't need to be extra elaborate or extra fast – they can just act the way they would as if they're at home or in the office.

  • Think out loud

    • Urge your test subjects to think out loud (and show their frustration) to help the interviewer and notetaker understand what is going on.

• Leave room for questions!

Interview

To understand the feedback you're getting from your interviewees, you need some context. The interviewer and the interviewee are mostly speaking. As little intervention as possible from is expected others. If the notetaker missed something, it's okay to speak up.

General tips:

• Ask targeted questions if there are multiple people in the room, don't leave it open-ended (in a remote situation this usually causes silence)

• Only ask one question at a time.

  • This is easier to answer

  • You'll most likely get an answer

  • It's easier to note down.

Questions

Think of what you need to know, and what to leave out. This depends from test to test.

Some examples:

Demography & Job – ask about their function, and what they do on a daily base. This can frame how often they would be interact with your product (or service or..).

Digital skills – if you have little knowledge about your interviewees, it will be helpful to know what their digital skillset is. If they spend a lot of time using their computer during their job, chances are high they'll onboard your application much quicker than someone that only uses their ipad during the weekend; and they might need more guidance.

Applications – if they use similar or other applications to complete the job you're interested in, you know which patterns they might be used to or what quality they expect.

Previous use - it's possible people have already used your application. You might want to figure out what they think about your application now; and see if it's different after they've seen your new features or adapted features.

These questions will also help you to create archetypes (persona's) later on, that can guide you when building features.

Filling out questions

• Add as many rows as needed to keep an overview.

• If you are interviewing multiple people at the same time (depending on your situation this can happen, even if it is not always ideal) add initials or tag of the person that is giving an answer or to the information you are writing down.

• If multiple people are editing, make sure you're not overriding what they just wrote down.

Timing

Decide when and how you're going to ask these questions:

• Up front (via mail)

  • Reduces meeting time; which everyone likes.

  • If it's a lot of questions, it can feel like an annoying assignment – chances of it getting completed get lower with every question. This is especially the case with busy people.

  • You can ask "why" or elaborate during the meeting.

• During the test

  • Elongates meeting time.

  • No "homework" feeling.

  • Easier to ask questions and go deeper when they are filling it out.

  • Can take long when doing this together.

Test

The interviewer and the interviewee are mostly speaking. As little intervention as possible from is expected others. If the notetaker missed something, it's okay to speak up.

General tips:

• Ask targeted questions if there are multiple people in the room, don't leave it open-ended (in a remote situation this usually causes silence)

• Only ask one question at a time.

  • This is easier to answer

  • You'll most likely get an answer

  • It's easier to note down.

Testing with a demo environment

• Make sure you have a separate, stabile test-environment – where anyone can safely add test-data. Warn the developers not to deploy when you are testing!

• Make sure people can share their screen if needed.

  • Have a back-up scenario available that allows you to to share your screen and perform the tasks they want to complete, just in case it doesn't work out. Make sure you don't steer or influence especially in this case.

Testing with wireframes or mock-ups, or with a local set-up on your machine

If you are testing raw ideas, you'll probably share wireframes (via screen sharing for example).

• Make sure people can see the relevant applications on your screen.

• Turn off notifications to not be disturbed.

• Make sure people don't have to see a cluttered background, irrelevant information or irrelevant tabs pass by.

• Take it easy. Don't make people motion-sick. Give them time to inspect.

Your testing questions will depend on which type of test it will be. The main setup is giving someone an assignment, see how they complete it, and ask them nudging questions to discover what is going on in their mind.

Type of assignments you can give

• When you have a clickable prototype:

  • "Create a new file"

  • "Fill out the form"

• When you have a concept in a wireframe or mock-up

  • "If you would want to create a new file, how would you go about it?"

Refrain from giving people answers. If they ask you "is it this button?", you can ask them to try it out, and discover it together. Assure them nothing can go wrong.

Filling out the fields

• Add as many rows as needed to keep an overview.

• If you are interviewing multiple people at the same time (depending on your situation this can happen, even if it is not always ideal) add initials or tag of the person that is completing the task/answering a question or to the information you are writing down.

• If multiple people are editing, make sure you're not overriding what they just wrote down.

• What to fill out

  • Opdracht gelukt Did they succeed?

    • Think of a system up front like "yes / no / almost"

    • This will help you when draw conclusions faster in the end

  • Omschrijving interactie How did it go?

    • Where did they get stuck?

    • What went well?

    • Add rows if needed

  • Opmerkingen If you have anything to add, like thoughts or suggestions (from anyone in the room), you can add it to the comments field.

Help people think out loud

To make sure you know what's happening inside people's minds, and why they make certain decisions, you can help them think out loud by asking questions:

• What do you see here?

• What can you do here?

• Did this button meet your expectations?

Guided test

The interviewer and the interviewee are mostly speaking. As little intervention as possible from is expected others. If the notetaker missed something, it's okay to speak up.

If you decide to do a guided test, and want to teach people how to work with your application (or service or..) more than discover what to improve, you can show them or help them complete a task. Then give them time to try it themselves. After a while you can ask them how it went.

• How did it go for you? Did it work out?

• If they don't give you a lot of details, you can ask what went well and what did not, and get into a conversation if needed.

Final questions

The holy grail

The one question that you probably want an answer to is "Would you use this application/service/...?" You can ask any variety on this, but this will probably give you the best insight.

Feedback

• Ask for feedback on your test as well, see if there is room for any improvement.

• This is also a great time to ask if you can contact them again in the future to be part of other tests or even enter your customer advisory board.

Follow up

Tell people what is going to happen.

• Will you send them an e-mail with follow-up? When?

• When will they be able to see something? Do you want to keep them up-to-date?

• Do you want them to participate in other tests?

  • Ask them if they want to!

Conclusions

When you're done with all the interviews, you can start drawing conclusions. This is will be different for each set-up. Up front, you can however define which conclusions you know you already want to draw – but you don't know what you don't know, so this will change.