Guide to Remote Research

This guide aims to define the tools, techniques and considerations that Airteam employs for remote research. It is intended to provide experienced UX practitioners with a high-level list of activities and tips to aid them in their switch to remote work. It is a living document that will be updated over time.

The COVID-19 virus has caused governments to put lockdown measures in place and people are staying home in an attempt to “flatten the curve”. So, you may be asking what impact this will have on your team’s user research?

It would be misguided to postpone research due to the barriers that social distancing has caused. In fact, now it’s even more necessary to do good research to uncover insights that will help products to survive and be of real value during this uncertain time.

With so many powerful tools out there, as well as a myriad of qualitative and quantitative methods that can be executed easily over the internet, we can continue to research remotely in the confidence that we’re still obtaining pertinent and valuable insights.

Remote research can be as effective as in-person testing and even has some key advantages

Greater Reach and Diversity
Remote research allows us to connect with a wider pool of participants and reach others that can't get to a physical lab. It’s common for clients to request in-person sessions that they can observe in a city lab. But it’s important that teams don't fall into the trap of only thinking about their Metropolitan users - and better to diversify and expose them to people in other geographies who may challenge their assumptions.

Natural Environment
User interviews are conducted in participant’s usual surroundings (at home, work, or on the go) on their own hardware. When it comes to testing mobile apps and products, we gain valuable context by observing users in their natural environment, from the dog barking to all those other distractions that make up real life.

More Efficient
Remote studies are more efficient for the researcher with reduced costs and travel time. When you’re working with an agile team, remote research allows you to fit many user sessions into a single day and iterate quickly.

Remote research methods

Most research activities can be switched to remote using creativity and a variety of online tools: from surveys and video interviews to moderated or unmoderated user feedback and testing.

Admittedly Ethnographic research, Contextual Inquiry or ‘Shadowing’ is hard to perform remotely, but we can attempt to gain similar insights into how someone lives or works by asking people to record video diaries or provide guided tours* remotely, as long as we safeguard their privacy.

*A guided tour is a field method by nature where a researcher gets immersed into the participant’s environment.

So, when can we conduct remote research?

The short answer is any time. The most successful team’s kick-off a project with research and never stop listening for information to help them improve.


Nielsen Norman Group charts user research by phases of a product design cycle.

Research methods can vary by phase. We may start out with exploratory research to understand our users, and we can use participatory methods to help us define and design solutions. We test concepts early for feedback and continue this cadence of validation after every design iteration through to delivery and beyond release.

After a product goes live, we can monitor audience behaviour quantitatively, and test variants to optimise the experience incrementally. However, we should continue to listen to users via qualitative means and other customer feedback mechanisms because it can provide deeper insights that are otherwise missed in quantitative testing.

Note: As products grow so does the amount of research data. We use Dovetail as our UX research repository so that no studies get lost or (worse) repeated.

Any tools mentioned below are listed in full (with URLs) at the end of this document.


Discovery

Before rushing in to build a new product (or feature) it’s important to understand your user’s current experience in addition to the business’ internal goals. You can conduct ‘generative interviews’ with participants via video conference to gain an understanding of user needs and desires or collect diaries online to understand longer-term behaviour.

Observing and discussing how people currently solve a problem or perform tasks, helps teams to validate or dismiss their assumptions and prevents wasted effort from building the ‘wrong thing’ or chasing the ‘wrong problem’.

Sample activities

User interviews
Encourage users to share their screen and artefacts if it helps to show their current process [Zoom business is better for recording and transcription]

Diary studies
For a topic or line of inquiry that benefits from being seen over time. For example, you could learn about someone’s recycling habits by seeing how/if/when they recycle over a month. [Obvi for text and MMS messaging, or, Dscout for rich media platform]

Competitive testing
If you can identify direct or indirect competitors it is invaluable to have users evaluate them. It will help ensure that the teams’ solution is at least at parity with or exceeds competitors’ offerings.

Stakeholder interviews
Spend time interviewing stakeholders to understand their requirements (and challenges). It’s important to land on the intersection of user AND business goals.
[Default to client’s video conference tool]

Sales and Support call monitoring
It can be hard to ‘shadow’ via a remote medium, so request aggregate data from users instead, i.e. call drivers and FAQs.

Surveys
Can be used at any stage to collect data using a range of question types.

Online focus groups
Use sparingly to get a broad, first pass at the audience you seek to serve.

Tips

Do a pilot test or ‘dry run’ so you can adjust technology and other factors as needed.

Consent and privacy
Gain consent and document digitally. Document participants’ consent to record the session as well as NDA (Non-disclosure agreement) as part of the recruitment process. Remind them of screen shots at the start of a session and wipe URL afterwards.

Confirm people’s technology in advance and provide any support. Allow time for technical challenges at the beginning of each session (10-15 minutes). If you experience choppy WIFI connection recommend plugging into the router with a cable instead.

Consider asking participants to do a short pre-session homework assignment to help break the ice and build rapport (if relevant).

Sessions will be disrupted sometimes by virtue of being in a home environment. Make it clear upfront that this is okay and to let you know if they need a break. Allocate more time than you would for an in-person meeting.

Build in extra time than usual between sessions to allow a buffer if a session runs over.

Recruit more participants than you need - especially for a diary study (some drop off over a long study, so you finish with the sample size you need).


Exploration

When you want to explore the problem space and understand how to address user needs appropriately, you can conduct ‘participatory research’ remotely.

Sample activities

Journey Mapping - interview participants who have solid knowledge about an experience and invite them to co-create a journey map. If it’s difficult to invite end users you can conduct this with frontline staff.
[Miro]

Card Sorting – Remote card sorting has many benefits. For one, it’s affordable, convenient and easy to set up. It also helps you to better understand peoples’ expectations and how they group information, as well as identifying terminology that could be misunderstood and information that might be difficult to categorize. [Optimal Workshop]

Tree Testing – helps you evaluate the findability of a proposed or existing IA. Tree tests are run on text-based versions of websites, without the influence of design elements, similar to a site map. [Optimal Workshop]

‘Desktop’ research – can be conducted by you and your peers using Google sheets.

Competitor Analysis - involves assessing competitor sites to see how they design for their users, potentially solving for similar needs. If you're new to working with a client, it is good for giving you the market context that the company operates in.

Heuristic Evaluation - Heuristic evaluation involves having at least 3 evaluators examine the interface and judge its compliance with recognized usability principles (the "heuristics").

Stakeholder Workshops and Co-design - See separate ‘Guide to remote workshops’
[Most workshops can be conducted easily using Miro]

Tips

Provide clear instructions and expectations upfront because it will probably take longer than less complex studies (e.g. journey map timeline, number of touchpoints, tools to use, types of tasks)

Before you launch a card sort study, test it by sending the shareable link to a small group of peers. Encourage feedback on the terminology used and find out if there are any areas where they got confused or misunderstood the content of the cards or categories.


Validation and Testing

To gather user feedback on a concept or prototype, or when you want to test a live website or app.

Moderated
Use Lookback especially for mobile because it records the screen of your app, along with the face and voice of the user, and all their taps and touches. Observers can join sessions, chat privately with the researcher and mark moments of interest on the video timeline to go back to later. Lookback offers the choice of moderated or unmoderated tests. Consider your needs as both have their advantages. Moderated enables deeper discussion with participants, whereas unmoderated allows you to run tests with more people.

Unmoderated (also known as asynchronous / self-test)
Usability Hub is good for navigation, first-click and preference tests – along with survey questions.

Loop 11 is good for task analysis whereby a browser bar prompts users to perform a task. Automated analysis includes task completion rate, time on task, first click, click stream, most frequent success and fail points.

Sample activities

Concept design & prototype feedback
[Lookback or Loop 11)

Qualitative Usability Testing – for testing functional versions or prototypes of your product. [Lookback or Askable Live]

Unmoderated Usability Testing / Task Analysis
[Usability Hub]

First-click - record the first click someone makes to accomplish a task

Preference - Preference tests can help you to choose between design variations, by simply asking users which one they prefer. They are commonly used to measure aesthetic appeal, but participants can also be instructed to judge designs based on their trustworthiness, or how well they communicate a specific message or idea.

Navigation – Navigation tests analyse how users navigate through your website or app. The user is presented with your interface and asked to complete a particular task. At each step, the position and timing of their clicks are recorded. Valuable when used to compare the performance of an original design to an improved version.

Tips

We test in order to iterate designs. Watch how people use them, rather than gather opinions. Note how well designs work to help people complete tasks and avoid errors. Let people show you where the problem areas are, then redesign and test again.

Ensure you have a strategy to keep prototypes secure and protect sensitive information.

Screen participants to ensure they meet the minimum device or OS version needed for remote sessions using your selected software.

Prepare participants in advance by sharing software to download prior to the session and expectations for them to turn on their webcam.

Perform a quick tech check 2 days ahead of the session to check people’s technology.


Ongoing Listening (Post-Launch)

After you’ve launched a product or feature, there are a variety of ways to keep the conversation going with users, like surveys, web analytics, and reviewing feedback channels. Try to coordinate with customer-facing and analytics teams to get the data you need to understand how the product is performing in real life.

Surveys
Consider intercepting users who are right in the middle of performing the task you’re interested in on your site in a process called Time-Aware Research. Users are interested in what they’re doing because they’re doing it for their own reasons, not because you’re asking them to. [Ethnio]

Analytics review [Google Analytics]

Heat and touch mapping [Crazy Egg]

FAQ and feedback review [Sales and Support teams]

A/B and multivariate testing [Optimizely]


Extra considerations for ‘going remote’

Be considerate of participants' changing circumstances and help them to feel comfortable. As researchers, we try to be objective but it’s okay to show empathy at this time. Also, you may have done this a lot, but it may be their first time participating remotely so take time at the start to explain things and reassure you’ll keep to time.

Recruit additional users for a remote interview or usability test because you can’t help if it is rendered useless due to unsolvable surprise technology issues on the user’s end, such as firewalls. Recruit more users than you think you need as a safety net.

Go slow and speak clearly as audio/video can produce a millisecond lag.

Have a protocol for technology hiccups. Ask them to raise their hand if they lose audio, or interrupt if they cannot see video or screen sharing.

Have a backup plan. Have an alternate video conference tool (Google hangouts) handy in case of issues along with participants phone numbers to SMS or call.

Let participants know it’s okay to quit if they are struggling with the tech and know when to shut down an unsuccessful session.

Protect yourself. If you’re switching on your camera or sharing screens, ensure there isn’t any information visible that displays your address or other sensitive information.

Replicate the focus of in-person sessions by asking people to switch off any notifications that may interrupt them (slack, email, phone).

Charge your phone beforehand in case you need to tether during the interview.


Observers

Remote research allows a large number of observers to watch sessions from their own computers. When teams get involved with research, they become more invested in the outcome and are more likely to take action based on the results.

Since you can’t pop into the ‘backroom’ organically when remote, plan a check-point with observers during the session. Decide in advance a way for them to communicate with you (Slack or other chat).

If using video conferencing or Miro ask observers to hide themselves and create a separate collaboration board for notes and observations. Ask them to put markers on video for key insights in order to export clips later.

If there are many observers, consider sharing note-taking responsibilities so each person is responsible for catching insights around a certain topic or task-flow.

Ask everyone to add context to notes or clips and it will save time later in analysis, e.g. tag all direct quotes “#quote” and it will allow you to quickly pull one for a presentation. Another example is #positive or #negative to assign sentiment to an observation.

An end of day video conference debrief is good for the team to exchange observations and themes will start to surface that will help your analysis.


Our favourite tools

Research

Analysis

Administration


Planning

Use our usual recruiters to source participants or try the Askable platform to hand-pick respondent’s yourself (bear in mind it is cheaper but far more labour intensive).

Other ways to recruit include social media, within the organisation (collect a user advocate panel) and via intercept on your live website or app.

Use this handy checklist from Askable to keep tests smooth and consistent.

Test plan checklist

  • Researchers’ names, roles and key activities
  • Purpose
  • Assumptions
  • Goals
  • Audience
  • Number of Participants
  • Session Length
  • Session Times
  • Technology (software and hardware)
  • Time zone conversions (for interstate)
  • Pre-session communications
  • Tech check plan
  • Welcome script (ice breaker)
  • Questions or tasks
  • Instructions for what’s next (incentive payments, follow up tasks)
  • Thank you script

Thanks for reading

This is a living document that Airteam have written to guide our User Researchers during a time when they are exclusively remote. We welcome feedback from our fellow UX research community, and we are here to field your questions.

Who is Airteam?

Airteam are a Sydney-based group of experienced researchers, designers and developers. We bring a human-centric approach to crafting digital customer experiences using the most cutting-edge technologies.

We believe that user research is the linchpin to great experiences. Talk to us about how we can tailor a study to answer your team’s questions and provide the clear, actionable recommendations you need for your product to succeed.