The Customer Discovery Handbook


This customer discovery handbook draws on long-standing practices from design research and usability engineering. I think you’ll find it a useful complement to your work in ‘design thinking‘, ‘Lean Startup‘, ‘Customer Development’, and the related topic of ‘business model generation‘.

Discovery/design research spans everything a business does around making things and selling things. The first step to success is structuring your ideas. I’ve organized the balance of the handbook around the following five areas:

Customer-Discovery-Handbook-Summary

These areas are tightly related to the Venture Design process and I reference those tutorials, templates and workshops below.

Persona & Problem Hypothesis

lean-persona-hypothesis

lean-problem-hypothesisI’ve paired these under a single heading since most of the related activities are done together. The key questions you’re trying to answer for yourself are:

Who is my customer and do I understand what makes them tick?

Have I identified one or more problems that are important to them? 

What alternatives are my target personas using to deliver on the problem today?

The persona is a humanized view of your customer, be they buyer and/or user of your product. For background, see the tutorial here on Personas. By problem, I mean an underlying job or need that exists for your persona(s). Anchoring to the right problem (vs. presupposing a solution) is central to the practice of design/design research. The aforementioned tutorial describes personas & problem scenarios.

Your goal at the end of this is to have clear working answers to the above, supported by notes that are vivid, actionable, testable for you and your collaborators.

Preparation

Persona-Think-See-Feel-DoHigh quality observation learning offers the only good answers to the questions above. That said, I highly recommend pushing yourself to 1) brainstorm a set of personas and problem scenarios and then 2) pick the most important and draft them.

Why? I’ve found my students & advisees get the most mileage out of the mentally and emotionally taxing work of finding subjects and interviewing them when they have a focused view of what they want to know. This section of the Venture Design page has tutorials, examples, templates, and workshops you can use to draft your first set of personas & problem scenarios: Tools for Creating Personas.

Once you’re comfortable you know what you want to know, it’s time to move to execution.

Execution

Ready to go discover who and where you can innovate and deliver value? Here we go!

0. Know What You Want to Know

Before you go out and execute discovery, you’ll want to have a clear idea of what you want to know. See the previous section on this.

1. Create a Screener

You have specific personas you want to learn about. What simple, factual question can you ask a potential subject to be sure they’re relevant?

This is more important than it may seem. We have a natural bias to go with subjects that are convenient & comfortable, which can dramatically limit actionable learning. Don’t blame yourself, but do screen yourself!

For example, let’s say we’re interested in problem scenarios around some aspect of network management, with the idea of possibly building an application for network engineers to manage transport elements like routers and switches. We have a persona(s) for the end user that we want to develop and validate. A good screener would be: ‘How many times last week did you log into a switch or router?’.

2. Design an Interview Guide

Not a survey and not a script, this a tool for you to organize your questions across your objectives and available best practices, many of which you’ll learn and incorporate as you go.

Do not worry about sticking to the script every time. You won’t create a useful persona by attempting ‘statistically significant results’.

Do worry about covering your bases on the persona and problem scenarios. Do worry about updating the guide as you learn what’s working for you and isn’t (yes, it’s OK to update the guide as you go along; it doesn’t invalidate the data).

Here’s an example of an interview guide for ‘Enable Quiz’, a startup-up that’s looking at problem scenarios around how HR managers screen technical recruits for engineering jobs.

PERSONA HYPOTHESIS

Question Format Example Questions (Enable Quiz)
Tell me about [yourself in the role of the persona]? Tell me about being an HR manager?
How did you choose that line of work? Why?
What do you most, least like about the job?
What are the hardest, easiest parts of the job?
I’ve heard [x]- does that apply to you?
Tell me about [your area of interest]? Do you do screen new candidates? If not, who?
Can you tell me about the last time? What was the trigger?
Who else was involved? What was it like?
Tell me your thoughts about [area]? Where do you learn what’s new? What others do?
Who do you think is doing it right?
How did you make your last decision?
What do you see in [area]? Where do you learn what’s new? What others do?
Who do you think is doing it right?
How did you make your last decision?
How do you feel about [area]? What motivates you? What parts of it are most rewarding? Why?
Tell me about the last time?
What would it be like in your perfect world?
What do you do in [area]? Would you show me your interview guide?
Example notes?
What the vetting process was like on the last few candidates?

PROBLEM HYPOTHESIS

Question Format Example Questions (Enable Quiz)
What are the top [5] hardest things about [area of interest]? What are the top 5 most difficult things about making good tech hires? Why?
How do you currently [operate in area of interest- if you don’t have that yet]? OR Here’s what I got on [x]- is that right? How do you currently screen for technical skill sets?Who does what?How does that work?
What’s [difficult, annoying] about [area of interest]? What’s difficult about screening technical candidates?How do you validate they have the right skill set?
How are the actual outcomes? Examples?
What are the top 5 things you want to do better this year in [general area of interest]? What are the top 5 things you want to do better in technical recruiting and hiring?
Why is/isn’t [your specific area of interest on that list]? Why is/isn’t screening for technical candidates on that list?

You may notice that the two parts of the interview guide overlap. That’s OK- particularly if you observe that you’re getting new and different responses between the different forms of the question. Subjects will remember things they wanted to tell you over the course of the interview and they will never understand all the questions exactly the way you intended (that’s just the limitations of human communication).

For a template with these items, see Interview Guide (Venture Design Template).

3. Find Subjects

Are you finding that getting to the right interview subjects is difficult, messy, and time-consuming? Good- you’re probably doing it right.

If I knew an easier way, dear reader, I would tell you straight away. The reality is that soliciting online and even using high-priced agencies tends to deliver atypical subjects, semi-professional participants that are not a reliable or useful source of actionable learning.

Find your subjects by hook or by crook and use the screener to make sure everyone’s investing their time appropriately- that’s the best way. Is your persona an oncology nurse? Call your uncle across the country who does the books at a hospital see who he knows. Half of those people will probably tell you never to call them again (that’s hyperbole). Take solace in the fact that quality learning at this stage will have geometric benefits downstream.

Should you compensate them? Yes, in some fashion. They’re generously giving up their time for your venture. If you’re a startup you probably can’t afford to compensate them at market rates. You should nevertheless present them some token of your appreciation (Starbucks gift card, etc.), acknowledging that it’s just a token and that you’ve appreciated their generosity with their time. Not only is it appropriate and polite, but it will make it much more likely that they help you find other subjects.

4. Interview Subjects

Never, ever, ever, pitch subjects your product, or even advocate a particular point of view. Your job is to learn something from the subject that will allow you to scale an offering to 100 then 1,000, then 10,000, etc. customers. It’s not to sell them something (or worse yet have them tell you what you want to hear).

That said, most subjects will not understand your approach or objective and that will make them uncomfortable. Being forthright with them about what you’re doing is key. Explain to them what you want to learn. That is your objective (not selling). For example, telling an HR manager ‘We’re trying to learn about what’s it’s like for HR managers to recruit technical talent.’ or telling a nurse ‘We’re trying to learn about what it’s like for an oncology nurse to file paperwork.’ is fine and useful.

Recording the session will make them uneasy at first, but most subjects will forget about it in 5 minutes or so. (If you do record them, be sure to abide by applicable laws and regulations.) That said, you’ll likely find you have little time to review these transcripts. Most important is to make notes as you go along and to sit down immediately after the interview and make notes on what you learned, ideas that came to mind.

Avoid interrupting the subject (this will dam up the natural flow of information) but balance that with the time you both have available and the topical coverage you want.

Freely update your interview guide as you learn what’s working. Don’t worry about it making the interviews non-comparable: your goal is not statistical significance. The reason I use a Google Doc for a template is that it eases ad hoc collaboration between team members.

4. Update Your Personas & Problem Scenarios

They’re never done. Not to harp on the Google Doc’s angle (and it’s certainly not the only place to put all this), but personas are the innovators spreadsheet. Everything changes constantly, and the successful innovators uses their personas (and problem scenarios, etc.) to keep track.

How do you know when you’re in relatively good shape? The closing section in this area offers a checklist.

Validation (Invalidation)

You’re never done, but the following are good indicators that for any given project it’s time to move on to the next step.

CHECKLIST: PERSONA HYPOTHESIS
Sub-Hypothesis Experiment

✔︎

This persona exists (in non-trivial numbers) and you can identify them. Can you think of 5-10 examples?
Can you set up discovery interviews with them?
Can you connect with them in the market at large?
✔︎ You understand this persona well. What kind of shoes do they wear?Are you hearing, seeing the same things across your discovery interviews?
✔︎ Do you understand what they Think in your area of interest? What do you they mention as important? Difficult? Rewarding?
Do they see the work (or habit) as you do?What would they like to do better? To be better?

✔︎

Do you understand what they See in your area of interest? Where do they get their information? Peers? Publications?
How do they decide what’s OK? What’s aspirational?

✔︎

How do they Feel about your area of interest? What are their triggers for this area? Motivations?
What rewards do they seek? How do they view past actions?

✔︎

Do you understand what they Do in your area of interest? What do you actually observe them doing?
How can you directly or indirectly validate that’s what they do?

CHECKLIST: PROBLEM HYPOTHESIS

Sub-Hypothesis Experiment

✔︎

You’ve identified at least one discrete problem (job, desire, etc.) Can you describe it in a sentence?
Do others get it?
Can you identify current alternatives?
✔︎ The problem is important Do subjects mention it unprompted in discovery interviews?
Do they respond to solicitation (see also value and customer creation hypotheses)?
✔︎ You understand current alternatives Have you seen them in action?
Do you have ‘artifacts’ (spreadsheets, photos, posts, notes, whiteboard scribbles, screen shots)?

Value Hypothesis

Yellow-Walkman-Featurelean-value-hypothesisYou can’t ask a customer whether they’d like a product you’re thinking of creating (or even that they may be thinking of buying). They’ll always say ‘sure’. For more on this see the Story of the Yellow Walkman.

This reality creates a stark boundary between the type of work you can do to enhance your Persona Hypothesis and Problem Hypothesis and the type of work that will help you test a Value Hypothesis.

For an actionable result on Value Hypothesis, you must incite a material exchange of value. If you ask a subject ‘Would you buy this yellow Walkman?’, they will say ‘Sure’. They want to be nice and they don’t want to argue you with you. But if you offer them either a black or a yellow walkman, one of which they actually get to take home, then you have a material exchange of value.

Here’s another example with a smaller absolute exchange of value but one which can still be material. If you spend an hour talking with a subject and then, standing over their shoulder, ask if they’d like to sign up for your email newsletter, that’s not a material exchange of value. The awkwardness of refusing far outweighs the cost of unsubscribing or just deleting the emails. But what if someone you’ve never met learns about your offer on the Internet and subscribes? Yes, then that’s a material exchange of value.

Preparation

I highly recommend having a working view of your Persona Hypothesis & Persona Hypothesis before you approach this area (see above). The outputs will give you what you need to create a working Product Hypothesis:

Product-Hypothesis

Investing time in understanding your focal personas and problem scenarios avoids excessive waste in formulating and testing your Value Hypothesis.

I also recommend a quick think on your ‘engines of growth’ (a term coined by Eric Ries). The proposition here is that there are three principal engines of growth and a new ventures knows which one is most important:

Viral– customers/users tell each other about the offer. Crucial here is some measurement of ‘viral coefficient’, the propensity of one customer/user to share in some fashion the offer with others. In this case, your ability to drive sharing/word-of-mouth is crucial.

Paid– you have a certain cost of customer acquisition based on the use of marketing and/or sales resources. Here, ascertaining the cost of acquisition and the value of a customer are key to understanding the validity of your unit economics.

Sticky– the lifetime value of customers is very high because the relationship will deepen over time. Here, testing your ability to retain and maximize the lifetime value of a customer relationship is key.

The validity of the fundamental Product Hypothesis and your point of view on a principal engine of growth are important focal items for the testing of a Value Hypothesis.

Execution

How do you engineer an exchange of value that tests your Product Hypothesis and (ideally) engine of growth without wasting money building someone that no one wants? That’s what the discipline of Lean Startup is about, specifically the use of an ‘MVP’.

The MVP is an execution that is viable to definitively test your Value Hypothesis with a minimum of resources. The ‘product’ term is a kind of necessary misnomer: if you can possibly help it, the point of Lean Startup is that you should avoid building a full-blown product.

This is a tricky, inexact science and it rubs against the grain of permanence and durability that runs strong in most of us. The good news is that there are several well established patters for creating an ‘MVP’, minimum viable product. Here are a few:

MVP Pattern Notes
The Wizard of OZ MVP Create a realistic demo recording/rendering of your product, post it online, drive some traffic to it, and see how you do on sign-up’s and sharing. This is a pattern that DropBox initially used.
The Google AdWords MVP This is a popular pattern for both new products and features- see if you can generate click-throughs and sign-up’s from a Google AdWords campaign. This is especially useful if your engine of growth has certain assumptions about paid CPA (cost-per-acquisition).
The Pre-Sales MVP There’s really nothing as definitive as getting customers to pre-pay (or even sign a letter-of-intent; it’s still an exchange of value) for your product, assuming the price point is viable. Make sure your pitch and the transaction are clear.
The Concierge MVP (Strong Form) Have the customer submit inputs on the front end and you manually execute whatever it is your product would do on the back end.We did this at a photo-social startup I was advising: rather that creating an app to do something (hypothetically) exciting with a set of photos, I challenged the team to execute hypothetically exciting action by hand. If the output was popular on social media, that’s a validation signal. If not, it isn’t.See also the Howe & Associates example on the case studies.
The Concierge MVP (Consulting)  Using consulting as a prelude to (hypothetically) building a piece of software is a great way to discover and validate all your hypothesis areas.Leonid Systems used consulting as a vehicle to solve customer problems while evaluating problem areas that looked ripe for automation and standardization with software.

These are all described in more detail on the Lean Startup tutorial, case studies section.

Validation (Invalidation)

The thing that separates a hypothesis from a notion is that it has a testable formation, one that allows a structured experiment to prove or disprove it. Your venture (whether it’s a startup or an internal project) is likely in a rush and starved for resources. I’ve been there.

It’s important to make a habit of thinking your MVP experiments all the way through, and you’ll probably find that once you get used to it, it really doesn’t add that much time on the front end. Before you invest in executing the MVP, think through how you’ll interpret the results for decision making.

What metrics (quantitative or qualitative) constitute validation? Invalidation? What will you do on the basis? (Generally, invest in scaling up the initiative if it validates and re-formulating+re-testing if it doesn’t). Even if it’s a simple email, write up the report you’ll issue on the experiment (with placeholders for results) beforehand.

Customer Creation Hypothesis

lean-customer-creation-hypothesisCOMING SOON

Usability Hypothesis

usability-hypothesis

Reusing existing best-practice interface models and maintaining a test-driven orientation are probably the best ways to consistently succeed on usability. The usability of any interface change is unknown, even with the best designers. Treating each change as hypothetical  avoids waste and expensive mistakes.

In the early phases of a big change (or new product), push yourself to diverge, developing multiple possible directions and completing preliminary testing on your favorites. The successful innovator is also a good economist and the economy of broad prototyping in the early phases is clear:

prototyping-economy-curve

I’ve organized the balance of section against three progressive approaches to testing: exploratory, assessment, and validation. These aren’t mutually exclusive- commonly, you’ll move from one to the other.

Exploratory testing advances the discovery process with the benefit of early user contact. You’ll start with an explicit prototype and test plan, but freely update them on a test by test basis as you learn what’s working and your observations encourage new ideas. Specific measurements are unimportant. Popular tools are paper prototypes, PowerPoint/Keynotes prototypes, and various other facile tools that discourage overemphasis on details and encourage experimentation and variation. Exploratory testing drives to a decision about the fundamental approach on an interface. 

Assessment testing focuses on measuring the fundamental efficacy of a new interface (or interface element). As opposed to validation testing, it’s less concerned with detailed measurements on specific items, but instead on whether or not most users can accomplish topline goals, noting the rough spots. Assessment testing typically provides inputs to further interface refinement (if it’s basically successful) or pivot and revision (if it is not).

As the name suggests, validation testing is for once you’re relatively sure you’ve got an acceptable interface, and you want a) a final check on that and b) some intuitive sense about the nature of what you’ll see with measurements you take out in the wild (through Google Analytics, etc. with users you don’t have the chance to meet and observe directly). Try hard to leave yourself enough time to make at least a few minor changes before releasing to the public- you’ll likely identify a few rough spots well worth fixing.

You can also execute all three of these tests on a comparative basis (comparison between alternatives).

Preparation

Relative to customer discovery interviews, user testing requires a relatively detailed set up. As with any detail-oriented task, it’s easy to get lost in the weeds. Resist the false gratification that task completion provides: the important thing is connecting what you’ve learned back to your core innovation engine.

User testing occurs relatively late in the overall discovery process. At this point, you should have validated personas, problem scenarios, and value propositions, something you can summarize in a Product Hypothesis. The diagram below describes all this in terms of the convergence-divergence pattern from design thinking. Your work on the Persona & Problem Hypothesis should ultimately converge on a focal problem set, to which you attach your value propositions. All this you can summarize in a Product Hypothesis to connect problem with solution.

The Product Hypothesis will include a Value Hypothesis and that will have certain assumptions attached to it. You should test those assumptions with the quickest, least expensive MVP product proxies you think are viable to provide a definitive, actionable result (see section on Value Hypothesis for popular patterns). Once you’ve converged on which problems you can provide adequately valuable solutions, encapsulate your solution ideas in agile user stories. Then prototype, test, refine, and test some more!

customer-discovery-divergence-convergence-v2

Every substantial interaction you plan to test should have a user story attached to it. The user stories will anchor your executions in the customer narrative and validated learning you’ve developed. For every material interface element, you should be able to answer these questions:

What user story did we design & execute this against?

What value proposition was that story delivering against? Why do we believe the user wants to do this?

What problem are we solving for the user with this value proposition?

Who is this user? Do we understand what makes them tick? What shoes they might wear?

This will not only help you converge on more valuable solutions; it will also help you zero in on the right place to start revisions when your execution doesn’t go perfectly (and it never does!). Also, successful innovation is a loop, not a line. I’ve presented most of the material as a sequence for clarity, but the successful innovator is constantly looping back through the hypothesis areas. People change, problems change (a little), and every solution is temporary.

Execution

0. Prepare

User testing will deliver useful results on the investment just about all the time. Don’t feel like you absolutely can’t do it if you haven’t completed the items in the preceding section. That said, a little investment in those other areas (personas, problem, MVP testing) will go a long way.

1. Decide Test Type

Is your primary test objective exploratory, assessment, or validation? You can have elements of each, but I strongly advise deciding in advance your principal focus. The implications for preparation, execution, and decision (post-test) or substantially different between the three. Clearly explain to the rest of your team the focus- that will help bring the right focus to the testing.

Here’s a brief assessment, if you’re having trouble deciding–

You’re in the early phases of creating a new interface/interface element: Exploratory

You’ve drafted one or more directions that are well supported by validated learning and a review of comparables: Assessment

You think you’re done: Validation

2A. Prepare Research Design 

This has several sections in addition to the test plan- I call use the term ‘research suite’ to designate the whole package for a given set of testing. At first, it may look like a lot of stuff, but every time I’ve done user testing I’ve found all this preparation well worthwhile (and the reverse is also true). The example and template I reference are in the Venture Design Template/Appendix B. Notes on preparing the various sections of the research suite follow:

:: Objectives & Methods

This is for you to use internally. Describe in the clearest possible terms what you want to have happen as a result of the testing and your principal testing focus (exploratory, assessment, or validation), linking that to your objectives. Particularly if you’re early in a project, you may want to couple additional customer discovery interviews (persona, problem hypothesis) with your user testing. This is the place to explain that.

Here’s an example from Enable Quiz:

There are three general types of tests:- Exploratory: for learning about customer problem scenarios in greater detail, sometimes with a paper or working prototype

– Assessment: for testing the usability of an early direction on product implementation

– Validation: for later stage final testing of an implementation

This test suite is exploratory and we’re preceding the user testing with customer discovery interviews from Appendix A [this is the appendix for customer discovery interviews in the template] to deepen and align our view of personas and problem scenarios with the exploratory test results.

:: Product Version

Note the version you’re planning to use for testing. See the next step (2B) on defining a test release. Preview, though, so you don’t get stuck: this need not be actual software or a real release- it could be paper prototypes or a prototype you make in PowerPoint or Keynote.

:: Subjects

Define your target subject count in terms of the personas you’ve identified (see above or tutorial on personas). You’re not trying to achieve statistical significance in most cases (and almost always in the early phases), so 5-10 subjects is perfectly OK.

Here’s an example from Enable Quiz:

Since enabling the HR manager persona to be more effective is central to our value proposition, our target weighting of subjects should reflects that. An ideal total and mix of subjects would be:
Helen (or Hank!)  the HR Manager 4Frank the Functional Manager 1-2The screening question for both these subjects type is:
How many technical job candidates did you interview in the last month?

:: Research Composition

Here you will summarize everything that happens to a subject and how long you think it will take. Here’s an example from Enable Quiz:

# Item Duration (min.) Notes
1 Intro. & Explanation 5 Here we will explain the objectives of the test and the parameters of their participation. We’ll also obtain the designated release & consent form*.
2 Discovery Questions 20 Using the interview guide, we’ll spend a few minutes to discovery to improve our personas, problem scenarios and user stories.
3 Test Tasks 15 We’ll introduce the test scenario and then ask them to complete the Test Items.
4 Post-Test Debrief 5 Make sure we ask if it is OK to follow-up with additional questions.

:: Pre-Session Checklist

There’s nothing worse than starting of a test with something not being ready or general not in the right state. This is a simple checklist to help you and/or your collaborators make sure they don’t have any false starts. Here’s an Example from Enable Quiz:

# Item Notes
1 Make sure have written versions of discovery and test questions to refer to
2 Make sure test instance is up and functional – log in- make sure app is on starting page
3 Make sure recording equipment* is up and functional

:: Session Design

This includes the intro you’ll do with subjects as well as the test items.

I strongly advise writing up the intro and practicing it- it takes work to put yourself in the subject’s shoes and as things get busy (and repetitive) you’ll easily miss things.

On the test items, you’ll notice each row has four items–

Enumeration (#): This is just for reference

Research Objective: This will help keep you focused. Each item should have a research objective (otherwise, why is it there?). If you’re running an exploratory or assessment test, your user stories can provide a great anchor for the objective (see example below).

Estimated vs. Actual Time: This is for setting expectations on duration as well as evaluation. If you’re running an exploratory or assessment test, you’ll be less concerned with actuals.

Notes: This is where you set up and design the testing. I like to break each of these into a set of notes for the moderator and a set of target outputs. The outputs should closely and obviously tie to the research objective.

Here’s an Example from Enable Quiz:

Intro
Thanks for making time to take part in our study. My name’s [name] and this is [observer]. [Explain participation and deal with consent form/obtain written consent]*We’ll be using a test guide through the rest of this, so I hope you won’t mind me referring to that.We’re here to learn about [an early version of a solution that allows HR managers to assess the technical skill set of a job candidate through an online quiz].I’m going to ask you some questions and give you some tasks. Feel free to think out loud if you’re comfortable with that. We’re not here to test you and there are no wrong answers. Our results just help us better understand our product and its audience.The session will take roughly [40-60] minutes.Do you have any questions before we start?Test Items
# Research Objective Est. v. Actual (min.) Notes
1 Exploratory Intro 5 MODERATOR GUIDE
Let’s say your job is to create one of these quizzes for an open position. Here’s a description of the position [Provide them sample job description and let them review.]. Let me know when you’ve finished reviewing it and if you have any questions.OUTPUT
Validation that the subject understands their goal and the job description, roughly as well as they would in their current position.
2 Assess primary navigation for new quiz creation 2 MODERATOR GUIDE
Let’s say you want to create a new quiz. What would you do?OUTPUT
Assessment of primary navigation for new quiz creation
3 How are we doing on this user story:As an HR manager, I want to match an open position’s required skills with quiz topics so I can create a quiz relevant for candidate screening.

?

5 MODERATOR GUIDE
Tell me what you think you’re seeing here?Let’s say you wanted to choose a set of quiz topics for the open position you just reviewed. Show me how you’d do that?OUTPUT
An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

 

NOTES ON TAKEAWAYS

Personas & Problem Scenarios […]
UI and User Stories […]

 

:: Post-Test Debrief

Do you really need this? Yes, probably. This is an after the fact checklist to make sure you cover your bases: seeing if follow-up questions are OK, compensating the subject (in whatever way you plan), seeing if they have other thoughts, seeing if they have ideas on other subjects. Here’s an example from Enable Quiz:

– Thanks so much. We’ll be using this to make the product and solution stuff like documentation better.
– Would you mind if we send you follow-up questions?
– (if you’re giving them some kind of tangible thank you, make sure that gets done)

*:: Note on Recording and Compliance

I don’t supply legal advice on this site and I don’t warrant these notes as fit for legal compliance. As well it should be, recording individuals is subject to various laws and regulations depending on who you are, who they are, where you are, and how the recording will be used and stored (among other factors). It’s important that you get advice from your legal counsel and maintain transparency and applicable compliance with your subjects. At a minimum, this means securing written releases for the recordings and making sure that the recordings are stored and accessed securely (if you store them at all). Regarding releases and consent, your specific compliance requirements will vary, but here are a few sample consent forms from US institutions:

Usability.gov

Indiana University

2B. Prepare Test ‘Release’

If you’re working on an actual piece of software, test what’s current but don’t make yourself crazy (and probably your subjects) by cutting it too close.

If you want to do an exploratory or assessment test against a prototype, say a Balsamiq mockup, there are alternatives to building working software.

:: Paper Prototype

Yes, you can actually get meaningful test results from playing with pieces of paper. It’s hard to believe until you do it.

To start, you need a prototype. The Balsamiq prototying process will serve you well, assuming your subject is software. You will essentially prepare a set of screens on paper and ask the user to interact with them- clicking (by pointing) and typing (by writing with a pen).

Modularity and layering will serve you well in your preparation. I recommend having a few templates that are regular paper layered on cardboard or a similar substrate. This will make it easier to physically handle the prototypes and exchange them with the user. The base template should look like your target device- phone, tablet, laptop, etc.

Then layer basic screens on top of those (with light paste or spray adhesive which you can buy at any craft store). On top of those you can layer additional controls (Balsamiq lends itself to modular disposition of controls). And finally on top of the controls you can layer Post-It’s (or strips thereof) onto which users can type (photograph the results after tests and then just replace the Post-It’s).

Try it a few times and you’ll probably find you’re not uncomfortable with the process.

:: PowerPoint or Keynote Prototype

This is the same basic idea as paper prototyping but you’re simulating the interaction with inter-slide links on PPT or Keynote. The advantage is that everything’s on the computer if you’re not a glue-and-scisssors fan, and the experience may feel more real to subjects. The disadvantage is that the linking can get confusing, improvisation is harder, and if you want the user to fill out text you’ll need to have a paper form for them anyway.

Create the various (static) screens you want as slides within your application of choice. Then add inter-slide links. In the current version of PowerPoint (Mac; I’ll guess it’s the same on PC but haven’t been able to check), you do this by:

– two-finger (or right) clicking on a shape

– selecting ‘Hyperlink’

-then selecting ‘Document’ in the pop-up

– and using the ‘Locate’ button to find an ‘Anchor’ (you’ll need to click the right triangle to unfurl the list of slides).

On Keynote it’s simpler: two-finger/right click a shape, select ‘Add Link’ or ‘Edit Link’ if you have one in place, and then select the target slide.

 

3. Prepare Test ‘Infrastructure’

When the alternative is doing nothing, you can finish a darn good test by sitting someone down in front of what you have, giving them a few goals to complete, and seeing what happens.

Few of you will have access to observation booths etc., so I’ll skip that.

If you have the team size, separating the facilitator and observer/note-taker functions is very helpful, leaving the facilitator free to focus on the experience of the subject.

Make sure the facilitator is close by, but ideally not immediately visible or over the shoulder of the subject. A good location is between the subjects 4/5-o’clock and 2′-4′ distant. The observers will generally sit behind the subject- as far away as possible where they can still see what’s happening.

A simple PC/Mac with a web-cam will do fine. For recording screen activity and a web-cam feed on a PC, I like Camtasia Studio. For the Mac, I use ScreenFlow. Make sure you have everything recording and rendering the way you expect beforehand.

Note: You have serious obligations (ethical and legal) to steward and safeguard your subjects’ privacy and obtain their explicit agreement on participation, particularly if you’re recording. See the above note on ‘Recording & Compliance’.

4. Obtain Subjects

First off, if you’re trying out a new test set up (not to mention if you’ve never done this before), find some subjects where you can ‘test the test’. This is anyone who could plausibly use the product, even if they don’t well represent one of your target personas. Things will break, you’ll fix them, don’t worry, it’s natural.

Following this, prepare a screener- simple, factual question or questions to quality the relevance of subjects. With usability you can be a little more lenient than with development of your persona and problem hypothesis, but watch for the bias towards subjects that are convenient & comfortable vs. relevant.

6. Execute Test Suite

If you have a research suite along the lines of what’s above/in the template, then you have a plan.

In working the plan, practice is the best tutor. Be careful not to coach subjects too much or make them feel judged. It’s painful to watch them struggle with something they don’t understand, but better to learn about that now than subject every future user of your product to it! Give them time to work through confusion. Eventually (set a threshold for yourself) you’ll need to help them move forward, but make sure you don’t do it too soon.

Don’t forget to thank your subject, compensate them (in whatever way you plan), and ask them if follow-up’s are OK.

8. Make Your Notes (ASAP!)

I recommend doing this right away. Most of the important insights you’ll have, you’ll have on the spot.

Validation (Invalidation)

As with any test, conclusions are the point. Success/a good result will vary by test type.

Exploratory: The results should help you better understand the likely journey of a typical user and, depending on where you are in designing/prototyping the interface, whether you’re headed in a workable direction. Comparison tests here are highly desirable given their low cost and possible impact.

Assessment: The key question here is whether the ‘pivot or persevere’ on a given direction. Lots of stuck and/or frustrated users means no. Be ready to iterate a lot- the change you need may not be radical. Comparison testing is also highly economical here.

Validation: Here you’ll generally have a quantitative target for time spent per task and in total on your major experience arcs. Validation is being within a reasonable deviation from that.

Creating Effective Screeners

It’s not hard to spend 45 minutes with a subject only to realize they’re really not the subject you’re looking for. Particularly if you dive into detail early, they may be informative enough to keep you going even if you’re on a road to nowhere. This applies to all the hypothesis types and research techniques above.

For this, we create ‘screeners’. Basically the screener is a simple, factual question or set of questions can you ask a potential subject to be sure they’re relevant. It shouldn’t take much.

For example, let’s say we’re interested in problem scenarios around some aspect of network management, with the idea of possibly building an application for network engineers to manage transport elements like routers and switches. We have a persona(s) for the end user that we want to develop and validate. A good screener would be: ‘How many times last week did you log into a switch or router?’. Let’s say we’re building software for plumbers. A good screener would be: ‘How many plumbing jobs were you out on last week?’.

The screener is more important than you might guess at first. We have a natural bias to go with subjects that are convenient & comfortable, which can dramatically limit actionable learning. Don’t blame yourself, but do screen yourself!

You’ll find both the Enable Quiz example usability test plan as well as another that tests automation platforms for social media (Hootsuite, Buffer, etc.) in the References at the end of this page

 

Reference A: Enable Quiz Usability Test Plan

Objectives & Methods

There are three general types of tests:
– Exploratory: for learning about customer problem scenarios in greater detail, sometimes with a paper or working prototype
– Assessment: for testing the usability of an early direction on product implementation
– Validation: for later stage final testing of an implementation

This test suite is exploratory and we’re preceding the user testing with customer discovery interviews to deepen and align our view of personas and problem scenarios with the exploratory test results.

Product Version

We’ll be using version [0.1] of the product for this exploratory test. [NOTE: They could easily be using paper or PowerPoint prototypes as this stage as well]

Subjects
Since enabling the HR manager persona to be more effective is central to our value proposition, our target weighting of subjects should reflects that. An ideal total and mix of subjects would be:

Helen (or Hank!)  the HR Manager 4
Frank the Functional Manager 1-2

The screening question for both these subjects type are-
Helen (or Hank!)  the HR Manager: How many technical job candidates did you screen in the last month?

Research Composition

# Item Duration (min.) Notes
1 Intro. & Explanation 5 Here we will explain the objectives of the test and the parameters of their participation. We’ll also obtain the designated release & consent form*.
2 Discovery Questions 20 Using the interview guide, we’ll spend a few minutes to discovery to improve our personas, problem scenarios and user stories.
3 Test Tasks 15 We’ll introduce the test scenario and then ask them to complete the Test Items.
4 Post-Test Debrief 5 Make sure we ask if it is OK to follow-up with additional questions.

Pre-Session Checklist

# Item Notes
1 Make sure have written versions of discovery and test questions to refer to
2 Make sure test instance is up and functional – log in

– make sure app is on starting page

3 Make sure recording equipment* is up and functional

Session Design

Intro

Thanks for making time to take part in our study. My name’s [name] and this is [observer]. [Explain participation and deal with consent form/obtain written consent]*

We’ll be using a test guide through the rest of this, so I hope you won’t mind me referring to that.

We’re here to learn about [an early version of a solution that allows HR managers to assess the technical skill set of a job candidate through an online quiz].

I’m going to ask you some questions and give you some tasks. Feel free to think out loud if you’re comfortable with that. We’re not here to test you and there are no wrong answers. Our results just help us better understand our product and its audience.

The session will take roughly [40-60] minutes.

Do you have any questions before we start?

Test Items

# Research Objective Est. v. Actual (min.) Notes
1 Exploratory Intro 5 MODERATOR GUIDE

Let’s say your job is to create one of these quizzes for an open position. Here’s a description of the position [Provide them sample job description and let them review.]. Let me know when you’ve finished reviewing it and if you have any questions.
OUTPUT

Validation that the subject understands their goal and the job description, roughly as well as they would in their current position.

2 Assess primary navigation for new quiz creation 2 MODERATOR GUIDE

Let’s say you want to create a new quiz. What would you do?
OUTPUT

Assessment of primary navigation for new quiz creation

3 How are we doing on this user story:

As an HR manager, I want to match an open position’s required skills with quiz topics so I can create a quiz relevant for candidate screening.

?

5 MODERATOR GUIDE

Tell me what you think you’re seeing here?
Let’s say you wanted to choose a set of quiz topics for the open position you just reviewed. Show me how you’d do that?
OUTPUT

An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

NOTES ON TAKEAWAYS

Personas & Problem Scenarios […]
UI and User Stories […]

Post-Test Debrief

– Thanks so much. We’ll be using this to make the product and solution stuff like documentation better.

– Would you mind if we send you follow-up questions?

– (if you’re giving them some kind of tangible thank you, make sure that gets done)

* I don’t supply legal advice on this site and I don’t warrant these notes as fit for legal compliance. As well it should be, recording individuals is subject to various laws and regulations depending on who you are, who they are, where you are, and how the recording will be used and stored (among other factors). It’s important that you get advice from your legal counsel and maintain transparency and applicable compliance with your subjects. At a minimum, this means securing written releases for the recordings and making sure that the recordings are stored and accessed securely (if you store them at all). Regarding releases and consent, your specific compliance requirements will vary, but here are a few sample consent forms from US institutions:

Usability.gov

Indiana University

Reference B: Usability Test Plan for Small Business Social Media Automation

Screener

How many times last month did you post to social media for your business? What services did you use?

Objectives & Methods

There are three general types of tests:
– Exploratory: for learning about customer problem scenarios in greater detail, sometimes with a paper or working prototype
– Assessment: for testing the usability of an early direction on product implementation
– Validation: for later stage final testing of an implementation

This test suite is an assessment test.

Product Version

We’ll be using version [x.y] of the product for this exploratory test. [NOTE: They could easily be using paper or PowerPoint prototypes as this stage as well]

Subjects

Our core subject has a small company or personal brand they’re promoting. [XYZ], is central to our value proposition of [ABC] so we’re targeting a composition of subjects as follows (organized against our personas):

Sam the Small Business Owner 4

Research Composition

# Item Duration (min.) Notes
1 Intro. & Explanation 5 Here we will explain the objectives of the test and the parameters of their participation. We’ll also obtain the designated release & consent form*.
2 Test Tasks 15 We’ll introduce the test scenario and then ask them to complete the Test Items.
4 Post-Test Debrief 5 Make sure we ask if it is OK to follow-up with additional questions.

Pre-Session Checklist

# Item Notes
1 Make sure have written versions of discovery and test questions to refer to
2 Make sure test instance is up and functional – log in

– make sure app is on starting page

3 Make sure the subject doesn’t have an account on [social media automation system] already
4 Make sure they have accounts on at least two of: FB, Twitter, LinkedIn, G+ and they know their username and password

Session Design

Intro

Thanks for making time to take part in our study. My name’s [name]. [Explain participation and deal with consent form/obtain written consent]*

We’ll be using a test guide through the rest of this, so I hope you won’t mind me referring to that.

We’re here to learn about a product that helps individuals and teams manage social media accounts.

I’m going to ask you some questions and give you some tasks. Feel free to think out loud if you’re comfortable with that. We’re not here to test you and there are no wrong answers. Our results just help us better understand our product and its audience.

The session will take roughly [40-60] minutes.

Do you have any questions before we start?

Test Items

# Research Objective Est. v. Actual (min.) Notes
1 How are we doing on this user story:

As a Sam the Small Business Owner, I want to sign up for the service, so I can give it a try.’

?

5 MODERATOR GUIDE

Let’s say you’ve heard that what you’re seeing here [name] is a great tool for managing social media marketing for your business or personal brand and you want to give it a try.
What would you do here?
OUTPUT

Validation that the subject understands and can complete the signup process.

2 Assess start page and major navigation elements 2 MODERATOR GUIDE

What do you think you’re seeing here?
OUTPUT

Assessment of user’s relationship to the available affordances, their prominence in the presentation, and their appropriateness to the top user narratives and tasks.

3 How are we doing on this user story:

As a Sam the Small Business Owner, I want to I want to connect my social media accounts, so I can create automated posts to them.’

?

5 MODERATOR GUIDE

I have it in the notes here that you post to LinkedIn, Twitter, a Facebook page, and G+. Is that right?
Let’s say you wanted to add your Twitter and LinkedIn accounts. Would you show me how you would you do that?
OUTPUT

An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

4 How are we doing on this user story:

As Sam the Small Business Owner, I want to schedule some posts for two days from now at 8AM [local time zone], so I know it’s going to post at the time I want.

?

MODERATOR GUIDE

Let’s say you wanted to compose and schedule a posting for 2 days now at 8AM [local time zone]. The text is:

‘Free fries from 3-6pm today!’

and the url is www.alexandercowan.com
I’ll help you make sure it’s not accidentally posted and we’ll delete it at the end. Let’s say you want it to post to both Twitter and LinkedIn. How would you do that?
OUTPUT

An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

5 How are we doing on this user story:

As Sam the Small Business Owner, I want to update the time on a scheduled event, so I know it’s going to post at the time I want.

?

MODERATOR GUIDE

Let’s say you wanted to change the items you just scheduled to post at 10am instead of 8am. Would you show me how you would do that?
OUTPUT

An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

6 How are we doing on this user story:

As Sam the Small Business Owner, I want to remove a scheduled event, so I know it’s not going to post.

?

MODERATOR GUIDE

Let’s say you wanted to remove that event so nothing goes out at all. Would you show me how you would do that?
OUTPUT

An assessment of the user’s relationship to the available affordances and their appropriateness to the current user narratives and tasks.

NOTES ON TAKEAWAYS

Personas & Problem Scenarios […]
UI and User Stories […]

Post-Test Debrief

– Thanks so much. We’ll be using this to make the product and solution stuff like documentation better.
– Would you mind if we send you follow-up questions?
– (if you’re giving them some kind of tangible thank you, make sure that gets done)
– (make sure to delete their accounts and all login, password, and personal identifying data)

* I don’t supply legal advice on this site and I don’t warrant these notes as fit for legal compliance. As well it should be, recording individuals is subject to various laws and regulations depending on who you are, who they are, where you are, and how the recording will be used and stored (among other factors). It’s important that you get advice from your legal counsel and maintain transparency and applicable compliance with your subjects. At a minimum, this means securing written releases for the recordings and making sure that the recordings are stored and accessed securely (if you store them at all). Regarding releases and consent, your specific compliance requirements will vary, but here are a few sample consent forms from US institutions:

Usability.gov

Indiana University

  • Sherman

    Great article…though I admittedly only read the first few parts of it and skimmed the rest (it only took me a few paragraphs to know you know what you’re talking about; Pocket for the rest).

    Regarding this piece:
    “3. Find Subjects

    Are you finding that getting to the right interview subjects is difficult, messy, and time-consuming? Good- you’re probably doing it right.
    If I knew an easier way, dear reader, I would tell you straight away. The reality is that soliciting online and even using high-priced agencies tends to deliver atypical subjects, semi-professional participants that are not a reliable or useful source of actionable learning.”

    Dear author, we do have an easier (but still not perfect) way – http://www.customerdiscovery.ninja
    Yes, the interviewees are paid, introducing some bias, but it’s been a useful tool for scaling customer discovery efforts. We’ve had over 8000 recorded minutes of interviews as of writing.
    I know the tool is useful because my co-founder and I built it out of necessity and use it frequently.

    I just filled out your contact; hopefully we can talk more.
    Steven

  • Jaffy

    This IS what I was searching for. I saved in Evernote, because this needs to exists in the future. Thanks from a random brazilian person.

    • Hi Jaffy- Thanks for writing, and I’m so glad you liked it!

  • j_mes

    I’d love to know when you’re planning to release the Customer Creation Hypothesis, as it’s what I’m struggling with most. Do you have any other resources you could recommend in the meantime.

    Thanks again for your content Alex, I can’t understand why this isn’t more popular — it’s incredible!