“Survey, Mavis? No, have a nice cup of tea and a sit down.”

Don’t send that survey! Here’s what to do instead

This article was inspired when I was asked to review a survey about internal communications for a growing company, but the principles are universal.

When your goal is to really understand people’s experience, a survey will never give you the rich insights you’re hoping for.

You need to do something simpler, more human and ever so slightly scarier: have some conversations.

Most people resist this idea. It feels more time-consuming than bashing out a quick survey. In reality, the conversations will be much faster and much more likely to work.

Yes, having conversations does take time, but you don’t have to chat with many people before you can start spotting patterns of needs, problems, pains and blockages. (The secret is to start with 5 people, see what you find, and repeat as needed.)

What’s important about research is not how quickly you can get through it but how quickly it gets you to the insights you need. And having conversations is at least 1,000% more likely to uncover the information you need than sending surveys.

Why? Because while surveys have their uses, they aren’t very good at getting to The Thing Behind The Thing.

There are two big reasons why surveys tend to be bad:

1. nobody can really tell you what they do, how they do anything differently from anyone else, or why they really do anything.
2. you won’t even realise this because “bad surveys don’t smell”

“It is too easy to run a survey … This ease makes survey results feel true and valid, no matter how false and misleading.”

– this is from On Surveys by Erika Hall. If you haven’t read this article, do so now. Go right ahead. I’ll be here when you’re done.

…

All done? So you understand why surveys are (mostly) bad. I’ve seen lots of bad surveys (heck, I’ve sent plenty) and they are riddled with dangerous and hidden pitfalls. But we can avoid a lot of them with a simple test from Erika’s article:

The simple test for every survey question you pose is:

Will the people I’m surveying be willing and able to provide a truthful answer to my question?

To test this:

1. define what you’re really trying to learn from the question. Crucially, what will you do differently when you know the answer?
2. try to answer the question yourself.
3. usability test it: get a couple of other people to answer the question.

For an alarming number of questions, you’ll find that either you can’t get the answers you want or you can’t trust the answers you get.

To see how this works in practice, I’ll test a couple of real questions from the survey that inspired this article and then suggest a better question.

Teardown time! Improving survey questions

(Background: the survey was about internal communications tools. The company used a peculiar wiki system that was quickly getting more arcane and out-of-date as they grew in size and they wanted to know whether to switch the tool or improve it.)

Survey Question 1:
What do you like about the current wiki?

My answer:
Not a lot. Seriously. I’m struggling to think of anything I like about it.

Breakdown:

I genuinely couldn’t think of anything else to say. This question can’t get at anything the company would hope to learn because “like” is a very long way from “solves a business need.”

To be able to answer the question the company is trying to get at, I need to understand this wiki in comparison with other wikis and in comparison with other tools we could use instead. Spoiler: I don’t. And nor does anyone else.

Survey Question 2:
What would you like to see added?

My answer:
I don’t know how to answer this. Not even slightly.

Breakdown

I have no idea how to make this question better. It’s a question that I’m always either unwilling or unable to answer truthfully. We can’t ask this question directly and get anything meaningful.

Erika Hall again: “Often asking a question directly is the worst way to get a true and useful answer to that question. Because humans.”

I think of it like this: if I have a need, then I’ve found a way of meeting it. It’s almost certainly not the best way of meeting it, but it’s the best way I know of.

If I’m not using our wiki to meet a need, it’s because my mental model of wikis just doesn’t include meeting that need. And asking me this question isn’t going to magically make me rethink all my needs and how they might relate to wikis.

It’s not the user’s job to know what they need. It’s the user’s job to know what problems, pains and goals they have.

(The first part of that quote was Steve Jobs (thanks Dorian Freeman) Still can’t remember who added the crucial second part. If you know, please tell me.)

As an aside, if I’m an employee being asked about improvements to business tools, then there’s yet another layer. Because the problem is to do with the business’ needs, not mine. And if I’m not doing something the business needs me to do, there are only two reasons: either I don’t know I need to do it or I don’t want to do it. In either case, I’m not telling you about it in a survey.

For instance, say I’m a developer. The business needs me to document critical pieces of information about my work in a central place because we waste lots of collective time finding and interrupting developers. But personally, I’d rather be interrupted than make documentation. I don’t want to use your damn wiki, so why would I do anything to change that?

Here’s a better question we could use:

Tell me about a time you used the wiki recently (whether to add, edit or find information)

In your narrative please include:

• what was your goal in using the wiki?
• why did you need to achieve that goal?
• were you ultimately successful?
• If not, what was the main thing that stopped you?
• If so, did anything make it harder than it needed to be?
• If you could wave a magic wand and have anything, what would you change that would have made your task easier?

My new answer:
I needed to post a set of user research notes. I wanted to get the last one out before [date] because I’m trying to spread the knowledge about our customers across the company. I was mostly successful because I’ve already learnt how to get around all the … let’s call them quirks, shall we … in the system.

One thing I couldn’t do was upload a PDF that had a set of notes from one of the observers because it wouldn’t allow me to upload a PDF. I don’t know why this is. I solved this by copying and pasting all the text directly into the wiki, but I was annoyed because it lost all the formatting. If I could wave a magic wand, I would want to be able to upload the PDF and have it show in the wiki page with all its formatting.

While we’re at it, I would like to be able to copy and paste from my drafting tool directly into the wiki and add images by dragging them into the window.

And do you know what? It annoys me that user research notes are hidden away at the bottom of a big navigation tree. I doubt anyone would ever find them. I get around this by spamming everyone in the company. I guess I could do them as a blog post, but those seem more ephemeral somehow.

Breakdown

My answer here is 1,000% richer and more helpful, because the only thing I need to understand is my own direct experience.

I still can’t answer completely truthfully. When I answered, I hadn’t used the wiki in over a month. Even if I had superhuman self-observation powers, there’s no way I could remember all the steps and problems.

But “tell me about the last time you ...” is almost always a powerful question.

This question is even better when it’s not in a survey

I had to type a lot to answer that question. Most people aren’t motivated to go into such detail for you in writing. On the other hand, most people are very happy to tell you in detail over a coffee.

Even better, if there’s an interesting point in a conversation and you want to know more, it only takes a moment to ask. Not so easy with a written survey response.

You still need to ask questions that people are willing and able to answer truthfully. It’s diabolically easy to slip into asking leading questions that destroy the value of the answers you’re getting back.

The real killer is that you really really want to ask questions about what people will do in the future, and they are the very questions people can’t answer truthfully.

Don’t ask “what would you do if…” questions:

• If we build this product, would you buy it?
• What product should we build for you (you know, so you’ll pay us loads of money?)
• If we installed Snazzy Product X, would you use it?

You just can’t get meaningful, true answers to these questions. Stop asking them.

Instead, these handy guidelines will help you get meaningful, truthful information:

3 simple rules for valuable conversations

These simple rules from book The Mom Test will help you have valuable conversations:

1. Talk about their life instead of your idea
2. Ask about specifics in the past instead of generics or opinions about the future
3. Talk less and listen more

There’s a skill in extracting the useful insights out of these conversations. But that’s a topic for another day.

Bonus level: watch people

An even more powerful approach is to observe how people behave. As creatures of habit, people develop workarounds to satisfice their needs. They’re completely unaware of their habits.

You’ll know about this if you’ve ever watched someone edit a Word document, and you’ve ground your teeth in frustration as they tapped the cursor key 400 times to move backward or forward. You know they could get there ten times faster by holding down Alt or Ctrl. But they don’t. You can observe this behaviour, but if you asked them to tell you how they use Word, they would never think to mention their cursor skills.

The scary thing is that you have blind spots just the same. Everyone does.

At least now you don’t have a blind spot about surveys though.

THE END ... for now.

If you want more like this, join awesome designers and smart business owners and get my weekly letter. I write about evidence-based product design, contrarian A/B testing, and how to be profitably wrong.