What if we stopped asking questions and just actively listened ? Case study

Intro

In the Discovery Training I teach, there is a session dedicated to the art of the Interview. People always want to know: “What is a good question?”. As time goes by in my Product journey, I am more and more convinced that questions, whatever the questions, are limiting most of the time if your goal is to get the deepest insights from your users.

Why ? Because every time you ask a new question, you stop the user in his thinking, he remains stuck on superficial insights with little value for you as a researcher.

So, what are you supposed to do if you can’t ask questions anymore as a researcher? Well, just listen (actively)! You will help the user access deep insights that he himself was probably not aware of !

In this article, we will first explain why questions are limiting most of the time before explaining what a user researcher should do if he doesn’t ask questions anymore. Finally, we will go through a real case study to demonstrate the power of active listening to access deep insights.

What is the problem with questions-based interviews?

Many interviews still go this way: the researcher prepares a list of questions that they want to ask during the interview. As the interview goes on and answers start flowing, the researcher feels excited and keeps moving to the next questions.

Was it a “good” interview? You probably succeeded in collecting a great number of insights (outputs) but failed to collect quality ones (outcomes). In fact, such an interview may even be a pure waste of time.

Why? When asking something to users, they reply with shortcuts. For instance:

• The most recent thing that happened to them (recency bias)
• What they think you expect them to say (courtesy bias)
• A simplified explanation that avoids too much thinking (heuristic bias)

I would say (I have no studies) that 90% of the time, their first answer is likely bad.

Accessing deep insights would require the user to make important and conscious efforts to go through his memory, recall the context, and analyze what really happened at that precise moment. Nothing easy!

Let me give you one exemple to make my point: how many times did you ask yourself if you really locked your door? We are talking about your house, which is one of the most precious things in your life. So, if you don’t really know about that, how can you give relevant answers to user research questions that are often very far from your daily concerns?

Every time you move to the next question, you stop this conscious process from happening. It’s like a “stop and go” process: you ask a question, the user starts thinking (without always realizing it) and when the next question comes, this thinking process is stopped. Sometimes, this process is so strong that it can still run in the background and the user may interrupt you by saying something like: “Remember what I was telling you 5 minutes ago? In fact it was not exactly what I meant. Can I add some elements?”.

What to do if we don’t ask questions ? Actively listen

So what are you supposed to do if you don’t ask questions anymore ? Active listening. I will state the main points (see “The art of interviewing” for more details).

First introduce the interview. In most cases, simply ask the user to tell you the last time they used your product or did something of interest for your research (behaviors rather than opinions).

Then, your role is to help the user access the true insights that may live somewhere far in their conscience. How to do that? Leave them time!

• Leave blanks that will force them to go one step deeper.
• Rephrase what they just said to help them challenge the relevance of their latest words, just by hearing them repeated by you (mirror effect).
• Challenge the shortcuts they make (meta-model).

At some point, the user will access the right level of consciousness and the true insight. The discussion might then start going into circles. It is your role opening a new narrative arc, saying something like: “It seems like we have uncovered the key elements of this first experience, would you mind telling me about this other part of your experience?”

At the end, you might have less insights, but way more qualitative ones! People usually fear lacking time for this way of doing. But to me, you better do 3 listening-based interviews than 10 question-based interviews!

A case study to see how it works

A story is worth pages of theory. Let’s see concretely how things can unfold depending on the way you conduct your interview (questions-based or listening-based)

Context

We will use a Saas Accounting tool through our exemple. The user, who has been a Saas1 client for almost 18 months, recently churned to Saas2. The Saas1 company wants to understand more precisely why some users churn and conducts research about that theme.

The User Researcher (UR) was given the task to call at least 20 persons to investigate churn. UR doesn’t have that much time because UR also has other assignments running in parallel. Besides, the UR already spent a long time finding relevant users ready to answer their questions and UR could only plan 30 minutes interviews with them.

Saas1 company is also thinking about launching a mobile app, the UR wants to benefit from this interview to get potential insights.

1. Questions-based interview

UR prepared their questions guide upfront. Here are the questions UR wrote on their guide and wants to address during the interview. Note that these questions are very common and might be considered as “best practices” (eg. opened, behavioral, tangible…):

• Why did you sign up on Saas1?
• What did you like most in Saas1?
• What did you miss?
• Would a mobile app have helped?
• What made you churn?
• Who did you choose as your new Saas accounting provider?
• Do you have any advice for us?

So let’s start with the interview !

UR: Thank you so much for your time! I really appreciate it. I would like to ask you some questions, is it ok for you? (user nodding). My first question is: Why did you sign up on Saas1 18 months ago?

The user is trying to remember, it’s a long time ago and many things happened since then…UR is making an effort not to interrupt and let them think.

User: I just started my new company, I was looking for an accounting software and I had seen many ads about your Saas1. It looked good, was not too expensive, so I went for it!

The probability the user gives the “right” answer is very low. It might be part of the solution, but the process must have been way more complex. He probably made a search on Google, compared prices, features, asked for referrals, even went through free trials. What “looked good” really mean? Modern? Trustful? Easy to use?

UR: very clear, thanks for your answer! (this kind of words might favor courtesy bias, putting the UR into a “teacher” like role). My next question is: as you started using Saas1, what did you like the most?

User is thinking… Hard question! He probably used Saas1, once a month, not really taking care, trying to do their accounting as fast as possible… Accounting remains a mandatory core for a large majority of users, the fastest they get it done the better…

This time, UR will interrupt contrary to the beginning of the interview, because UR wants to get answers, move to the next questions, etc…

UR: (trying to help) Was it about price? Functionalities? Customer service? (by suggesting answers, the UR is influencing).

User: (still not very sure, but seeing that UR is expecting an answer. User feels urged, and will make a shortcut) Customer Service was ok… Functionalities also.

UR keeps pushing the User to get answers…

UR: Which functionalities?

User: Well… Can’t really remember…

UR: (again, trying to help) VTA automatic calculation? Categorization…

User: (interrupting) Categorization was really helpful.

Categorization is certainly the most frequent task the user performed, was it really the most “useful”?

UR: Ok, categorization. On the contrary, are there things that you missed?

User: (this time the user looks very confident) I definitely missed an expert who would have been able to answer my “technical” questions. The customer service was not very trustful, they answered with emojis… (kind of distortion and omission that should be clarified)

There is definitely an issue at that point. But the user talks in terms of “solution” (accountant) while what would really be interesting would be the need answered by the solution (probably trust, even serenity to be compliant with the legal regulation).

UR chooses to focus on the emojis, perhaps because he is curious about them and wants to know more?

UR: you did not like the emojis?

A rewording would have been more powerful to let the user explore what these emojis triggered as emotions

User: Not at all, we are dealing with accountability, it’s important!

“It’s important” is an omission that should be clarified… What’s important for that user? For instance, if the user had been told that people behind the chat were accountants and still using emojis, would that still be a problem?

UR: I understand… To be honest, I would certainly have felt the same (trying to empathize, but this is not the most efficient way. We don’t care about what the UR feels, it might influence the user and foster courtesy bias).

UR is switching to a new question as time is running.

UR: Do you think a mobile App might have been useful to you?

UR has broken the flow the user was in…

User: A mobile app? I am not sure…

UR: (the UR himself is really convinced of a mobile app criticity, especially for one given use case. Confirmation bias will come into play) Let me give you an example: what about downloading a receipt at the restaurant?

User: Indeed, it might probably help…

Being external to the conversation, the user does not really seem to be convinced! Instead of challenging the answer, the UR takes it for granted and confirms their bias. What the UR misses at that point is that the user agrees because that’s precisely what he already does, but through their banking mobile app…

UR: (UR is happy to validate this key feature and moves directly to the next question). What finally made you churn at the end? Who did you go with as your new accounting Saas?

User: I told you, I did not trust Saas1 any more. At the end of year 1, there was a big mistake nobody noticed. It became clear to me that I needed to change software…I had had many great feedbacks on Saas2. Besides, unlimited access to true accountants is included in their subscription. This was exactly what I needed!

UR: True accountants were what you needed? (this is a great rephrasing)

User: exactly!

If this is really what the user desperately needed, how is it that the user waited another 6 months before switching to Saas2?!

UR: is there anything we could have done better?

User: perhaps reach out sooner? I appreciate you calling me, but it is too late…Perhaps you could be closer to your customers, especially when they go through problems?

UR: thank you so much for your precious time, it really helped!

User: you’re welcome!

At the end of this interview, the UR might be convinced that if Saas1 had been able to offer the user “real accountants” (solution A), the user wouldn’t have churned. Perhaps a better onboarding (solution B) to avoid the kind of mistakes the user went through on his first annual balance sheet might have been enough to keep the trust high (vs. offering “real accountants”, one of the conclusions of the first interview) and far less complex to implement. In every case, UR should have climbed up to the need and context level to really understand the user (see this article about the Product Pyramid for more details about solution, problem, need and context)

Let’s now see the other version of the interview. You might be surprised by what really happened to the user.

2.Listening-based interview

Let’s redo the interview using only active listening. We will just work on the first sentence, because listening-based is way longer! In the epilogue, we will give the “real” insights so that you can see the gap between what the user first said and what was really deep in their mind.

Let’s start to explore the first answer by rephrasing it.

UR: If I listened to you well, you said you were looking for accounting software and you had seen many ads about our Saas1. It looked good, was not too expensive, so you went for it, is it correct ?

This rephrasing is much more powerful to empathize than trying to be nice! It doesn’t create biases and gives the user the feeling to be perfectly heard.

The user is frowning while listening to what they just said… Their eyes go up and down, a proof that the user is processing information. Don’t interrupt, even if it seems to last hours (to you)!

User: That’s right… Also, when I founded my first company ten years ago, I suffered from being very dependent on an old-fashioned accountant…

UR: (rephrasing to question the omission) Being very dependent on an old-fashioned accountant?

User: Yeah… I mean, I had to send him my documents that I scanned before, we had no shared tools that I could have used when I had time to… When he asked me to, it was never the right moment, so I did it two days after their request… Not very efficient…

UR: (rephrasing again the omission) Not very efficient?

User: Definitely not… I wanted to be able to work at my own pace, when I wanted to… I wanted more autonomy.

UR : (rephrasing to see if we went far enough) Autonomy?

User: (this time very sure). Definitely !

The user looks right into the UR eyes, no need to wait for more information at this stage.

The UR reminds the user about the price as he wants to know more about it.

UR: So autonomy was important for you (rephrasing to strengthen the quality of relation and empathize).. You also said: “the price was not too expensive”?

User: (thinking) You’re right… Definitely less expensive than a true accountant… (The user is still thinking, as proven by their eyes looking up)… I mean, I used to pay thousands euros per year, with accounting software, we are talking of hundreds euros per year… I am not close to ten euros, but thousands to hundreds makes a big difference…

It is important here to notice that the user doesn’t say Saas1 is cheap compared to Saas2 but compared to “real” accountants (usually, we are very focused on competition, and not substitutes).

UR: you are not close to ten euros? (the UR is rephrasing this omission)

User: Not at all. I don’t care if I pay 50 or 70 or even 90€ per month as long as the service is right. It will always be far cheaper than a “rea”l accountant…

We just learned two very important insights: price sensitivity for this user is very low even if he said price was important to him at the beginning because he compares the Saas accounting category to the “real” accountant category. Second, we just got a new insight about an important criteria for the user: trust! So important for them that the user is ready to pay an extra fee for it.

UR: (rephrasing) As long as the service is right?

User: exactly (feeling deeply listened to). I mean, I really care about being clean with the accounting part, I don’t want to get into problems because of that. That’s why I asked one of my relatives who is a fiscalist to check my first annual balance sheet with Saas1… And I got really disappointed by Saas1 (and really happy to have asked for a check) when he found a really big mistake on it… Had he not checked, I would have sent wrong information to the public administration. I just paid Saas1 to avoid that kind of mistake. That really upset me. I sent something nice to my relative to thank him because he spent time on it, it cost me money… So that’s why I am telling you price, to a certain extent, is not an issue for me… But the trust was broken from that point…

UR: Trust was broken?

User: Definitely…

UR does not interrupt as the user seems to be still thinking.

User: It became clear to me I needed to switch to another solution…

Again, the UR is not jumping to investigate the other solution. The UR is right when you see what comes next:

User: In fact, I became dependent on my relatives… Instead of being dependent on an accountant… I was back to my situation 10 years ago! Except that my relative is no accountant… (blank). Plus I don’t want to ask him too much, I am feeling uncomfortable…

UR leaves again a long blank since the user seems to be lost in their thoughts. After a while, UR feels they went far enough on this narrative arc and will make the interview move one step further. Since UR took a few notes (too many notes would have damaged the relation between UR and the user), UR is able to make a recap using the user’s own words and thus strengthening the relation.

UR : Let me recap where we stand now. You went to Saas1 because it was cheaper than a “true” accountant. Also, Saas1 enabled you to be autonomous in your accounting. It turns out that you lost confidence in Saas1 and became dependent on one of your relatives you asked for support. Is this correct?

User: Definitely, yes! (feeling deeply heard).

UR: What happened next?

This question is very neutral and behavior-based. Let’s hear what the user answers.

User: I thought I would churn to another Saas solution…

UR: You thought ?

User: Yeah… I thought… In fact, nothing happened…

Quite interesting as information if you remember the conclusion of the first interview!

UR: Nothing happened?

Even if rephrasing might seem robotic from the outside, it is extremely helpful for the user to move on into becoming aware of what really happened.

User: Yeah… Because even if Saas1 was far from perfect, it did the job… (blank). After the first troubles, I knew what to take care of… Doing my accounting only comes back once a month, it takes about one hour… And you move to more important stuff… I was not motivated enough to conduct a new “research”, migrate my data, get familiar with a new interface…

There would be many things to investigate : what does “conduct a research” mean for the user? What does the user think he would have to do when saying “migrate my data”? What is very clear is that churning was an option but the mental load that it would have created would not have compensated for the pains the user had with Saas1 at that moment.

We will move forwards to the last narrative arc to end this case study.

UR: you finally churned. Can you explain to me how it went?

User: You’re right, I churned. In fact, in parallel to my consulting business, I started a new company with no legal structure behind. I wanted first to check if there was a market fit before creating any legal entity, so I billed everything through my consulting company. When it became clear that there was a market fit, I needed to create a new legal entity dedicated to my new venture. My situation had become legally way more complicated as it was at the beginning: two companies, a market-place activity, third party payment... Saas1 was not adapted anymore to this new situation. I benefited from this event to consider Saas2…

Very important information: compared to the first interview, the churn was not really motivated by Saas1's quality but rather by a new legal context for our user. This new legal context is not suited anymore for Saas1…

UR: to consider Saas2?

User: I had some great feedback about Saas2 from peers who started their own business too. Especially about the services beyond pure accounting software…

UR: beyond the pure accounting software?

User: Yeah… They have true accountants in-house. I mean, these accountants are not third-party accountants, they are real employees on the Saas2 payroll… I felt very confident to move with them.

UR: you felt confident moving with them?

User: Yeah… I had a call where I told them about my legal issues. Their advice was extremely relevant.

UR: Extremely relevant?

User: definitely. To be honest, I also had a call with a fiscalist and another company working with a third party legal advisor, Saas2 was by far the most relevant in their advice!! Plus, they were the cheapest… Even if I would have been able to pay more to be totally transparent!

UR: Saas2 advisors were the most relevant?

User: Yeah….

Blank

UR: any other insights about your move that come into mind?

Blank

User: Hmm… Not really sure… in fact there is a last element worth talking about. I told you about my concerns about the migration. Well, they told me there was no import to do! It relieved me! I had no more doubts about moving to Saas2.

We can see that Saas1 could not really prevent the user from churning since the user’s needs really changed when they created their new company. Had their situation remained the same, the user probably wouldn’t have churned even without “real” accountants!

Conclusion

We hope that the case study convinced you of the power of actively listening rather than asking questions.

When you are on the field, having questions to ask is very reassuring. With time constraints, other duties on your plates, you don’t even realize that you miss critical insights…

It takes way more time to conduct listening-based interviews but it is really worth it if you want to uncover worthwhile insights !

If you want to be alerted about my latest articles, feel free to register to my newsletter (mainly written in French but you can use google translate for instance).

I also teach a 5x1H30 Discovery Training that will give you the key principles to do Discovery that impacts !