Worry less about introducing bias and more about accounting for it

When it comes to biases in in-depth interviews, pick your poison. I chose a conversation full of wit, charm, and humor over an impersonal professional interview. There’s a lot said about introducing biases in UX and market research but these discussions are theoretical – in practice, the researcher will introduce bias no matter what. Look no further than the “Hawthorne Effect” where people act differently when they know they are under study.

We could all come up with a list of potential biases and how to avoid them. Sticking too closely to this flattens the researcher’s personality. Instead of acting naturally, researchers replace part of their personality with a non-relatable professionalism. This censoring of the researcher dehumanizes the interaction. One wonders what the user withholds when met with this calculating demeanor. Certainly it doesn’t make for good conversation. It’s like trying to enjoy dancing with a partner who is rigid and reluctant to move or smile. Attempts at preventing bias introduce a bias of its own.

If the goal is to elicit complete transparency and forthrightness from the user, we should frame studies less as lab experiments and more as engaging, albeit messy, conversations. We often want to understand the complex thought processes of users. These don’t come pre-formed and compartmentalized, just waiting for extraction. In fact, some of the most informative responses seem to develop in real-time through self-reflection on the part of the user. Open self-reflection is an exercise in vulnerability where you give another person a window into your being. Here, flaws and idiosyncrasies become apparent, and deeply held beliefs and desires are revealed. It goes without saying that this only occurs if an authentic human-to-human connection has been made between the researcher and user. 

Also, developing complex ideas on-the-fly is cognitively intensive. This can be helped with a give-and-take discourse between the user and researcher. The user iteratively makes claims and the researcher tracks these thoughts, giving feedback along the way, with the goal of coming to a mutual understanding about what the user means to communicate. A true mutual understanding can only be achieved through honest and blatant feedback between the two parties. This is made more difficult under the pretense of professional etiquette. All of this is to say that in-depth interviews should feel more like collaborative conversations between two friends than they do one-sided interviews.

Of course, this brings up the question of bias. The researcher is sure to slip up and bias the user in some way during the course of natural conversation (e.g. imposing an opinion on the user). But, if you’re anything like me, you’re painfully aware of the mistakes you make while moderating. This is why I argue, we should worry less about introducing bias and more about accounting for bias. As human-centered researchers we intuit when something is wrong and the user is withholding some part of the truth. For example, we are sensitive to indicators like an incongruence between body language and speech, tonal inflections, lack of eye contact, and so on. The point is, experienced interviewers should be able to course-correct for these biases during the interview.

And since they aren’t constrained by a phony professionalism, researchers can assertively address problems of bias with the full force of an authentic personality and conversational wit. While the purely professional researcher can only voice the trite line we’ve all used, “I’m not part of this product so I want to hear the good and the bad.” On the other hand, the charming researcher can humor the user with, “I think you’re only saying that because you don’t want to offend me, what do you really think?”, followed with a disarming smirk. Done correctly, this frames your challenge as half serious, half joking and puts the user at ease when expressing negativity or disagreement moving forward. Also, this assertion of yours demonstrates to the user your commitment to getting to the bottom of things. It’s small instances like this that engage the user and set the stage for a more enlightening conversation.

I find that bringing my authentic personality to the forefront cures other common interview problems. The examples below may be my idiosyncratic ways of handling these situations – I’m sure we all have our go-to’s; however, the point is that we should be free to troubleshoot problems of human interaction in the same way we handle them in our everyday lives. For example, if the user is underwhelmed, then add an undercurrent of humor, poke fun at them or satirize the interaction. If the user is nervous, feign subject-matter ignorance and toss them “softball questions” to incrementally build their confidence. If the user isn’t talkative or not revealing much about their personal experiences, then show some vulnerability and tell a quick, related story about yourself. This is all common etiquette in day-to-day conversation and it’s foolish to think we can skip right past it in a professional research setting without negative consequences.

Perhaps we succeed as moderators to the degree that the conversation engages the user.  Even the most mundane or awkward topics can be made interesting and approachable. For this, researchers must become great conversationalists and avoid crutches like reading verbatim from the discussion guide as much as possible. There’s a reason unstructured interviews are considered more advanced than structured interviews. It reminds me of Anthony Hopkins’ acting process where he reads his lines up to 200 times before the shot, he “absorbs the text”, and when it comes time to act, he then advises to “relax and let the part play through you.” He says this is exciting because “you’re not quite sure what happens next.” This creative spontaneity, shared between two or more people, is what makes for great conversation.

This is why I say that in-depth interviews are more an art than they are a science. Sure, a fully structured interview with no probing is the equivalent of an oral survey. But as soon as the researcher reacts to what the user is saying or doing, there is no longer a repeatable scientific method. The best we have is an honest attempt on the part of the researcher to relate to the user in the here and now. This is why human-centered research, especially qualitative research, is so dynamic – humans appear to me infinitely complex and, therefore, infinitely interesting.

Leave a Reply

Your email address will not be published. Required fields are marked *