20% of Essential Feedback is Lost in User Interviews
Customer-Centricity | Industry Savvy

20% of Essential Feedback is Lost in User Interviews

on / by Oliver McGough

  • “In my experience, many people confuse being cowardly with being nice.”

—Robert Kiyosaki

User interviews have a key flaw, a flaw many of us would be reluctant to admit. With the expectant eyes of our superiors bearing down upon us, we daren’t show any sign of weakness. This is a flaw which threatens to undo and unravel all of our hard work.

People are too nice.

The issue is eternal. As humans, we are conditioned to save face – to not hurt people’s feelings. To play to people’s sensibilities. The issue when testing a design is that sensibilities are redundant. We want something that works, irrespective of our designer’s feelings. Our job is not to forever justify our designs, our job is to design something that requires no justification to its user. And for that? We need criticism.

Listening to ‘lies’

Michael Wilson wrote a superb article detailing his issues with this testing process. He explains his first hand experiences of test participants “lying” about their thoughts. Lies which ruined his whole research process.

His experiences are not unique. Three months ago, we had an intern at Usabilla looking into the UX of dashboard design based on online user feedback. Her method was build to two dashboards – one based off of the current usabilla dashboard, and one new design based on user feedback.

Through face-to-face interviews, she tested these two dashboards with current Usabilla users – gauging their sentiment for each.

Despite so much negative feedback for our current dashboard, her results showed that people actually favoured the current one over the new!? Quite a contradiction. This set the alarm bells ringing on the differentiation between online and offline feedback results.

Allow your users to vent at the machine

Participants – people – are scared to truly tell you how they think. Scared to show that they don’t know what they’re doing because of your clunky interface, or scared to tear apart the fruits of your labour.

We have to bypass this cowardness to speak the truth and to uncover the source of the problem.

Our intern saw a situation where despite online frustrations, our users preferred what they currently had when we asked them in person. A fact which is quite hard to understand, but which actually makes perfect sense when we change medium:

User sentiment directly conflicts with itself when compared online to offline.

In fact, we see a 20% increase in negative feedback through anonymous online channels – a striking disparity.

Our users will will gladly scream expletives at machine, less so at a pretty little girl. Whether this is merely societal upbringing or human nature, replacing the human with a machine is effective in coaxing real feelings from our users.

Takeaways

  • “Really pay attention to negative feedback and solicit it …Hardly anyone does that and it’s extremely helpful”

—Elon Musk (Tesla, Space X, Paypal)

Negative feedback is essential. Only by understanding where our UX pitfalls and shortcomings are, can we create truly great experiences. By remaining an insensitive machine, we can take flack, discover our pitfalls, and change accordingly.

User testing should never be over reliant on just one medium, 20% of your feedback is at stake. Always run a variety of tests – online and offline, face-to-face and anonymous; always understand the big picture.

| | | |
Article by

Oliver McGough

Passionate UX Designer and Marketer.

Share your thoughts

  • All qualitative research is going to have bias that you can’t account for. But, I think it can be dangerous to measure the usability of something through an interview “gauging sentiment” or from your anonymous online comments. Much better is a usability test that asks participants to actually accomplish tasks. With a usability test you can take out more of the “people pleasing” component, because as Michel Wilson said “At the end of the day, the user can only tell you so much. However, the things they can show you are limitless”
    http://www.uxbooth.com/articles/encouraging-negative-feedback-during-user-testing/

  • Alex Barker

    Interesting observation Oliver, but surely it confirms that any significant and valuable user research should be a mix of both qualitative AND quantitative; offline AND online; moderated AND unmoderated.

    That way you can gain all the valuable and unpredictable nuanced finding, balanced with the statistical significance (and negative balance) of the wider, machine-based data set.

    We’re increasingly including Social Media Listening into this mix to get an even more warts-an-all view of design or situation to focus more on the negative, than the predictable lies suggesting the positive!

  • Mykel de W

    Methods & Techniques are very valuable as tools for gaining understanding, IF wielded “correctly”. However they will never yield a picture as objective as it can be according to theory. To me, negative feedback is the single most important input for determining whether I’m actually looking at the right problem, before solving it.

    There’s few things as misleading as successfully resolving “the wrong” problem. Under budget andahead of schedule are pointless at best of one solves the wrong problem.

    • Oliver McGough

      I’d certainly agree :) Negative feedback is essential to actually discovering the problem at hand. In all walks of life, we need criticism to improve our ways and fix our weaknesses.

Pin It on Pinterest