20% of Essential Feedback is Lost in User Interviews
User interviews have a key flaw, a flaw many of us would be reluctant to admit. With the expectant eyes of our superiors bearing down upon us, we daren’t show any sign of weakness. This is a flaw which threatens to undo and unravel all of our hard work.
People are too nice.
The issue is eternal. As humans, we are conditioned to save face – to not hurt people’s feelings. To play to people’s sensibilities. The issue when testing a design is that sensibilities are redundant. We want something that works, irrespective of our designer’s feelings. Our job is not to forever justify our designs, our job is to design something that requires no justification to its user. And for that? We need criticism.
Listening to ‘lies’
Michael Wilson wrote a superb article detailing his issues with this testing process. He explains his first hand experiences of test participants “lying” about their thoughts. Lies which ruined his whole research process.
His experiences are not unique. Three months ago, we had an intern at Usabilla looking into the UX of dashboard design based on online user feedback. Her method was build to two dashboards – one based off of the current usabilla dashboard, and one new design based on user feedback.
Through face-to-face interviews, she tested these two dashboards with current Usabilla users – gauging their sentiment for each.
Despite so much negative feedback for our current dashboard, her results showed that people actually favoured the current one over the new!? Quite a contradiction. This set the alarm bells ringing on the differentiation between online and offline feedback results.
Allow your users to vent at the machine
Participants – people – are scared to truly tell you how they think. Scared to show that they don’t know what they’re doing because of your clunky interface, or scared to tear apart the fruits of your labour.
We have to bypass this cowardness to speak the truth and to uncover the source of the problem.
Our intern saw a situation where despite online frustrations, our users preferred what they currently had when we asked them in person. A fact which is quite hard to understand, but which actually makes perfect sense when we change medium:
User sentiment directly conflicts with itself when compared online to offline.
In fact, we see a 20% increase in negative feedback through anonymous online channels – a striking disparity.
Our users will will gladly scream expletives at machine, less so at a pretty little girl. Whether this is merely societal upbringing or human nature, replacing the human with a machine is effective in coaxing real feelings from our users.
—Elon Musk (Tesla, Space X, Paypal)
Negative feedback is essential. Only by understanding where our UX pitfalls and shortcomings are, can we create truly great experiences. By remaining an insensitive machine, we can take flack, discover our pitfalls, and change accordingly.
User testing should never be over reliant on just one medium, 20% of your feedback is at stake. Always run a variety of tests – online and offline, face-to-face and anonymous; always understand the big picture.