Death to false user testing

September 12, 2018
 
by
 
Adam Babajee-Pycroft
,
 
Managing Director (UX)
 
, in
 
Strategy
Death to false user testing

User testing should be at the core of every design project and it’s great to see that as a best practice, it’s happening more and more in businesses up and down the country. It’s a simple enough concept - evaluating your product, software or service by testing it on users. And yet, it’s so often done wrong!

I’m on a one-man mission to stop the use of false user testing techniques. Unless you're testing with people who use your actual product, who've been recruited properly there’s absolutely no way that you’ll have the accurate, qualitative data you need to make informed decisions.

Testing with your family

“I’ll get my cousin to come in next week and run through the new site. It’ll be cheaper than recruiting externally”

Are you designing something specifically aimed at an audience which has the exact same buying habits, goals and level of experience as your cousin? No, I didn’t think so. Even if they say they won’t and even if they don’t realise they’re doing it, people who love you, like you or even just know you, will lie. They will never give you the whole savage truth for fear of upsetting you or hurting your feelings. Therefore, this is most definitely false user testing. Nothing compares to the brutality of mercenary participants. Good or bad, they’ll tell it like it is.

User acceptance testing

We ran our new software through tasks with user acceptance testing (UAT) before launching”

Professional testers and developers conducting UAT (also known as Quality Assurance) is not user testing. QA testing is done to ensure something isn’t broken by making sure that the functionality behind it is working as it should. Making sure that error and success messages show as they should, that exceptions and rules are followed and so on. That said, good QA is an often overlooked foundation of delivering good UX because let’s face it, no one likes the broken stuff!  

Unless you’re one of the very few people out there designing UAT/QA software, this is a classic example of false user testing!

Testing with your colleagues

“We took it in turns to test the product out so that we had a rounded view on it”

When’s the last time you paid the full price for the product or service the company you work for offers? Have you ever even bought it? Even if your colleagues try to look at your new product objectively, they’ll fail. Why? Because you’re completely biased. They already understand the terms used to market your product and are immersed in the company’s voice, brand and products. There is no way that they can perceive it in the same way as a potential customer.

Even if you’re designing internal-facing software, there’s an argument for testing externally because you’ll hire new people eventually and need to know that it works with people who aren’t already living and breathing your company.

false user testing with Dilbert
Taken from Dilbert.com

Showing the design to your boss

“My boss hasn’t been involved in this project at all so I asked them to test it out and share their thoughts”

Just stop. Your boss, like your colleagues, whether they worked on the project or not, is fundamentally biased. They will almost certainly have an opinion on what’s been designed and what you’re planning to deliver. They should also (if they’re doing their job right) be interested in whether the design achieves its commercial objectives.

By all means, show your boss the design but make sure it’s backed up with research and a healthy dose of user testing! And then, if your boss does have an opinion, you have the facts to back up what you’ve done and why!

Asking an expert

“Bob’s wife works in UX. Let’s get her to take a look tonight. It’ll be quicker than hiring an agency to do the research for us”

Ok, I’m sure some clever clogs out there will be saying ‘what about heuristic UX reviews’. Sure, UX experts can always spot some issues, based on their expertise and best practice but they aren’t likely to be representative of your users. Spoiler alert - you guessed it - this is also an example of false user testing.

And finally, although I’d hope that by now you’ve realised the answer to this one:

Doing it yourself

“I’ll just do it myself. I’m able to switch my mindset between design and user”

I’ve often heard people suggesting they can use their design “as a user”. Having designed it, or planned it, or developed it themselves I can confidently say that they are 100% wrong. This is absolutely false user testing!

My earlier points show why your colleagues and boss can’t accurately user test your products. If they don’t count, there’s clearly no way you can objectively use something you’ve designed “as a user” whilst you’re still in the throws of designing it. Later down the line, you’ll wonder what you were thinking at the time!

In conclusion, user testing with real, unbiased participants is the only way to truly understand whether your product works, what needs changing and what to do next. My advice? Stop running fake user tests. Spend that time wisely - testing with sorting out some real participants. Either run the tests yourself or hire someone who can. Hint hint.

Find out more about our user testing and customer research services.

Adam Babajee-Pycroft

Managing Director (UX)

Our founder Adam has over 13 years of experience in UX. He’s fuelled almost exclusively by coffee (using one of his seven coffee making devices),curry and heavy metal. Before founding Natural Interaction in 2010, Adam managed UX for AXA Life’s UK business. Since then, he’s worked with a range of clients across the automotive, eCommerce and tech startup sectors, delivering impressive results for brands including BMW, Mini, The Consortium and National Trust.

Increase your Ecommerce Conversion today

Get the book

Discover 21 ways to increase ecommerce conversion with this free eBook, written by Kathryn Davies, Head of User Research.

You may also like