5 Effective Ways for Usability Testing to Play Nice with Agile
Customer-Centricity | Industry Savvy

5 Effective Ways for Usability Testing to Play Nice with Agile

on / by Jeff Gothelf

Usability testing has been a fundamental tool in the UX arsenal for decades now. The value of actually meeting your customers and letting them experience your product makes a significant impact to the shape of that product. In it’s most formal version, testing can be a multi-day, multi-thousand $/€ process that delivers final analysis days if not weeks later. With many organizations moving to an Agile philosophy and methodology, UX practitioners are finding it difficult to integrate formal usability testing into this faster-paced, iterative approach to software development.

See? Lions and zebras can get along. So, too, can Agile and Usability Testing.
See? Lions and zebras can get along. So, too, can Agile and Usability Testing.

The main challenge is time. The need to keep the teams moving forward, coupled with typical two-week development cycles between releases (aka sprints) doesn’t provide an obvious point to insert testing. Indeed, fitting a formal usability testing cycle into a series of two week sprints is challenging if not impossible. Instead, consider the following techniques to help get that critical customer feedback and still keep your team’s momentum.

1. Start small.

Jakob Nielsen has proven through scientific method that 5-8 is the optimal number of participants for qualitative tests. Consider testing only 3 people each time instead. The goal of agile usability testing is to clear the boulders from the road – not to provide pixel-level design direction. Three participants will, very quickly, validate (or not) your product’s core workflows and give your team a sense of where to focus next.

In addition, three participants take less time to test than 5+. Whereas a traditional rubric would have a team tied up for a day or two, a 3-person study can be completed in a morning. This is especially helpful if you don’t have a dedicated research staff. Your UX generalists can test, get feedback quickly and be back optimizing the design within one day.

Finally, three participants are cheaper. There is greater value in testing multiple iterations on more customers than one design on many customers. It allows you to move forward more quickly and keep your Agile team’s momentum.

This approach assumes you are performing moderated usability testing in the context of a lab, work site or some other location. As an additional data point, consider using remote usability testing as a way to provide additional perspective on your findings.

2. Test *every* week.

Pick a day (preferably not Monday or Friday) and make that your “testing day.” Every week, on that day, you’re in the lab, coffee shop, or customer site testing. Make it clear to your teams that this is testing day. Put recurring meeting invitations on your calendar and forward out to stakeholders. The night before testing day, send out a reminder with a participant schedule.

The goal is to make that day synonymous with testing. Teams should stretch to get work into that day’s test and use the cadence to drive productivity.

What ends up happening is the organization becomes accustomed to a steady stream of qualitative insight coming from the UX and product teams. Testing day becomes a target for quick-turnaround validation and the team becomes almost reliant on that stream to ensure the quick decisions made at iteration planning line up with business and user goals.

In weeks where testing day gets de-prioritized (hey, it happens), a concurrent micro-study using an automated online tool can maintain the flow of insight until you get back on track.

3. Show them whatever you have ready.

Customers will react to whatever you put in front of them. Yes, the reaction will differ based on what that item is or it’s fidelity but you’ll get *some* feedback — and that’s much better than nothing.

If you have code in production or a qa environment, show it. If you have wireframes or mockups, show them. If you can string together a series of mockups in fireworks, visio or even PowerPoint, do that and show it to real customers. A napkin sketch is also viable. Anything that will get your ideas in front of users will provide value. Even if you have nothing, it’s still worth sitting down with a customer and talking about their pain points.

4. Invite everyone (or bring the testing to them).

*Note: this tip assumes you have some kind of lab onsite. If you don’t, and typically do remote usability testing, guerrilla testing or site visits, point #5 below is most crucial to your success.

Key to getting the findings from these weekly tests into the iteration is buy-in from the rest of your scrum team and the organization as a whole. The most effective way to do this is to get the engineers, product managers and stakeholders to the testing sessions.

Most testing software broadcasts over your local network and if yours doesn’t you can always use Webex or GoToMeeting to broadcast. Setup multiple viewing areas convenient to your team. Ensure they know where to go and how to turn viewing on.

Another option is to enable viewing at each person’s desk. Again, most usability and general screensharing software allows this. This is most convenient as it doesn’t require the team to go anywhere to participate.

After viewing one or two customers the team gets a sense of where the problem areas are leaving you free of the responsibility of convincing them later. In addition, most team mates start to like the immediacy of the feedback on their work and start attending regularly without much reminding.

5. Report back quickly.

This one is critical. Turn feedback around to your team and to the organization as quickly as possible. No need for a fancy report or designed layouts. Type up a quick executive summary followed by recommendations for the pain points discovered and send out via email. Many tools offer some kind of findings visualization that allows you to, very quickly, generate a scatter plot or heat map of user activity. Ideally you should keep a copy of the findings in a commonly accessible place like a wiki or shared folder.

The goal here is twofold. One, you are getting customer feedback to the team so they can adjust their work accordingly. The other goal is to tell the broader organization that, despite moving quickly, you’re validating your approach with real people.

*5.5 – Use an outside recruiter.

Getting the right customers to your studies is critical to the validity of your findings  Keeping up the pace of recruiting participants to your studies on a weekly cadence, however, is a time consuming endeavor. It is highly recommended that you find a resource external to your team to perform this task. If you can afford it, hire a recruiter who specializes in this task. If not, minimize time on task for this by spreading the work across the team using a shared calendar.

Agile development practices pose some challenges to traditional usability testing methods. However, with a bit of modification and a focus on outcomes as opposed to output, getting to validated learnings can be successful and rewarding.

What’s been your experience? Share your feedback in the comments.

This is the first guest post on our blog, and we plan to do more. Interested in writing a post for us? Please contact loucas@usabilla.com

| | | | | | |
Article by

Jeff Gothelf

Jeff Gothelf is a user experience designer, blogger, speaker and Lean UX advocate based in metro NYC. He has spent his 13 year career defining and designing engaging experiences for clients big and small. He is currently the Director of User Experience at TheLadders.com where he helps executive jobseekers and recruiters make meaningful connections with each other. Previously he helped shape the designs at Publicis Modem, AOL, Webtrends and Fidelity. Jeff publishes his thoughts on his blog and on Twitter @jboogie.

Share your thoughts

  • Great advice — especially focusing on outcomes — not output in the Agile context. Responding to change, rather than following a plan…

    We have revolving-door UX sessions weekly with user surrogates face to face. We have 10 people who are committed to 15 minute sessons on the half hour every monday. That way, we can divvy up the projects and each of the 3 scrum teams gets at least 3 sessons. We also swap our participants around to avoid fatiguing them with the same project. Generally, we are validating ixd concepts with Balsamiq click-thrus that represent our “happy path hypothesis” We record those sessions with Silverback. The UX person gives the report out immediately when the sessions are done and archives the recorded sessions so that we can do further analysis later.

    We also do longer bi-monthly sessions with Real Users (customers who have signed up to be “UX Lab Rats”. These are 45 minute sessions — remote testing. I use WebEx and share an application — live code or a prototype. We record it — but also encourage developers and product managers to dial in (stay mute) and observe real-time.

    This program has been the most influential in getting people to understand that what UX and UCD is all about — people using the stuff that we are building to solve their problems.

  • This is great stuff. Quick and effective usability feedback is so important yet many people seem to forget to do it at all. Your product and approach sounds very relevant within an agile project. There is of course room for different types of usability test, including things like our own Kupima product and other similar services, which can augment and enrich the feedback you would get from something like Usabilla. The great thing about the web is that nobody is forced to just do things one way – variety of approach can be a very useful strategy to adopt.

  • Really interesting article. We’re currently going through this process, integrating usability testing into this process.

    We’re working with an agency who are delivering the high-fidelity Axure prototypes according to the scenarios developed to test function and flow. One of the biggest time stealers in this actually seems to be the production of the prototype and all the different screens to meet the scenario requirements.

    As usability testing (especially with a lab) is expensive you want to be getting the most out of each session which is why we’re working with hi-fidelity mockups.

    We’ve worked with our agency (who have been great) to streamline this and make it more manageable but it would be interesting to see what you recommend for streamlining high fidelity prototype production to lead into usability testing?

  • Dean Morrow

    In response to Graham – ditch the high fi prototype, test whatever you have. See Jeff’s text above.

  • james

    I like the testing approach. I mean it is obviously different if the developer is dog-fooding it, as they are going to make routines and interfaces which suit them best, so getting outside opinions, ones from outside the customer base, and outside the organization, count.

  • Adrienne Durand

    I think this post has very little about using Agile with Usability testing – I think this isn’t a good post by the standards of this blog.

    1) You can use remote usability testing
    2) Do it so it fits in with your Scrum/Agile cycle

    This seems very obvious to me. Am I the only one who thinks this?

    The ideas seem also very similar to tips I have previously read post here: http://goo.gl/YrfS6

  • @Adrienne
    Remote usability is definitely one of the tools of trade for rapid tests. There’s no such thing as THE usability testing tool, so depending on what you’d like to learn you pick a tool (offline, online, functional, qualitative, quantitative, etc). What I personally like about this blogpost is a broader take on how to fit tests into development cycles. Quite some organizations I meet are struggling with this theme.
    Take a look at this video, which is a nice example of Mindflash (http://mindflash.com), which includes both qualitative offline tests and remote usability test (including Usabilla tests) in their development cycles: http://uxweek.com/2010/videos/video-2010-paula-wellings-cameron-gray

  • Carlos

    I have to agree with Adrienne, not the best post I’ve seen on the topic – I would like to see a lot more detail. This post doesn’t tell me much at all. I think the Smurf one was much better! I guess there is some similarity between the posts too.

    Paul, you make a good comment though, I know a lot of organisations who simply don’t see a way to fit in their testing. Quick, simple tools make it more realistic for them to test on a regular basis. Old fashioned testing is simply too time consuming and expensive. I would love to know more about your experiences with large organisations.

  • Pingback: Lärdomar från ett agilt UX-projekt « Helt sonika()

  • Pingback: 5 Effective Ways for Usability Testing to Play Nice With Agile « UXasm()

  • Fundamental to success with agile practices being successful is continuous testing. While continuous integration is something most Agile practitioners adhere to continuous testing is equally if not more important to ensure there is continuous delivery of solutions that meet the business requirements of cost, time and quality.

  • Pretty nice post. You can write in more details but still I liked your article. Thanks for sharing.

  • Pingback: Morae – apoio a testes de usabilidade « Enio de Aragon()

  • Pingback: #User Friendly 2011#敏捷开发与用户体验 | 落花流水——elya妞╰_╯()

  • regarding 5.5 – I founded a site elusivestars.com which should be a great tool for finding test candidates. The service is specific to Mobile applications. It uses a crowdsourced model and provides the ability for filtering candidates using device and demographic information.

    The reason I believe the crowdsourced model is a great fit for agile testing is the ease and speed which test candidates can be engaged.

  • What would you say is the difference between usability and user experience? Are they they same thing? It seems that most organizations don’t really plan to test for user experience or usability. Do you have any methods other than the usability lab aka jacob neilsen methods with greater than 5 users?

  • @ software usability

    Basically, we see the user experience of a product or website as something that follows the usability. The usability describes that something is useful and usable, that it allows us to accomplish a task effectively and efficiently at the same time. The user experience goes beyond that. It involves our emotions and personal perception, the way we think about the product during and after usage, whether or not we are going to tell others about it, etc. In order to create a great user experience, you really need to know what makes your users tick and how you can really get them engaged.

  • fantastic submit, very informative. I ponder
    why the other experts of this sector do not realize this.
    You should proceed your writing. I’m sure, you’ve
    a great readers’ base already!

    • Thanks for your kind words. It’s always good to hear that people enjoy our content.

  • Daniel Maddux

    I struggle to reconcile what I think of as usability testing with my company’s business model. We have a massive software package that takes a good deal of work to set up, and a limited user base (operators of large convenience store chains). We can’t just ship a bunch of customers a piece of software, or even a single feature, and expect them to be able to use it. I suppose that extensive beta testing can take the place of some usability testing. Do you have any thoughts on performing usability testing in this kind of environment?

Pin It on Pinterest