Budget In House Useability Testing

[wige]

Usability testing is an important element of the web site design process, as it reveals problems that prevent users from accessing the site effectively, and reduce the site’s effectiveness. However, usability testing seems to be rarely done because of the complexity of the testing and the costs involved.

I am about to begin some usability testing and wanted to post some of the setup and logistics we plan to use for an informal, low cost, in house test process, both to get your suggestions on how we might improve this process, and hopefully to provide this information to other small web businesses so that they can use this procedure to improve the user experience on their own web sites.

I have seen some professional usability labs used for this process, that involve a camera on the computer screen, a camera on the user’s face, key logging software, screen logging software, CCTV and one way mirrors. We are going decidedly lower tech, with a private area, free from distractions, with a comfortable computer setup and a place for the observer to sit. Since our budget doesn’t include money for extensive recording, we will be focusing on notes to record each session. The computer will be running Microsoft Virtual PC, which is free and lets us set up the browser in a way the user would be familiar with, and which can be secured from the rest of our network. VPC also lets us reset the user session back to a default state after each test at the click of a button.

For recruiting, we will be posting ads in local business publications, as most of our customers are professionals in our area. We have a budget that will allow us to test about 3-5 users a month, which should give us enough of a sample to find and correct the most pronounced problems and then in the next test cycle test those changes and find any other problems.

The test format we plan to use will be taking the users through the ordering process starting from a SERP landing page or from the home page, using the interests of the user to promote a more natural flow through the order process. (IE, if we sold sports memorabilia, we would find out what players or teams the user is interested in and use that information to determine an ideal starting point from that user in hopes their behavior would be more natural and less guided)

All in all, we expect that, including reimbursing participants, this process will cost under $500 per cycle (group of five testers), and for practice we will be starting with the free pool of testers known as friends, family and coworkers. I know this won’t get the in-depth results that hiring a professional usability consulting firm for several thousand dollars would, but we hope that by following a philosophy of “better to miss some things, than miss everything by not looking at all” we will be able to markedly improve our customers’ experience on our site, and improve our conversions in the process.

So, that’s our plan. Any feedback or additional suggestions or personal experiences with this type of small-sample in house testing would be most appreciated.

[Dcrux]

I have seen some professional usability labs used for this process...

Then you probably have more real experience than many who are going to opine on usability. I've seen similar setups at companies like Progressive Insurance, so I know what you're talking about.

I use Progressive, and not other companies because of the way they approach the problem of cost. They host the local UPA group NEOUPA, and other groups so they have a pool of potential users who fit various profiles they need.

Progressive doesn't even have the "Two-Way" Mirror or VPC.

As far as the several thousand dollars thing, I know several interaction designers and usability people who charge from three-hundred bucks. This is probably the biggest misconception about the field.

Progressive runs their test as a profit-center. (I don't know if it's literal profit as I don't know what they were charging ...but they were charging and getting money). Suffice to say, if it's costing you money, you're probably doing it wrong -- on a number of levels.

But that's why Progressive is a big company, and the reason so many people quibbling with contractors for pennies aren't.

Finally, I would look at the ad you're running. For example "planning 'X purchase" within the next month or season or whatever fits." Depending on what you are trying to find out, how you select your audience could be very important. And how you write your ad will determine who you get.

But so far, you seem to be approaching it several orders-of-magnitude better than most. I would look toward using your practice sessions as a shake-out and you should do fine.

Related:

Creating a Useability Testbed for Free. Note the errant "e" and you're pretty much up to speed on Usability on this forum. It's not about a usability testbed -- has almost nothing to do with usability -- but what you're doing is in fact a genuine usability testbed. And it is far from atypical. Just something to keep in mind most people literally can't spell usability - let alone do usability testing. But hey, free -- how expensive is that?

Return on Investment for Usability Once again, if it's costing you money, you're doing it wrong. And like so much else, the most expensive things about usability are the numerous misconceptions about usability -- not the fee.

Cafe Testing Like for almost everything, price is the first objection for usability or interaction design testing. Of course, it's an excuse and false objection -- Cafe testing shows why.

And basing the price of hiring Jacob Nielsen as the average of the industry is like taking the highest paid athlete in a field and extrapolating every athlete gets that much. Happens all the time. Isn't true.

[wige]

Actually, I wrote the first link you mention in preparation for this one. Although I put that post more in terms of testing for compatibility, it is similar to what I would use for my usability testing, because it will give users a desktop environment that is familiar, it is more secure, and it can be reset for each test so every tester is exposed to the same environment. And I also think compatibility testing is a part of usability - if the site doesn't display, after all, its not very usable.

Now you mention that Progressive Insurance runs their usability testing in house as a profit center. Is this because they contract out their testing services, or they because they credit increased revenue resulting from the improvements made after testing testing to the department? Ideally, if you look at usability testing from an ROI standpoint as long as you find issues needing correction the testing should give a net profit through an enhanced user experience.

In my case, the company I work for conducts business primarily over the phone and through our store. We had a web site that was created 9 years ago and left unedited until last year. The site is only now starting to become a major part of our total revenue, but still accounts for less than 10% of orders. In this case, it is hard to make the business case for hiring outside consultants and a formal usability study. (Why invest there, when we can invest in growing places that drive 90% of our business already.)

This is one of many cases where the DIY approach is beneficial. In my case, I will be able to find issues that should help improve the user experience on my site, and as the conversion rate of the site increases and the site makes up a larger percentage of the company's revenue, I will be able to make the case for a more professional usability study. And, I would probably continue doing informal in house testing as well as the site grows and evolves, just to be sure I stay on the right path.

[Dcrux]

And I also think compatibility testing is a part of usability - if the site doesn't display, after all, its not very usable.

True enough. But far too often that's where it stops, at compatibility. And there is an unfortunate tendency to assume a train of thought, because something can be seen the usability job is complete. As your next part here shows, it's barely getting started.

Without the kind of control and feedback and proof testing provides, there isn't a business case for the original site. Other than the usual one, "throw something on the web and see what happens." Performance procedures like usability are basic to the definition of a business case. That's not for justifying usability -- for justifying the original website.

You make the business case from the revenue figures comparable competitors are getting.

Many websites were developed "to have a site." Now the test methodology has to jump through the hoops the original web design didn't.

Related:

A Business Case for Usability.

Online Competitive Analysis Tested How analyzing your competitors can identify your strengths and weaknesses and strengthen your bottom line. Competitive analysis -- again, websites got a free pass -- is the basis for many interaction testing programs. If the competition is making three or five or ten times more from their site than you are, something may be wrong.

Conducting a competitive analysis is an important part of the job if you're a usability engineer or information architect. A good competitive analysis not only produces usability metrics but also aids decision makers in their strategic goal-setting and planning. Done right, a good competitive analysis can steer a Web development project in the right direction.

-- How to conduct a Web site competitive analysis

In other words, if you don't have a competitive analysis you don't have a business case for justifying a website. At this late date it is easy to advocate for usability, the business cases already exist. The most expensive way to get a business case is to use yourself as the case. The least expensive way to use business cases is to learn from the experience of others. That's why you hire people who can get you the business cases.