THE THIRD WAY: BETWEEN NUMBERS WORSHIP AND ARTISTIC INSTINCT

One of Google's visual designers apparently quit in despair over having to prove every tiny graphics decision with clickthrough data, instead of having management rely on his artistic instinct:

> http://www.kottke.org/09/03/google-and-design

I'd argue that both approaches are wrong. Moreover, it's a false dichotomy to choose between numbers and art.

The third way is called insight, and is found through qualitative research.

Of course, in reality, the very best design blends all 3 methods: qualitative insights, hard numbers, and pure aesthetic taste. But I think that qualitative should be the driving factor.

It's obvious why taste shouldn't be the overriding factor: countless websites look great but don't sell. You have to moderate pure art with what customers need.

It may be less obvious why the hard numbers should be secondary to soft insights. The reasons are that (a) you may measure the wrong thing, and (b) even if you measure the right thing, such as conversion rate, you can sub-optimize individual screens while undermining your long-term strategy and overall user experience.

Putting A/B Testing in Its Place: > http://www.useit.com/alertbox/20050815.html

Risks of Quantitative Studies > http://www.useit.com/alertbox/20040301.html

To take the example of which of 41 gradations of blue to pick for a design element: yes, one of these will have the highest clickthrough rate, and thus be the local optimum for the design of that one page. But it's quite possible that another shade would be better for the overall look of the site and make the complete set of pages feel more like a coherent user experience, which would enhance user confidence and the site's credibility, and lead to long-term loyalty, as more people return to buy next year.

And more important, while the best shade of blue might generate 0.1% more business than the second-best, it's almost certainly the case that there is some other aspect of the design that would lead to 50% more business if you could identify it. To do so, use qualitative research to observe deep user behaviors.

The things you know to measure are rarely the ones with the big impact. The things you DON'T KNOW are the place to focus usability efforts.