Re: [Usability] user levels, etc.

[Adam Elman <aelman users sourceforge net>]

On Monday, November 12, 2001, at 05:44 PM, Luis Villa wrote:
> User testing suffers from strong systematic biases. The main ones are
> twofold:
>
> It has a selection bias towards the uneducated and inexperienced (the
> educated and experienced have better things to do than sit and play with
> boxes while being watched.) This means that what user testing measures
> is not 'what is best for users' but 'what is best for the users who
> spend less time in front of the computer and are generally clueless.' We
> should work on making GNOME easier to use for these people- but we
> shouldn't assume that because they show up for user testing that they
> are the complete universe of computer users.

Have you ever actually done or attempted to do any user testing?

> [And please don't take the comments about education the wrong way;
> anyone who has tried to do political polling will tell you the exact
> same thing.]

Sorry, this is incorrect.  I have done quite a few usability tests, and 
I have never had any trouble recruiting people who are both educated and 
experienced.  You offer them money, you offer them food, you offer them 
the chance to help improve products that they're interested in seeing 
improved, and in fact you'll find that they are willing to help.

> User testing also prioritizes short term UI issues over long term UI
> issues. Not to harp on this example, but... everyone I know who has
> tried using ~/ as their desktop has found it vastly superior (from a
> utility perspective) and stuck with it. User testing won't reflect that.
> It'll reflect that when given 30 seconds to find something in the
> configuration screen, the additional option of using ~/ instead of
> ~/Desktop clutters things up.
>
> Because user testing is biased towards these things, it is not and
> cannot be the end all and be all of UI design. A strong notion of common
> sense has to be injected, and many of the things we're talking about
> here don't pass that second test.

User testing is one tool in an arsenal of software design tools; other 
tools include user and task analysis, which attempts to understand what 
users are trying to do and designs software to help them do that.  
Common sense is another tool, and I very much disagree that many of the 
things we're talking about here fail that test.

The ~/ vs. ~/Desktop is, indeed, a case that short-term user testing 
doesn't really show.  That's one where some good user and task analysis 
would help designers to understand _why_ exactly you and your friends 
find ~/ as your desktop directory to be more useful, and how the 
software could be improved to make it even more useful.  That doesn't 
necessarily imply that allowing users to choose any arbitrary directory 
as their home directory, or even allowing them to choose between ~/ and 
~/Desktop, is the right approach -- perhaps those who chose ~/Desktop 
were wrong, and _everybody_ should just have ~/ as their home directory?

The latter seems unlikely to me, but you get the idea.  User testing 
should not be the end-all and be-all of UI design, but neither should be 
your own personal preferences of how things should work (which is very 
different from "common sense," as I'm sure you well know. :)  There 
_are_ ways to get this kind of information by observing real users, even 
hackers.

> [...]
> More importantly, and this brings us back to where I started, I don't
> think we have to make a choice about being people-friendly and being
> hacker-friendly. It's a false dichotomy; it's the lazy way out. There
> are very compelling reasons to work past that and allow both camps to
> coexist, and so we should.

Absolutely.  And to do this, we need to use all the tools in the design 
arsenal.  We need to do user testing on representative users, including 
hackers.  We need to do user and task analysis on users.  We need to 
_design software for users_, not just hack stuff up for hackers.  The 
facts are simple:

1) Hackers are people.
2) Not all hackers are "advanced" users in every way.
3) Not all hackers are the same.

Making stuff more usable for hackers will make stuff more usable for 
anyone, and vice versa.  It's a false dichotomy.

Adam