file selector testing data

[Anna Marie Dirks <anna ximian com>]

Greetings All. 

I month or so (more, probably, time flies) ago, I used Ximian's usability testing facilities to run some tests of the new gtk fileselector (in its "Open Dialog" incarnation).  I thought that perhaps it would be useful to you if I sent it along. 

If you have questions or concerns about any of this, please let me know. 

1. The Raw Materials

I created a set of paper prototypes, based on Federico's design of the file selector, which I used to run this test.  The test subjects were asked to interact with the paper protoypes using their index fingers as mouses (mice?), and to think out loud during the test.


2. The Tasks

Background: You are spending the weekend at your cousin Tina's house. Tina's
daughter, Amy, is trying to use a drawing program (KidDraw) on Tina's
new desktop computer. Unfortunately, she is having a hard time finding
her and opening some of her drawings. She asks you to help her to find
her missing art files. 

Task 1: Please find and open the picture in Amy's "Drawings" folder
called "bear.jpg". 

Task 2: After looking at "bear.jpg", Amy decides that she was thinking
of another drawing. She can't quite remember what the name of the
drawing she had in mind was, and asks you to open the last picture that
she was working on. Please find the picture that Amy most recently
edited.

Task 3: You want to make it easy for Amy to find and open her "Drawings"
folder in the future. Please create a shortcut to the folder so that she
will always see it when she is using KidDraw.

3. The Testers
(Note: "Platforms" refers to what operating systems, or desktop environments, the test subject has used before. "Programs" refers to the applications that the test subject uses most frequently.)

Tester 1:
Name: Edgar 
Age: 26
Job: Theoretical Physicist 
Computer Experience: Moderate
Platforms: Windows and Unix
Programs: Web Browsers

Tester 2:
Name: Andrew
Age: 32
Job: IT Consultant
C. Experience: Expert
Platforms: Windows
Programs: Web Browsers , HTML Editors, Presentation Programs 

Tester 3:
Name: Michael
Age: 27
Job: Unemployed Software Engineer
C. Experience: Expert
Platforms: Mac 
Programs: IDEs, Web Browsers, Media Players

Tester 4: 
Name: Alison
Age: 25
Job: Barista
C. Experience: Slight (NB: I would have said intermediate for this user.)
Platforms: Unix, Windows
Programs: Web Browsers, command line utilities (mh, ls)

Tester 5: 
Name: Sarah 
Age: 26
Job: Coffeehouse Manager
C. Experience: Slight
Platforms: Windows
Programs: Web Browsers, Outlook Express

Tester 6:
Name: Josh 
Age: 24
Job: QA Contractor
C. Experience: Expert
Platforms: Unix
Programs: vi, misc tools for scripting/programming

Tester 7:
Name: Eric
Age: 38
Job: Programmer
C. Experience: Expert
Platforms: Windows and Unix
Programs: Web Browsers

4. Test Data

Tester 1: 
Task 1: 5 seconds. Double clicks "Drawing". Double clicks "bear.jpg".

Task 2: 10 seconds. Clicks once on "Date" column. Clicks a few more times, until he has a firm idea of which the direction the files are sorted. Clicks the first file in the list when he is confident that he has it sorted properly.

Task 3: 20 seconds. Clicks "Up". Remarks "This is the hardest thing you've asked me to do." Selects the Drawing folder. Clicks "Bookmarks".  

Observations:
Quite a double clicker. No preset idea about what direction sorting should default to in a column. Remarked that his memory is not "spatial". 

Tester 2:
Task 1: 8 seconds. Double clicks "Drawing". Clicks bear.jpg once. Clicks open button.

Task 2: 8 seconds. Scrolls list, scans for most recent date. 

Task 3: 10 seconds. Clicks "Up", drags "Drawing" folder to the desktop.

Observations: 
Thinks of the desktop as the most easy place for Amy to find a folder. Didn't seem to know (or care?) that he could sort columns in the Open dialog. 

Tester 3:
Task 1: <5 seconds. Double clicks "Drawing". Double clicks bear.jpg. 

Task 2: 10 seconds. Clicks once on date column. Scrolls list to bottom, selects file. Clicks "Open".

Task 3: 20 seconds.  Clicks "Back". Hesitates. Tries to right-click on the Drawing folder. Gives up. Drags "Drawing folder" to the folder list on the right. Confused about whether adding a folder to "bookmarks" would have added it to the Open dialog permanently, or just for this app.

Observations: 
"This seems perfectly functional. I would learn how to use it fast." Wanted a context menu on items.

Tester 4: 
Task 1: 8 seconds. Stares at dialog. Double clicks Drawing. Clicks "Open". Clicks "bear.jpg" once. Clicks "Open".

Task 2: 8 seconds. Scrolls list of files, comparing dates. Clicks "eddie_the_cat.jpg". Clicks "Open".

Task 3: 2 minutes. Stares at dialog for a long time. Clicks "Desktop". Looks at the files on the Desktop. Remarks that she wants to add the drawing folder here. Remarks that she likes the nav buttons at the top of the screen, because they remind her of a web browser, which she feels comfortable with. Asks to give up. (Granted). After she's given up, she asks if she can drag the folder to the left pane. I tell her that she can try. She does this, and is pleased (and apparently surprised) that she can drag folders.

Observations: 
Tended to be very, very timid and to assume that all of her actions were wrong. Actually had a firm idea of how to make the drawing folder more visible ("what i really wanted was to drag it to the desktop"), though she was afraid to try it.

Tester 5:
Task 1: 15 seconds. Double clicks "Drawing". Asks if she can double click "bear.jg". Is encouraged to try. Double clicks, 

Task 2: 7 seconds. Scrolls file list, finds "eddie_the_cat". Clicks "Open".

Task 3: 10 seconds. Uses option menu to go back to "Amy's Home". Drags "Drawing" to "bookmarks".

Observations: 
Took a (relatively) long time with the first task, seemed to adjust to the dialog's navigational paradigm quickly.

Tester 6:
Task 1: <5 seconds. Double clicks "Drawing". Double clicks bear.jpg. 

Task 2: ~ 8 seconds. Clicks "Date" columns header. Repeats. Has no idea what the default sort order should be. Checks file list to be sure that his clicking is doing something. Selectsappropriate file when it is at the top of the list. 

Task 3: ~ 8 seconds. Uses option menu to go back to Amy's Home. Selects "Drawing". Clicks "Add to Bookmarks". 

Observations: 
Very fast, very confident. For someone who doesn't use gui apps, he certainly had no problem adjusting to this one. 

Tester 7:
Task 1: 10 seconds. Double clicks "Drawing". Clicks "bear.jpg". Clicks "Open".

Task 2: Clicks "Date" column once, seems surprised that anything happens. Tries to type the name of the most recent file into the entry, asks if autocompletion works. (I say it does -- is this correct?) . Clicks "open".

Task 3: 5 minutes. Quickly gets very stressed about the scope of bookmarks. Are these per app? Shouldn't the shortcuts to folder be in the app's preferences somewhere? I asked him if he wanted to try the task using the dialog I provided. Looks at the open dialog for a while. Tries opening but not selecting anything from the option menu. Gets upset about the scope of bookmarked folders, again. Finally tells me that the task is impossible. 

Observations: 
Very unwilling to experiment with dragging or moving files. Was rather upset by not knowing if the drag of a file would result in the file being moved, or in a ink being created. Argumentative.

5. Conclusions
So, what conclusions can be drawn from this data? The following come to mind:

• No one had any real problem finding, opening and searching for files.
• Test subjects didn't have a clear idea of what the scope of the "bookmarks" folder should be. Per app? Global?
• Most test subjects conducted searches by sorting the columns.
• The two test subjects who told me that they had the least computer experience both also remarked that having nav buttons at the top of the window (web browser style) helped them to understand how to navigate.

So -- not very dramatic results, were they? I do hope that something in here is useful to you, in any case. 

best wishes,
Anna

--
Anna Marie Dirks <anna ximian com>