Re: Where is the data?

[Felipe Contreras <felipe contreras gmail com>]

On Sat, Aug 20, 2011 at 8:03 PM, Shaun McCance <shaunm gnome org> wrote:
> On Sat, 2011-08-20 at 18:09 +0300, Felipe Contreras wrote:
>> On Sat, Aug 20, 2011 at 6:02 PM, Shaun McCance <shaunm gnome org> wrote:
>> > On Sat, 2011-08-20 at 14:43 +0200, Tomasz Torcz wrote:
>> >>   I think his objections were justified.  There is really no raw data
>> >> at those URLs.
>> >
>> > Except Allan never claimed he was providing raw data. In fact,
>> > he explicitly said that he does not do write-ups of user tests.
>>
>> So what's the point of replying to a mail asking specifically for data?
>
> When somebody asks me for something I don't have, I usually
> respond telling them so, and explaining why I don't have it.
> I think it's rude to ignore people.

Well, that's true, but he claimed there was "lots of data". I think
that was an exaggeration.

>> > I also do user tests when working on the help. I also don't do
>> > write-ups. I fix problems or I pass information on to those who
>> > are in a position to fix the problems.
>> >
>> > Just because I don't publish reports doesn't mean I don't do
>> > user tests. And the constant assertions that nobody is looking
>> > at feedback are getting a bit insulting.
>>
>> User tests, like surveys, are not perfect and can be both misleading,
>> and not significant enough.
>
> You're right, of course. All methods have flaws. But user testing,
> at least, gives results that are personal and actionable. When I
> get results like "all 5 users were uncertain where to click when
> instructed to click the 'user menu'", I know what to do. I have
> no idea what to do with "63% of respondents report they are less
> happy than they were a year ago".

Yes, that kind of user testing gives you clear short-term actions, but
it doesn't tell you if people happy with the whole system overall. It
doesn't tell you if you are missing something essential, nor does it
tell you when you are bleeding user-base.

But you missed my point, my point is that they can be improved through
*collaboration*, you publicize them, and people make suggestion to
improve them.

>> If those tests are to be taken seriously, they should be published so
>> that they can be scrutinized, otherwise they are not evidence of
>> anything, not to the rest of the world.
>
> I agree there are problems with transparency. A lot of things
> get done on IRC, because high-bandwidth communication is great
> for rapid development. I've been a strong supporter of public
> logs for IRC. I think we should discuss ways to better record
> what we do and the decisions we make.

Indeed.

> But, I don't want to be in a situation where we have to wait
> for committees to scrutinize data and approve proposals before
> we can make changes. That sounds like an awful project to write
> software for.

I don't think anybody is proposing that. All people are looking for is
some public record. Just by doing that you might realize that perhaps
you didn't have such strong reasons to switch to something as much as
you were thinking, after all. Also, people could tell you; you are
misinterpreting the results _there_.

-- 
Felipe Contreras