Feature:Linux Usability Testing

Jeremy Arnold has written an essay on what he calls LUTE- the Linux Usability Testing and Evaluation project. It could help make programs more usable by organizing volunteers to test software and clean things up. Far to often great programmers aren't the best at creating the ideal user interface. Perhaps this project could put people who understand the human factors in touch with the guys who write the killer code. Hit the link below to read it and throw in your 2 bits.

The following was written by Slashdot reader Jeremy Arnold

The LUTE Project

Linux Usability Testing and Evaluation

Last week on Slashdot an article was posted about an essay on user interface design. As a comment to that article I brought forth an idea for a group of people who would do usability testing for Open Source Linux projects. The response to the idea was generally positive, so I have now prepared this more formal proposal. If I can find a place to host web pages and a mailing list and Slashdot readers agree there is a need for this group and some are willing to volunteer their time to be part of it, the Lute project should be running shortly.

What is the Lute project?

LUTE stands for "Linux Usability Testing and Evaluation". A bit redundant, but I like acronyms that form words. :) Usability testing is the process of testing the user interface for a piece of software to make sure that it makes sense to the user. "Makes sense" refers to the interface being intuitive and consistent, and generally that the user interface helps the user rather than frustrating them. Note that there is a difference between "usability testing" and just plain "testing". Lute does not exist to find bugs in software. The purpose is to find flaws in the user interface.

Most commercial software development includes a usability testing phase, and the most usable software generally includes usability testing throughout the development cycle. Large software development companies usually have a "usability testing lab" with lots of (somewhat) fancy equipment to aid them in this testing. This, of course, translates to money, which most open source projects do not have. Usability testing can be performed (though somewhat less effectively) with no special equipment, but it is an area which many open source developers are unfamiliar with, and most do not take the time to do it.

As Linux becomes more accepted in the commercial world and starts to be used by more "common users" rather than the traditional hacker types, software usability will become more and more important.

The Lute project is an attempt to address these issues by performing usability testing for open source projects. This testing will be performed for free by Lute volunteers. On the web, Lute will also serve as an information center for good user interface design as well as techniques for usability testing and evaluation.

Lute will not force developers to use certain toolkits (such as KDE/Qt or Gnome/GTK). In fact, when testing a program, Lute volunteers won't even be looking at the source code. In addition, Lute will not enforce strict standards (like forcing you to use 5 pixels between buttons and 10 point bold Helvetica font for menu text). Upon completing testing, the Lute volunteer wil l give an evaluation back to the developer about what aspects of the interface work well, and, more importantly, what aspects don't work well. The evaluator will also try to give the developer ideas on how to improve these aspects. It is up to the developer to choose whether or not to implement these changes, and how to do it.

How will Lute work?

These details are subject to change, but this is how I envision Lute working. The developer will submit a request for an evaluation of his/her project. (Of course, most open source projects have multiple developers. In this case, one developer would act as the liaison to Lute.) This request will include a short description of the program, a list of features to be tested (sometimes this will be the whole program, other times just a few features may need to be tested), and a developer contact (email address).

Once a request has been submitted, it will probably be appended to a web page list of pending projects and sent out on a Lute mailing list. Lute evaluators can then choose to accept the project. The evaluator will write up a list of tasks to use for the test. These tasks are the things that the user will perform in order to give the evaluator a good feel for what works well and what doesn't. After preparing this task list, the evaluator will discuss it with the developer to make sure that the developer thinks it adequately covers the expected uses of the program. The evaluator would then find about 3 "average users" to test the program with, and report back to the developer with the findings.

Note that the definition of an "average user" could be a bit different for each project. For example, the average user of a programming IDE would be a programmer, while the average user of a web browser might be the evaluator's mother. Also note that the average users will generally be somebody the evaluator knows, and they must be able to meet in person to perform the evaluation. (If you disagree with this condition, go read some articles/books on usability testing and then post an informed comment here.)

This "formal testing" process could take a bit of time, and is probably overkill at times, especially when usability testing is performed regularly throughout the development cycle. In this case, a full formal test should certainly be performed from time to time, but at other times the developer might just need an "expert opinion" about how the usability of an application could be improved. For this, the developer might put in a request for an "expert opinion" (or perhaps this should be "somewhat informed opinion"), at which time a Lute evaluator could volunteer for the project and then look at the program and make recommendations based on their prior experience with and knowledge of user interface design and testing. Note that this is a supplement to the formal testing, and not a replacement. The formal testing can be much more effective; it just takes a bit more time and effort to do.

What roles are involved?

Here is a description of the various roles which need to be fulfilled for this to work:

• Project Maintainer

  I am currently this person. I will be in charge of setting up the infrastructure for the project, including things like web pages and mailing lists. I will also be in charge of facilitating communication between Lute volunteers, and make the final decisions in policy matters. Most open source projects have some type of "benevolent dictator". Lute's "Project Maintainer" is the same type of thing.

  In case somebody cares, I am Jeremy Arnold ([e-mail]). Last spring I graduated from Utah State University with a bachelor's degree in Computer Science, and I now work as a Java Performance tester for IBM. I have little formal experience with user interface design, and my usability testing experience consists of about a week doing usability testing on a program I developed during a summer internship. I certainly don't claim to be an expert in this area, but it is something I find very interesting and in many ways it comes naturally to me. I have been a Linux user for about 4 years now, although I have never really (previously) taken a very active part in the Linux development community.

• Evaluator

  These are the people who will do most of the real work for Lute. They will then volunteer to evaluate the programs submitted by the developers, and report their findings back to the developer.

  While I don't expect the evaluators to have years of experience in usability testing, they will be required to have at least some knowledge in the area. At a minimum, this will probably mean they have to read some on-line information about it, and perhaps for their first project or two a more experienced evaluator will also evaluate the program and then give some feedback to the new evaluator about any areas they might be able to improve in. This is not meant to scare off people who would like to help; I just feel it is necessary in order to provide quality feedback to developers and not lose credibility for Lute. Anybody who has done some work with user interface design and knows the challenges involved could probably pick up the necessary knowledge to be an evaluator pretty quickly.

• Developer

  Developers are, of course, the people who are developing open source projects. More specifically, they are the ones developing open source projects with user interfaces. This does not necessarily mean graphical user interfaces, although most programs evaluated by Lute probably will have GUIs. Any program that interacts with the user has some kind of user interface, and can be evaluated by Lute.

  Lute will only evaluate programs at the request of a developer. The programs to be evaluated must be open source (as a matter of principle....I am not against closed source development for some projects, but Lute is really designed to help open source developers) and freely available (if the developers are charging users for their product, then they can afford to pay somebody for usability testing). I refuse to get into a debate about whether or not KDE is really free, but programs written using KDE/Qt fit the definition of free used in this paragraph, and are eligible for Lute evaluation.

  Lute will encourage (and perhaps require) the developer to read some general information on user interface design before we will evaluate their program. This is simply because it is a waste of time for us to evaluate a program that has no consideration for well-known design principles and then report on all the ways their program violates these principles. Any required reading will be as short as possible.

  Programs to be evaluated must have a reasonably functional interface. It is best to evaluate the user interface of a program before it is completed, so that the framework can be fixed before a lot of code is developed over it. However, even if the program is not complete, the user interface must be functional. If your program has a button labeled "Find a cure for cancer and generate world peace" (not necessarily a good name for a button), it doesn't have to actually find a cure for cancer and generate world peace, but if I press that button it should pop up a window that says that the cancer-curing peace-generating functionality is not implemented, rather than just sitting there or crashing the program or something.

• Average User

  The evaluator will test the program by observing average people use the program (performing the list of tasks given by the observer). These users are very important, as they can show the evaluator the types of difficulties that most people will have when using the program.

  Average users are not direct volunteers to Lute as the evaluators are. This is because evaluators need to be able to watch the users as they run the program, which means they must be geographically near the person. Because of this, evaluators will generally find average users among the people around them. Luckily, average computer users are pretty abundant in the world, so this shouldn't be a problem.

  Besides the need for geographical proximity, volunteers to be average users would not be desirable. If a person is an "average user" for 50 usability tests, they are no longer average. That person should consider becoming an evaluator.

• Web page/mailing list host

  As mentioned far above, the Lute project is also in need of somebody to host the needed web pages and mailing list(s). If the project becomes popular, I could afford to get a permanent connection to the Internet and use my machine as the host, but I would rather not spend that money until the project gets moving and looks like it is going to succeed. CGI (or perhaps Java servlets?) access on the web host would be really helpful, as I hope to get much of the process of submitting requests for evaluation and stuff automated. Even if the project does become quite popular, I wouldn't expect the mailing list to be extremely high volume or the web pages to need high bandwidth (except of course if the URL was posted on Slashdot).

Comments? Questions? Suggestions?

Post them here, and I will try to answer them. A lot of the details remain to be decided, most likely once there is a team of Lute evaluators. Note that all of the policy and procedure stuff in this article is not set in stone. It is just my thoughts on how things would work best. Like I said, I don't claim to be an expert, and it is quite possible (and even likely) that things could be done better.

"I want to help. What do I do?"

I'm glad you would like to help. If you would like to be a Lute volunteer as an evaluator, web/mailing list host, send me email ([e-mail]) and let me know. Personal replies are unlikely for evaluators, but I will let you know once the mailing list is set up.