Do-it-yourself assessment

Once you have invested valuable time and resources in setting up and maintaining a course Web site, it is well worth the effort to evaluate its effectiveness. Evaluation can reveal unexpected results: perhaps your students are using only specific areas of the site, or perhaps they're using the site in ways you had not anticipated, or perhaps they're not using the site at all. This knowledge, though potentially disheartening, can save you from misspending time and effort. Understanding the strengths and failings of your approach will allow you to adapt your strategy and to make best use of the medium.

Two main areas of your site warrant assessment: its usability and its effectiveness as a teaching tool. Usability testing is a method for evaluating the effectiveness of your site's information architecture, navigation, and design. It measures how successful users are in locating information on your site and how they felt about the experience: Were they lost or derailed at any point in the process? Learning assessment measures the effect your site has on the learning process: Did students learn the materials as presented?

There is great demand for substantive evaluative studies on the effectiveness of the Web as a tool for teaching. Some institutions are investing heavily in such studies, particularly when the Web is used in place of regular classroom teaching. Formal assessment is complicated, costly, and time-consuming, however, and it must be done by professionals to ensure experimental rigor. For projects with limited resources, consider conducting an informal assessment study. You aren't likely to reach a scientific conclusion about the overall effectiveness of computer-based teaching, but you will gather useful information about the effectiveness of your course Web site. With this information you can structure your further development efforts.

Establish a testing environment

To test a Web site, all you really need is a quiet room with a couple of chairs and a computer. Make sure the computer is fully configured so that users won't encounter technical difficulties during the test. Choose your browser software, install all the necessary plug-ins, and make sure there is enough memory available on the system. A nonessential but highly valuable addition to the environment would be a video camera and tripod to record the testing sessions for review purposes.

Define your objectives

You can achieve a number of objectives through informal testing. You can measure the success rate for participants using your site to perform a specific task, such as finding assignment information. You can measure the time participants take to complete the task and their overall satisfaction with the process. You can also test learning outcomes for certain tasks - whether the participant adequately understands the materials presented.

When defining test objectives, emphasize the tasks that you expect your students to perform most often. Use this opportunity to test areas of your site you are unsure about, for example, complex simulations or interactive features. Also focus your attention on aspects of your site that you feel are critical to your students' understanding of the course content.

Test objectives may include:

Is the site navigation effective?
Is it clear what materials are offered on the site?
Does the media content take too long to download?
Is the textual content read online or printed?

Develop a task list

Next, design a set of tasks for the test participants to perform. These tasks should expose the participants indirectly to the areas in question. For example, one objective might be to test the effectiveness of your site navigation. Instead of pointing out the navigation and asking users to explain how they think it works, create a task that requires use of the navigation, such as, "What reading is assigned for week 7?" Then observe the participants' use of the site navigation to complete the task.

Write a task list, noting the task, how long participants have to work on it, and what you consider successful task completion to be.

Sample task list for ChemLab site
Fact retrieval	Time	Success
How do you contact your TA?	2.5 minutes	Correct answer
When do you need to turn in your lab notebook?	2.5 minutes	Correct answer
What do you do if you get chemicals in your eyes?	2.5 minutes	Correct answer
Under what conditions would you use a buret?	2.5 minutes	Correct answer
Online activities
What do you get if you mix barium chloride and baking soda?	5 minutes	Correct result
How do you make an ion exchange column?	5 minutes	Explanation of procedure

Create a test plan

Next you should establish a procedure for the test sessions. The nature of your test objectives will determine your method. If all you want is a general sense of how students work with the materials on your site, you can simply set participants down in front of the Web site and observe and record their interactions. If, however, you have specific objectives, such as determining whether students read the instructions before using an online quiz module, you will need to structure the test sessions more formally to ensure that your objectives are met.

Consider the following when constructing a test plan:

Participants. Because the purpose of this study is to gather insights, not statistics, you can use a small number of participants. One participant would yield useful information, four to five would expose most of the flaws in your site design. More is certainly better, but much of what you need to get accomplished can be done by testing a handful of users. Gather participants that have diverse profiles - different learning styles, familiarity with the Web, ages, and genders - but are representative of your typical users.
Test monitoring. A test monitor facilitates the test sessions by initiating and timing the tasks and logging data, such as how long each task takes, as well as any errors or other observations. The monitor can also be available to respond to or guide the participants. It is generally better to have an objective person handle the test monitoring. If you decide to monitor the test for your own course site, remain impartial and focused on the task at hand.
Test materials. Including such test materials as scripts and checklists can help with monitoring the test. For example, you might have the test monitor read an orientation script at the start of each session so that all participants hear the same explanation of the purpose of the test. You could also create lists that specify which data the test monitor should collect, such as "number of errors" or "time spent on task."

Sample test materials
Test material	Purpose
Orientation script	Read at the beginning of the test session to describe the purpose of the test and outline how the test session will proceed.
Questionnaires	Pre-test: Gather information about the participant's background and comfort with the Web. Mid-test: Measure the participant's feelings about the site, particularly any frustration or confusion. Post-test: Get feedback about the participant's overall impressions of the site.
Task list	List the tasks to be completed during the test session, with the amount of time allotted and a measure of successful completion.
Data log	Record such things as the amount of time spent on a task and the number of incorrect actions taken. Also use to record general observations during the test session.

Thinking aloud. When test participants "think aloud" they narrate their thought process as they perform the test tasks. With this technique you can monitor the less-observable aspects of the session, such as the participant's level of confusion or satisfaction. Keep in mind that some participants may be more comfortable speaking their thoughts than others. Also, the technique is tiring and cannot be sustained over long periods.
Videotaping. If test sessions are videotaped, the test monitor doesn't need to be as rigorous in taking notes, because the video is available for review. The monitor can also use the video in the debriefing session to review the test with the participant and collect additional feedback. If you use videotaping, make sure that participants know they are being taped before beginning the session. It may be wise to ask participants to sign a consent form.
Questionnaires You can use questionnaires to gather information from participants at various stages in the test. Administer a pre-test questionnaire to determine each participant's experience with computers. Gather feedback during the test by issuing a questionnaire after each task. Administer a post-test questionnaire to gather the users' overall impressions about the site. The questionnaires should cover items that cannot be ascertained through observation, such as the participants' thoughts and feelings about the site.
Debriefing. A debriefing session allows the participant to communicate his or her feelings about the tasks. It also allows the test monitor to seek clarification about the participant's actions during the session. The test monitor can use the video or notes to review the session with the participant and ask such questions as, "Why did you pause before clicking 'Submit'?"

Write up a test plan outlining the structure of the session. Describe the tasks and the method in which they will be administered, and define the role of the test monitor. Prepare any test materials, such as questionnaires, scripts, and data collection lists.

Sample test plan
Step	Time
1. Greet participant	2 minutes
2. Administer background questionnaire	5 minutes
3. Read orientation script	3 minutes
4. Have participant perform fact-retrieval tasks	10 minutes
5. Administer mid-test questionnaire	5 minutes
6. Have participant perform online activity tasks	10 minutes
7. Administer post-test questionnaire	5 minutes
8. Debrief participant	5 minutes
9. Transcribe notes and impressions	15 minutes

Administer the test

With participants selected and test plan in hand, you can administer the usability test. For the sake of simplicity, let's assume that you are the monitor for this test scenario.

Set up. Check the testing environment to make sure the computer is properly configured, that the video equipment is set up and functioning, and that you have assembled all written materials such as scripts, checklists, and questionnaires.
Pre-test. If you are videotaping the session, ask the participant to sign a consent form. Administer any pre-test questionnaire and read the orientation script. Explain how you would like the participant to behave during testing - whether they can ask questions or ask for assistance, for example. If you would like them to "think aloud" during the session, explain and demonstrate the technique.
Test. Describe the tasks to the participant either verbally or by providing a written task list. Record the start time and collect relevant information during the test. Where appropriate, interact with the participant, but remember to keep your statements impartial: for example, say, "Did that not take you where you expected to go?" when a participant shows frustration after following a link. Record any behaviors you want to have clarified at debriefing time. Time each task, and gently guide the participant through the session. Have the participant fill out any questionnaires developed for use during the test and the post-test questionnaire at the end of the test session.
Debrief. Review the questionnaires and video with the participant still present to gather additional feedback. For any unclear behaviors noted during the test session, ask the participant to explain his or her actions, for example, "Why did you click on the book titles on the course readings page?"
Summarize. After the participant leaves, spend a few minutes summarizing your thoughts about the session before starting with the next participant. If you are using videotape, you can record your impressions directly on tape.

Analyze the data

What you do with the data you've collected depends on your objectives and your approach. You may through observation have already formulated a plan for developing or adapting your site based on the insights you gathered during the test sessions, in which case your assessment study is complete. You may, however, be looking for more concrete findings, in which case you should compile and summarize your data for analysis.

If you plan to pursue findings, transcribe all notes and quantitative data to the computer as soon after the test sessions as possible, while your impressions are fresh in your mind. For example, if you want to calculate statistics on the participants' timings - say, the mean time to complete each task - record and calculate those times using a spreadsheet application. If you collected data using questionnaires, collate the responses in your word processor for easy grouping and comparison.

Review the data in summary form to identify the problem areas on your site. Focus on tasks that most participants were unable to complete in the allotted time. Compare the different behaviors with the task to see if you can identify the cause of the difficulty. For example, if participants were asked to define a term using the site, but none clicked the "Resources" link that would lead them to a glossary, it may be that you need to include a "Glossary" link as one of your sitewide navigation links.

Page information

Go back to Articles menu