Collecting Assessment Data
In many cases, learning outcomes assessment processes may already be in place: many programs, for example, review theses, dissertations, or culminating experiences at faculty meetings or hold informal discussions of performances or oral presentations. If your program does not already have a process of this nature, we ask that you operationalize a straightforward approach that works within existing structures.
Step 1: select assessors
For many programs, the selection of assessors might be a foregone conclusion. If your chosen artifact of student work is a thesis or dissertation, the advisor(s) or committee would work perfectly! And if all of the artifacts come from a single course, that faculty member would be an ideal assessor.
However, if artifacts come from a range of courses, there are two main options. Each faculty member could individually assess the artifacts from their courses. This in situ approach has several benefits, including working within existing structures and providing context for each artifact. The other approach (ex situ) is removed entirely from an individual course and involves faculty assessing artifacts from various contexts. This too has unique benefits, including establishing stronger assessment norms, underscoring the distinctions between this process and grading, and allowing greater knowledge-sharing and community-building.
STEP 2: Gather data
There are several different ways to gather data from assessors; the best depends on your context and existing norms in the program. Many Dartmouth departments simply ask faculty to return completed evaluation tools to the DA or chair; this is especially valuable if there are few students in a given year. Others have elected to use surveys in a program like Google Forms, in which assessors complete a survey for each artifact.
There are plenty of other options. Some programs use portfolios (often supported by DartWrite), which includes the benefit of providing students with a venue to showcase their work across the program. Others export information directly from Canvas if the evaluation tool is built in. No matter what, the goal should be to find something familiar to faculty and replicable across future years.
Step 3: Review data
Although programs are not required to report their findings every year, there's no reason not to look through the data more frequently. As you do so, consider the following questions:
1. Is there a sufficient range of results? If there is a narrow range, the results will like be unhelpful and unactionable. If so, consider revising the evaluation tool and/or engaging in a norming session with all assessors to clarify the scale.
2. Were any of the criteria irrelevant? If a certain criterion is often absent in the selected assignment(s), it might suggest that the learning outcome should be rewritten; in other cases, the criterion may not fully reflect the outcome and would need to be revised.
3. Is there a clear outlier? If results are considerably lower for a certain criterion or outcome, it might be worth digging further into that specific criterion, perhaps by revising the evaluation tool (subdividing that criterion might be a good idea!). That outlier might also suggest the need for curricular changes.