Constructing an Evaluation Tool
The most common evaluation tool, and a good starting point for this discussion, is a rubric. As defined by Dannelle Stevens and Antonia Levi, a rubric “lays out the specific expectations for an assignment. Rubrics divide an assignment into its component parts and provide a detailed description of what constitutes acceptable or unacceptable levels of performance for each of those parts.” (Introduction to Rubrics, 3). There is no single term used to describe exactly what a rubric is measuring. Stevens and Levi’s definition uses “component parts,” others use “dimensions,” “expectations,” “sections,” and a host of related terms. For clarity, we use the term adopted by the Dartmouth Center for the Advancement of Learning and used most frequently in pedagogical literature: criteria. In their simplest definition, criteria are the skills or knowledge on which a student is assessed in the assignment. In a traditional, grid-shaped rubric, criteria fill the first column, with the following columns providing a scale of achievement of each criterion. Criteria can range from highly specific to very broad, but the more specific, the better, as that specificity leads to usable data.
There is no perfect way to construct a rubric. Some include 3-4 criteria, some more. Some use a scale of 2-3 levels, others include additional levels. Some are formatted as grids, others look vastly different. This is why we prefer the general term evaluation tool, which incorporates devices firmly distinct from rubrics (like an answer key) and welcomes innovations to rubrics themselves, example of which can be found here. We have no standardized approach to rubrics or other evaluation tools at Dartmouth, and you are strongly encouraged to create or adapt an evaluation tool in the format that is most applicable to and useful for your program.
If you already have an evaluation tool for the selected student work, please do not hesitate to use it for this process. Some may have to be tightened to best match the learning outcome; others may require slight reframing. Either is welcome!
Creating an evaluation tool like a rubric from a learning outcome is first and foremost a process of breaking down the learning outcome into measurable parts. One straightforward approach, separating the clauses of the learning outcome, is simple but not always enough, depending on the format and scope of the learning outcome.
If you’re struggling to enunciate how your program’s outcome could be broken down into specific criteria, that might be a sign that the outcome should be adjusted. In nearly every case, this will be a minor tweak of the verb or gerund (“identifying” over “appreciating,” for example) or greater detail about the exact content or skill. If there is no clear, straightforward solution, that could suggest that the selected learning outcome may need to be removed or rewritten.
Approach #1: Separating Clauses
For some learning outcomes, breaking apart the outcome into clauses, actions, and key concepts can automatically generate acceptable criteria. For example, the American Chemical Society's teamwork learning outcome, Students will demonstrate "effective coordination, direction, and engagement of team members; experience in persuasion and negotiation to accomplish goals and meet deadlines; and ability to resolve conflicts and critically evaluate team members," could be broken out verbatim into four criteria for assessment, as follows:
- Effective coordination, direction, and engagement of team members
- Experience in persuasion and negotiation to accomplish goals and meet deadlines
- Ability to resolve conflicts
- Ability to critically evaluate team members
Building an evaluation tool from here is easy. For a rubric, use a key word or phrase for each criterion--coordination and engagement, persuasion and negotiation, conflict resolution, and critical evaluation, for example--as the first column. The rest of the information in each criterion (for example, "to accomplish goals and meet deadlines") can become part of the scale--that is, the rest of the columns in the rubric.
Approach #2: Working Backwards
However, other learning outcomes that either describe a single skill (or type of knowledge) or are less specific might require working backwards. For an outcome like "Students will write effectively in a manner appropriate to the discipline"--which, in some form, is common to many Dartmouth programs--it would be helpful instead to start from the ideal student work that achieves the outcome. For example, an essay that showcases a student's ability to "write effectively in a manner appropriate to the discipline" might include the following criteria (which are deliberately hazy so as to resemble work in multiple disciplines; your own criteria would likely be more specific):
- Use of a clear and defensible thesis statement
- Logical structure and organization
- Fluency with discipline-specific terminology
- Engagement with secondary sources
- Engagement with primary sources
- Correct, discipline-specific citational practices
- Proper grammar, spelling, and proofreading
- Original thought
- Appropriate research methods
These nine bullets have much more detail and apply immediately to the context of the student work being assessed. If you’re happy with nine items to assess, you could move on from there, or you could condense them.
Condensing these nine bullets works well because of their significant overlap. Rather than using all nine as criteria, you could combine bullets under larger headings and use related bullets as part of the descriptions or anchors in the scale. For example, you might combine the bullets together under the following descriptive headings:
Research
- Engagement with secondary sources
- Engagement with primary sources
- Appropriate research methods
Argument
- Use of a clear and defensible thesis statement
- Logical structure and organization
- Original thought
Mechanics
- Fluency with discipline-specific terminology
- Correct, discipline-specific citational practices
- Proper grammar, spelling, and proofreading
You are now left with three criteria (underlined) that speak to the learning outcome, track only three concepts rather than nine, and retain the original bullets’ foundation in the context of the assignment.
You also have many of the points in your scale identified. With only the addition of relative qualifying terms (italicized), this process organically results in an evaluation tool, presented below as a traditional grid rubric:
| Criterion | Outstanding | Satisfactory | Unsatisfactory |
| Research | Essay engages frequently with both secondary and primary sources and shows continual use of appropriate research methods | Essay engages occasionally with both secondary and primary sources, or mainly with only one or the other, and shows sporadic use of appropriate research methods | Essay engages infrequently with both secondary and primary sources, or engages with only one or the other, and does not show use of appropriate research methods |
| Argument | Essay uses a clear and defensible thesis statement, logical structure and organization, and original thought | Essay uses an opaque or indefensible thesis statement, ambiguous structure and organization, and relies primarily on the arguments of others | Essay does not have a thesis statement, has unclear structure and organization, and lacks original thought |
| Mechanics | Essay demonstrates strong fluency with discipline-specific terminology, correctly uses discipline-specific citational practices, and always showcases proper grammar, spelling, and proofreading | Essay demonstrates some fluency with discipline-specific terminology, most often uses discipline-specific citational practices, and almost always showcases proper grammar, spelling, and proofreading | Essay demonstrates poor fluency with discipline-specific terminology, infrequently uses discipline-specific citational practices, and rarely showcases proper grammar, spelling, and proofreading |
About Evaluation Tools
Although evaluation tools are required for this process, they frequently serve an important role in everyday teaching and learning at the course level. As part of the compliance process, your program will only be required to use an evaluation tool to measure the achievement of your program’s student learning outcomes, but it is our hope that these approaches and processes will form part of the pedagogical culture at Dartmouth.
|
Additional Benefits of Evaluation Tools 1. Greater specificity. While crucial skills like critical thinking, creative thinking, writing, oral communication, teamwork, intercultural knowledge, and many others are difficult to define for students and faculty alike, the criteria of an evaluation tool allow you to break these skills down into constituent parts. For example, instead of a criterion that asks students merely to “think laterally” in an essay, describing how students should “integrate course material from at least two [time periods/geographic areas/units] into their discussion,” “draw on material from at least one other course in the major,” and/or “compare and contrast primary sources from at least two contexts” provides this clarification. |
For further examples of evaluation tools, see here.