Go back to Spotlights menuMERLOT promises efficiency and quality control
Barbara Knauff, Instructional Technology Specialist, Dartmouth College
Using online materials in your teaching doesn't have to mean authoring a wealth of materials from scratch: what's so great about the World Wide Web is that unless an author chooses to restrict access to her or his materials, they are accessible to users world-wide. Rather than spending a lot of time creating content, designing a Web interface, and putting the materials online, one's teaching goals can often be met just as well by pointing students towards existing Web resources. This is especially true for learning content that is relatively static and tends to be replicated from site to site: just how many ways are there of presenting the rules for definite and indefinite articles in French, for example?
The problem arises when one attempts to locate suitable materials. Search engines tend to return a staggering number of links, and evaluating search results for suitability and quality can be extremely time consuming - often more so than authoring a new Web page would have been. Not to mention the fact that high-quality sites that are not highly used or attuned to the search engine's way of seeing the Web often fall through the cracks entirely. It's sometimes quicker to reinvent the wheel than to find its schematics online. Wouldn't it be nice to have a mechanism to quickly find the best teaching materials for a given subject, in a given discipline?
This is one of the very real problems regarding Web teaching materials that MERLOT (Multimedia Educational Resource for Learning and Online Teaching) attempts to address. MERLOT functions as a repository for Web-based teaching materials, but beyond merely archiving and categorizing materials, MERLOT aims to evaluate materials by adding peer and user reviews, and to build virtual teaching communities by providing a forum for authors and users of teaching materials.
History and definitions
MERLOT came into being in 1997, spearheaded by the California State University Center for Distributed Learning, and started out as essentially a repository for online learning materials. Since then, MERLOT has grown considerably, both in size and in scope. From a four-member consortium in 1998 (adding the University of Georgia and the University of North Carolina systems, along with the Oklahoma State Regents for Higher Education) it has grown to a 23-member member cooperative which includes a number of large academic powerhouses (for a list of 2000-2001 members, visit the MERLOT Tasting Room). MERLOT's goals and ambitions have become quite ambitious. Well-defined criteria for evaluating what MERLOT terms "learning objects" (typically, Web-based teaching modules) are in place. Materials are rated on a scale of 1 to 5, and in three separate categories, namely quality of content, potential pedagogical effectiveness, and usability. This nuanced three-pronged approach to rating materials is invaluable when it comes to choosing materials for a concrete learning situation in the real world, where conditions such as user computer literacy or bandwidth limitations can vary considerably. MERLOT has also built an impressive interface for searching for materials. Learning objects - currently over 5000 - are categorized by subject area, and an advanced search feature allows the user considerable control over search results. It is possible, for example, to search by material type (tutorial, simulation, quiz, reference, lecture, etc.), primary audience (grade school, high school, college, graduate school, etc.) or technical format (HTML, Java applet, JavaScript, Flash, Shockwave, PDF, etc.). Further search options include author, institutional affiliation, keywords, availability of source code, and of course, quality of peer review or user ratings. This allows the user to cut to the chase quickly and pinpoint materials that are appropriate for his or her audience and technical set-up.
Thanks to MERLOT's straightforward but complex search features, it is equally simple to locate any one of the almost 5000 MERLOT members. Individual membership in MERLOT is free and open - online registration is all that is required - and members can create searchable profiles based on their skills, interests, subject area, professional affiliations, etc. Additionally, contextual cross-links connect learning materials and member information. A simple click, for example, gives access to peer reviews or user comments authored by a given member.
In sum, MERLOT has created a viable review mechanism and Web interface for efficient targeting of Web-based teaching materials, and for connecting with educators in one's area of interest.
The strengths of MERLOT's approach
MERLOT is an ambitious and timely project. It attempts to cut through the clutter of Web-based materials via comprehensive indexing and searching, and to adopt quality control measures from the world of traditional media through a review mechanism. The latter is important not just from the standpoint of the user who is trying to locate quality materials useful in a given situation, but from that of the educator authoring Web-based materials. The evaluation mechanisms currently in place in academe have not yet adapted to Web publishing; consequently there is currently very little incentive for faculty to invest time and resources in Web development, since such work tends drop off the radar in tenure and promotion reviews. A rigorous and confidence-inspiring peer review system for such materials might just be what's needed to persuade reluctant administrators to give due credit to Web development. Recognizing this need, MERLOT offers to mail official copies of its peer review to the site author's department chair or dean, so that they can become part of the individual's file.
Where MERLOT falls short
As a relatively young initiative, MERLOT's current incarnation has a number of weaknesses. Some of these may disappear over time, while others might benefit from a review of MERLOT's governance.
What seems to be the most urgent issue is the relatively small number of peer and user reviews. As of this writing (July 2001), MERLOT indexes 5421 learning objects. Of these, only 176, or 3%, have received official MERLOT peer reviews, despite the fact that about 180 faculty members in 12 discipline-specific teams are at work. Currently, review teams are still defining and streamlining the review process, so it is safe to assume that the number of peer reviews will increase significantly in the near future. What is not immediately clear to the user is how submissions are chosen for review. A conversation with one review team co-leader revealed that all submissions are scanned for what look like promising entries; yet this first level of evaluation doesn't seem to be formalized. User reviews, which can be submitted by any individual member of MERLOT, are more numerous, but still cover only about 20% of the materials. Additionally, it is very rare for materials to have more than one user review, so that as it stands the user review category probably reflects individual biases (good or bad) rather than the opinions of a group of users.
The ratings scale poses another problem. While the scale ranges from 1 to 5, MERLOT explicitly discourages the use of all but the highest two categories, since its review process is so public and accessible. This deliberate ratings inflation is certainly helpful in inviting submissions for peer review, but it undermines the usefulness of the reviews considerably. Currently, 83% of peer reviews fall in the top two categories. In part, these very positive ratings may have to do with the preliminary scanning for good learning objects that takes place before a peer review is launched.
But the ratings problem is also related to the public nature of MERLOT's peer review process. The traditional peer review process is both private and anonymous. Reviewers generally evaluate submissions without knowing who the author is; and the author receives a review without knowing the reviewer's identity. This brings considerable integrity to the review process and protects both the author and the reviewer in the event of a less-than-favorable review. Negative reviews usually have no consequence for the author beyond a bruised ego and the need to look for publication elsewhere. MERLOT's Web-based review process, on the other hand, is by nature transparent. The identities of authors and of the members of the review team are known and reviews are posted for public consumption. This transparency, however, may come at the price of undermining confidence in the review's integrity, and might make a MERLOT review a harder sell to administrators as well.
MERLOT's membership structure also seems somewhat problematic. While individual membership is free and unencumbered, institutional membership comes with considerable strings attached. An aspiring member institution faces an annual fee of $25,000 and a number of formal commitments towards MERLOT: apart from an institutional action plan and a formulation of goals regarding the institution's participation in MERLOT, the most significant commitment is the sponsorship of between six and eight faculty members as peer reviewers for MERLOT during the membership year. Since sponsorship involves a 25% workload reduction at the home institution, or commensurate compensation, this commitment translates into a financial obligation equivalent to the expense of between one-and-a-half and two full-time faculty lines. What's in it for the institution is primarily the fact that a strong statement in support of technology-enhanced education and distance learning has been made. Given these relatively high barriers to institutional membership, it is not surprising that the majority of MERLOT's members are large state university systems for which the impact of membership requirements would be somewhat distributed. This clearly leaves out a large segment of the American educational scene: for example, high schools and liberal arts colleges with smaller budgets and faculties. This becomes a concern when one remembers that the peer review teams are recruited exclusively from the member institutions. The reality of teaching situations varies greatly between different types of institutions, and it would seem that this range should be reflected in a peer review team assessing potential pedagogical effectiveness of materials.
Lastly, MERLOT's individual members don't seem to make enough of a commitment to the site - many members never return beyond their initial visit, leaving the user review shelves virtually unstocked. Much of the infrastructure and community-based philosophy behind the MERLOT model has been successfully integrated in the dot-com realm - many commercial Web sites are driven by user reviews and comments. But what makes or breaks the success of such sites is user participation. Amazon.com would lose a good deal of its usefulness to the browsing consumer if user reviews were few and far between and limited to one review per item. It's up to MERLOT's users to show their support for MERLOT by using the site and contributing feedback and reviews.
Improving with age
Many of MERLOT's problems will undoubtedly disappear as the project gathers steam. MERLOT's leadership recently hired a CEO, and there is hope of transforming MERLOT into a not-for-profit organization that would no longer need to rely on institutional memberships. MERLOT's leadership is also pursuing affiliations with professional academic associations, especially with regard to the peer review process. The goal is that the peer review process would be adopted by these organizations and would end up closer to the well-established peer review process in place elsewhere in academe. Many other initiatives are also in development, such as an online tutorial that would provide an introduction to MERLOT's peer review criteria, and MERLOT TWO (Teaching Well Online), a guide to the successful integration of teaching materials. If these new initiatives take off, and if the number of peer reviews increases and users begin to actively contribute feedback to the MERLOT community, searching for quality-controlled online teaching materials might soon be as easy as navigating MERLOT's user-friendly search engine.
Page information
By Barbara Knauff
Copyright 2001 Trustees of Dartmouth College
Added: 17 July 2001