Features and Functions Are Merely Trifles in the Selection of a Course Management System

• 06/01/04

The memo just landed on your desk - it's a mandate to put your school's courses online. But despite the breathtaking speed with which an impulse can turn into a Web site these days, there's a lot more to putting courses online than just creating a few Web pages. You need a course management system (CMS) to properly organize content, courses, sections, faculty, students and grades. The CMS also supplies tools for interaction, including assignments, quizzes, discussion groups, chat sessions and whiteboards.

However, all course management systems aren't created equal; features and functions differ. Yet, by the same token, there's a lot more to e-learning success than scrutinizing vendors' lists of features and functions and picking a winner. What lies beyond features and functions are considerations such as a vendor's underlying architecture, customer support, pricing strategies, partnerships, and the ever-important vendor-customer chemistry.

When tasked with selecting a CMS, the first step many responsible colleges, universities and consortia take is to name a selection committee of faculty, administration, students and possibly IT staff. They do some Web searching and talk to colleagues, friends and counterparts at other institutions. This same committee also defines goals, objectives and selection criteria. They then assemble a long list of vendors and send out requests for proposals (RFPs) - inevitably a long list of features and functions that the institution requires in a CMS. This plan of attack is better than flying blind. However, the biggest mistake a college, university or consortia can make is treating the RFP as the Magna Carta and selecting a software product on features and functions alone.

The Smoking Gun

Features and functions are important, but they generally dictate who should be on the initial vendor list rather than who comes off the list. Since no vendor will go for long without a feature that a competitor offers, comparing features and functions is virtually meaningless. If a company is missing a key feature, they'll likely have it soon. If a company has a unique feature, that d'esn't automatically mean it's useful, important or well-functioning. In fact, most vendors have the same basic lists of features and functions.

Some selection committee members will correctly argue that how vendors implement their features is critical. That may be true, but selection committees in general put far too much emphasis on minutiae - differences between implementations that are ultimately meaningless. For example, I served on one selection committee in which the interface expert performed a detailed analysis on the "usability" of two different course management systems. One offered slightly more customizability in the whiteboard function. The distinction struck some committee members as a smoking gun. Yet surprisingly, nobody in the room knew of a single online course, on campus or anywhere else, which employed the whiteboard function. Remember the 80/20 rule: 20% of the application is used 80% of the time. In real life, the smoking gun fired a blank.

The Ease-of-Use Paradox

After the RFP, the next step in the selection process is typically the ease-of-use analysis. What exactly is ease of use? We think we know it when we see it, but it's really a complex concept. Consider this: A good way to evaluate course management systems' ease of use is to sit a faculty member down and ask him or her to load a course. If it g'es in, great. But don't stop there. Ask them to load courses with different file formats, including multimedia files, Web sites and standard learning objects. At Drexel University, we found that the CMS which was easiest to load was an absolute pain when it came to loading more complex course content. Conversely, the CMS that was supposedly trickier to load easily took course content in any file format - it was file-format agnostic. The putative ease-of-use victor simply couldn't take some of our files.

So here's the paradox: What appears to be ease of use on first impression may be a direct result of limitations that may make the CMS difficult to use for more ambitious pedagogy. Or more succinctly, perceived ease of use is inversely proportional to flexibility and robustness. The lesson here is to perform extensive ease-of-use testing. Try a day or two rather than a couple of hours. Test against real-world scenarios and demanding, shifting pedagogical conditions.

A common ease-of-use question is: "Easy for whom?" Committees sometimes make the mistake of tackling CMS selection from the point of view of the institution - the faculty loading the courses, the administrators exporting grades, and the IT staff figuring out how to accommodate it. But ease of use is also a concern for the end customer - in this case, the student.

Another key ease-of-use question is: "Easy at what stage of the implementation?" Any new application will likely be difficult to learn at first, but how long d'es that last. Think of your e-mail and word processing programs. Presumably, each new version of the software brings features that you have to learn but then come to love. So ask yourself whether your CMS, while easy for beginners, is going to be too limited for power users. Ease of use up front can become a pedagogical obstacle on the back end.

Once you've dealt with features, functions and ease of use, it's time to peel away the rest of the layers of the CMS selection onion. Below are 10 big considerations beyond features and functions for making a great CMS decision. (For the detailed list, including a formula to figure out your real cost of ownership, see the end of the article.

1. What d'es IT think?
2. Who will train users, including faculty, content developers, instructional designers, administrators and students?
3. How will you roll out the application - in pilots or full bore?
4. What applications would you like to integrate and when? At Drexel University, we selected WebCT Campus Edition and, more recently, WebCT Vista because of the company's integration with our SCT Banner student information system.
5. Is your vendor a "partner"? Every vendor says they are, but the bottom line is whether your vendor is accountable for your success or your failure.
6. What is the real cost of ownership?
7. What are your goals? The answers may drive your selection.
8. How will your CMS choice affect the student experience?
9. What's in it for the faculty? D'es your choice expand pedagogical alternatives?
10. What are the hidden criteria?

Buying into a Vendor Relationship

All of these questions illustrate the fact that when you buy a CMS you're investing in a lot more. What you're buying is a foundation for an e-learning program, a foundation with far-reaching implications. You're also buying a relationship, for better or worse, with the company from whom you purchase the system.

And as in any close relationship, you need to learn all you can about your prospective partner before plunging in. Find out how strong they are in their market, because healthy companies are more likely to survive the test of time as your e-learning program matures. These healthy companies are also more likely to have the brains and the staff to provide the kind of training and support that will make your program thrive.

Find out about the other relationships in which your prospective partner is engaged. This includes partnerships with compatible technologies you might want to integrate such as student information systems and/or portals. It also includes partnerships with vendors who can easily expand the range of features and functions in your CMS, such as audioconferencing software providers or content publishers.

It's also important to assess prospective vendors as you would a job applicant. What e-learning success stories can they document? Ask for references, do site visits and read reviews of their software. Ask around to find out if customers are switching CMS vendors, and if so, in which direction and why.

These suggestions can't prevent failure or second guessing, but they can significantly increase the odds of success. Whatever decision you make will be wrong to some people, especially the holy warriors who are emotionally or irrationally wed to a particular brand. But with careful planning and keen anticipation of day-to-day challenges, the rigors of selecting and implementing a CMS will rapidly give way to the power of e-learning to transform education, improve access, foster choice and deliver significant financial benefits. In the end, selecting a CMS is more than features and functions. It d'esn't have to be a daunting task; it just has to be an informed one.

10 Considerations (Beyond Features and Functions) for Making a Great CMS Decision

1. What d'es IT think? Too often IT staffers are ignored, undervalued or underassertive. For whatever reason, they don't get their say until it's too late. If you fear their input may detract from educational criteria, rest assured that technical constraints will affect education sooner or later.
2. Who will train users, including faculty, content developers, instructional designers, administrators and students? An enterprise application will require training to create proficiency.
3. How will you roll out the application - in pilots or full bore? Managing growth is part of the challenge, and growth results from two activities. First, training faculty results in linear growth. Each new participating faculty member will bring on a set number of courses, and each course might serve 25 students. Once exposed to e-learning, those students will inevitably go on to demand that their other courses offer e-learning components, thereby creating geometric growth. Never underestimate students' ability to drive the implementation process.
4. What applications would you like to integrate and when? At Drexel University, we selected WebCT Campus Edition and, more recently, WebCT Vista because of the company's integration with our SCT Banner student information system. That was a big consideration for us, because their main competitor didn't have the integration at the time.
5. Is your vendor a "partner"? Every vendor says they are, but the bottom line is whether your vendor is accountable for your success or your failure. What is your vendor's vision for higher education? D'es your vendor understand that you're their best client when things go well and would be their worst client if they failed to address a problem?
6. What is the real cost of ownership? It's a lot more than the license fee. Consider the cost of maintenance, support, upgrades, hardware, software, staffing, administration, training, and content development. The license is typically less than half the total cost of ownership and less than 20% in more cases than you'd think.

* Here's a simple formula:

Cost of Ownership =
Vendor Software - license, maintenance, support.
+
IT - additional hardware, software and personnel.
+
User Support - additional hardware, software and personnel, including help desk, training, content development, instructional design, system administration and user management.
+
Administrative systems - more hardware, software and personnel for managing enrollments and the student information system (SIS) interface.

7. What are your goals? They probably reflect that original memo you received regarding putting your courses online. But are you measured by the number of students, the number of courses, the uptime, educational performance, revenue or cost containment? Are you adding more students or giving your students more? The answers may drive your selection.
8. How will your CMS choice affect the student experience? While faculty may not give a hoot about integrating the SIS with the campus portal and course management system to provide a single sign-on, it sure would make life easier for the students as well as allow support staff to focus on adding value.
9. What's in it for the faculty? D'es your choice expand pedagogical alternatives? The "Holy Grail" is individually delivering learning in forms and sequences tailored to each student's pace and learning style. This is not common today, but it may be within the next few years, and it will pay to possess a flexible system built on open standards from a vendor credible enough to deliver this functionality in the near future.
10. What are the hidden criteria? It pays to nullify religious arguments - blind loyalty to a particular vendor - and to smoke out any vested interests. A typical one: "My brother-in-law attends University X and says vendor Y's CMS breaks down a lot." Merely questioning the argument takes diplomacy. Since anecdotes make poor evidence, treat them accordingly.