Testing Multimedia in the Community College Classroom
Expenditures in education for electronic and multimedia instructional gadgetry are increasing daily. Limited systematic study has been done to identify whether multimedia offers measurable improvements over traditional instructional techniques.[1,2] Essentially, teachers are in a "hyper-VCR" era: we turn the equipment on and assume learning occurs.
Under a grant from the California Community College System, the authors developed and implemented a test design to assess the efficacy of multimedia technology as a non-traditional form of instruction. In particular, the research sought to identify how students with different cognitive learning styles performed in courses that integrated multimedia technologies as opposed to courses using the traditional lecture format. The research attempted to broaden the evaluation of multimedia's impact on the curriculum, looking at the effect of using a variety of different multimedia titles and delivery options on student performance in a number of class sections.
Growing Out of Previous Work
The current project grew out of earlier research conducted by the authors under a grant sponsored by a consortium of San Diego area community colleges and IBM Corp.[3] Under this grant, the authors developed a PC-assisted multimedia application entitled The Vietnam Experience to supplement an existing curriculum in American Government. Using text, graphics and video images, the tutorial takes students through an introductory lesson on the history of the Vietnam conflict.
An efficacy analysis of this instructional approach was made by comparing performances of test groups of students exposed to both the multimedia material and traditional lecture to those of control groups of students receiving only traditional instruction. While quantitative results were inconclusive, qualitative assessment of students' experience with the alternative instructional format was encouraging enough to warrant further development.
Current Practices in Community College Teaching
As part of the current grant, multimedia practices throughout the California Community College System have been surveyed.
The modality survey of practices indicated that a large majority of instructors use multimedia in the classroom as a form of lecture support (80% of respondents indicated this is their primary use). Interestingly, the next highest response rate (59%) was for using multimedia as a tool to stimulate class discussion: introduce the multimedia element, then lead directly back from the technology to the student. A sizable percentage use multimedia outside the classroom to support courses as either tutorials or self-paced study tools (46% and 43% respectively). The number using multimedia as a tool for class feedback to establish lesson effectiveness or student remediation was, for the authors, surprisingly low (only about 30%).
Faculty responses were positive about the impact of multimedia on students, with 79% believing it increases classroom performance markedly and 86% believing it increases student attention. Only 3% of faculty responded negatively to these questions, indicative of overwhelming support of multimedia as an instructional method.
Further, the faculty survey overwhelmingly rejects the notion that multimedia is too difficult to use: 84% "disagreed" that the technology is too complicated, with a slightly lower number "rejecting" the notion. Only a small percentage agreed with these statements.
The survey results reinforce the general consensus among multimedia users that the technology has definite advantages over traditional teaching methods. Multimedia improves instructor efficiency and students' attention and participation, without presenting too great a technological hurdle.
The Platform
Based on the survey, the authors concluded the best platform for in-class multimedia presentations would have these characteristics:
- Easily portable;
- CD-ROM capability, either integrated (preferred) or as a peripheral;
- Fully multimedia-ready (sound, graphics and video cards);
- TV- or overhead projector-compatibility interfaces;
- 75 MHz or better;
- l6MB RAM or more; and
- Cross-platform capability (PowerPC, etc.) preferred.
Based on this criteria, the authors decided the best platform for in-class experimental testing is a laptop computer interfaced with an overhead projection system. Basic system configuration was a Macintosh PowerBook 5300 enhanced with a peripheral 4x CD-ROM drive and a Proxima LCD panel.
Political Science Courses as Testground
From the survey, an integrated multimedia curriculum plan was developed for two course offerings: Introduction to Political Science and Introduction to the American Political System. Students were divided into experimental sections receiving multimedia-enhanced instruction both in classroom and lab settings, and control groups receiving traditional lecture. The project commenced the summer of 1995 and continued into the summer of 1996.
Multimedia approaches were experimentally tested in collaboration with Mesa College's Accelerated College Program and affiliated high schools of the San Diego Unified School District. Over 200 students in ten sections composed the test population.
Student cognitive learning styles were identified using a measuring instrument derived from surveys like those used for the Cognitive Styles Learning Inventory, the Teaching Style Inventory (TSI) and the Productivity Environmental Preference Survey (PEPS). Cognitive categories of Tactile, Auditory and Visual learning styles were identified for measurement. Student performance in each experimental and control section were correlated with learning styles. Data, both qualitative and subjective, was gathered on subsequent student performance in both groups and correlated to address the research hypothesis that multimedia techniques could improve student performance.
Insertion Points for Multimedia
From the surveys, the following were identified as insertion points for curriculum integration:
- In-class CD-ROM and software presentations;
- In-class software simulations;
- Independent study-based (learning centers);
- Use of videodiscs;
- Computer simulations; and
- Out-of-class CD-ROMs to supplement classroom material.
In-class multimedia components were used for lecture and discussion enhancement. Out-of-class multimedia components reinforced specific lessons and supplemented the text. We originally considered looking into providing at least some students in the study with Internet access, for use as a research tool for student papers. But our college district d'es not currently have this option available universally for students, so we were forced to abandon this approach at this stage.
Multimedia components were integrated directly into the established course outlines written by the project directors for the first phase of testing. As part of the grant we reviewed a wide variety of CD-ROMs, videodiscs and computer software described as suitable for beginning courses in political science and American government.
We were dismayed to find very little electronic media actually suitable for an introductory course in political science (dealing with political philosophy, history and so forth). A greater variety of material existed for courses in American government. Most comprised CD-ROMs issued in support of various government texts. We also reviewed various authoring systems for multimedia presentations (such as Asymetrix Toolbook) and settled on a simple presentation slide manager using ClarisWorks to present personally developed curriculum items.
From these two sources, pre-existing and original, we assembled sufficient material to integrate into the course, Political Science 102: The American Political System. These materials formed the basis for experimental testing.
Objective Test Results
Multimedia curriculum material comprised CD-ROM graphics and QuickTime videos, text, plus charts and diagrams, all produced with authoring software by the project's directors.
Prior to experimental testing, data was collected on the experimental section's class performance in other courses to establish an evaluation baseline. Students in control sections were given the standard traditional lecture. Data recorded on their test results served as a separate control group by which to evaluate experimental results. All students in both control and experimental groups were administered a cognitive learning style inventory. Traditional studies of learning style find a strong correlation between student learning style and grade evaluation.
In addition to testing the hypothesis that multimedia education enhances aggregate student performance, the research further attempted to delineate how multimedia might impact the performance of students with differing learning styles. Experimental data was correlated with student learning style to evaluate this relationship. Five sets of statistical analysis of variance and correlation were conducted to address the research hypothesis:
- Comparison of test groups' performance to its own previous performance.
- Comparison of test groups to control group performance.
- Comparison of test groups' cognitive learning style to test performance.
- Comparison of test groups' cognitive learning style to control group's cognitive learning style performance.
- Comparison of test groups and control group performance varying with outside assignments.
Analysis of Results
Research results were mixed. From the analysis of the quantitative data the research hypothesis of a significant link between multimedia and student exam performance was not strongly substantiated. The data did substantiate a correlation between learning style and test performance; this was anticipated from the literature. The data established a weak statistical link between multimedia instruction and overall student performance. Students with a strong visual learning sty1e correlated with higher test scores in both traditional and multimedia instruction.
One result of particular interest to researchers was that students participating in out-of-class assignments showed no major difference in grade performance between the control group (using reserve readings) and test groups (using CD-ROMs); both groups performed superior to the rest of their population group. Given that students in these groups were volunteers, and volunteers for extra work tend to come from the "higher scorer" population, these findings lend support to the old hypothesis that good students do well, regardless.
To date, only limited statistical support has been generated in other studies to support the hypothesis advanced in this project.[4,5,6,7] Two possibilities exist here. First, the current, as well as past, researchers' preoccupation with quantified test results as the main measure of success may well be misplaced. As the current research reaffirms, students with a predisposition towards academic aptitude (usually associated with the strong visual and auditory learners) tend to do well whatever the teaching method. Good students do well, poor students do not.
However, the definition of good and bad are predefined by the teaching method. Given that student performance was a measure of their performance on traditional written examinations, which directly favor the traditional learner, it may well be that the research tests' design "predetermined" its own outcomes. This suggests a future research focus that uses measurement alternatives other than quantified test measures to evaluate the performance of multimedia, whose real strengths as a teaching method may well lie in harder to measure benefits in students' attention span and their retention levels.
Qualitative Results
Qualitative results were obtained from attitudinal surveys on multimedia distributed to students in experimental groups exposed to multimedia teaching methods. Two attitudinal surveys were administered. The first assessed attitudes towards the classroom multimedia presentation. The second elicited attitudes towards the out-of-class CD-ROM used as a student tutorial.
Two trends were apparent from the attitudinal survey data. First, overall student response to multimedia as a tool in lecture is predominantly favorable. Second, attitudes towards the use of multimedia improved with the amount of student exposure.
Qualitative research results more strongly supported the hypotheses that multimedia improves student performance; 40% of students felt multimedia improved understanding, while only 25% did not. Plus 73% found multimedia to be a positive addition to the course, 64% preferred it to traditional lecture, and 67% found it easier to see than the chalkboard. Finally, 56% of students "enjoyed" it, while only 17% did not. Smaller percentages preferred multimedia to traditional lecture (49%), wanted to see more courses using multimedia (47%), found they took more notes (39%), believed it ultimately improved the lecture (52%), or believed more multimedia elements would have improved the lecture (50%), though in each case agreement with the statements exceeded disagreement, usually by a ratio of two or more to one. The only large amount of dissatisfaction with multimedia was in the area of notetaking, with 41% of students not believing the approach facilitated more notetaking on their part. In general, the qualitative surveys reinforced the hypothesis that students respond favorably to multimedia, and, in many cases, respond more favorably to it than to traditional lecture.
One response from the student survey was particularly instructive: 64% of the students reported little or no prior exposure to multimedia in the classroom, while only 21% reported some or substantial multimedia experience. This indicates we are still at the beginning of the development of this methodology. The large majority of students still receive instruction via primarily pre-digital-age technology, not unlike that used by Socrates to educate his pupils 2,500 years ago. Few other professions have such a link to their pasts as d'es teaching; indeed, were we all to wear togas, comparable to the ancient Greeks, little difference would be noticeable in most classrooms. This is not a good thing. One can only imagine that even Socrates, given the chance, would have picked up a laptop.
Students Respond to Experiences
Students were also given an opportunity on the survey to comment on what they liked and disliked about the multimedia curriculum, and how it might be improved. Two points stand out from their subjective remarks.
First, the ease with which material is presented in the multimedia format (clicking a mouse rather than handwriting everything on the board) tends to "rush" a presentation. The natural pauses an instructor takes while working at a board (hunting for chalk, erasing, etc.) disappear. Thus greater practice and patience is required on the part of the instructor so as to avoid scrolling through an instructional presentation like so many slides from last year's vacation.
The second point is more technical. The LCD projector/computer system used, while acceptable, presented difficulties in transport, setup and room setting. It became apparent to the authors that for multimedia to be truly effective, it should be used in classrooms designed and dedicated to the task, with proper equipment and lighting.
Students were asked to volunteer, for extra credit, to spend time out of class in Mesa College's Independent Learning Center using a specific CD-ROM from a college textbook publisher in support of its American Government textbook. Each student spent an average of five hours working with the CD-ROM over the length of the course, and was then surveyed. Evaluation methodology was the same as for in-class lecture surveys. On average, 80% or more of students agreed or strongly agreed that using the CD-ROM was a positive experience, both in terms of ease of use and its contribution to the overall course of study. Most problems reported by students, and their suggested improvements, sprang from improving the technology, not from a pedagogical problem with multimedia in instruction.
Overall, the qualitative data provides a stronger case for the hypothesis that multimedia improves student, especially under-represented student, performance. Given that students, by and large, enjoyed and preferred the multimedia methods, this may well lead to increased retention, or at least in students reaching their personal level of maximum retention more efficiently. Moreover student "enjoyment" of materials may be a contributing factor to increased retention and decreased student drop rates from the course. Both of these concepts need further investigation.
Conclusions
The current research demonstrates one of the major challenges still confronting multimedia as an educational tool, even a generation after its being heralded as the panacea for our w'es and aches. Is multimedia popular with students and educators? Yes. D'es it improve student performance? Maybe. Is it worth the cost? Perhaps. Must we continue to explore this innovative pathway to education? Definitely.
An educator's role must exceed that of simply being a "talking book." One avenue to accomplishing this task is to introduce "electronic interactive pathways" -- multimedia tools -- into the classroom, freeing the instructor to do that for which she or he has been trained: to enlighten and not to bore, the student mind.
The utility of electronic media in instruction as an alternative to traditional media lies in its ability to provide additional pathways to student learning and retention. As American colleges and universities strive to meet the needs of an increasing student population into the next century, this flexibility can become an invaluable educational tool.
Carl Luna is an instructor at the Department of Social Sciences and Accelerated College Program at San Diego Mesa College in San Diego, Calif.
Joseph McKenzie is an instructor at the Department of Social Sciences and Accelerated College Program at San Diego Mesa College in San Diego, Calif.
References:
- Smith, J. Jerome (1993), "SPICE Project: Comparing Passive to Interactive Approaches in a Videodisc-based Course," T.H.E. Journal, 21(1), August, pp. 62-66.
- Wright, Jim (1993), "Presidential Multimedia," Multimedia Solutions 3 (Nov./Dec.), pp. 27-30.
- Luna, Carl J. & McKenzie, J. (1994), "Application of Multimedia and Interactive Video Technologies to the Political Science Curriculum," Paper delivered to the Western Political Science Association's Annual Meeting, Albuquerque, NM.
- Songer, Tim & Gretes, John (1989), "Using Interactive Videodisc for Assessment of Adolescence and Adults: A Learning Style Survey," Research Paper, Interactive Knowledge, Inc.
- Huang, Samuel D. & Ali, Jane (1991), "The Impact of Using Interactive Video in Teaching General Biology," The American Biology Teacher, 53(5), May, pp.281-284.
- Janda, K. (1992), "Multimedia in Political Science: Sobering Lessons From a Teaching Experiment," Journal of Educational Multimedia and Hypermedia, pp. 341-354.
- Opheim, Cynthia & Stouffer, W.B. (1995), "Using 'Capitol Hill' CD-ROM to Teach Undergraduate Political Science Courses," Paper presented at the American Political Science Association Annual Meeting, San Francisco, CA, August, 1995.
Products mentioned:
- American Government, Version 1.0 CD ROM; Allyn and Bacon, Needham Heights, MA, (800) 223-1360.
- Toolbook; Asymetrix, Bellevue, WA, (800) 448-6543, www.asymetrix.com
ClarisWorks; Claris Corp., Santa Clara, CA, (800) 544-8554, www.claris.com.