Educational Effectiveness of Computer Software
by DR. NETIVA CAFTORI, Assistant Professor Northeastern Illinois University Chicago, Ill. This paper describes a study in which an attempt was made to demonstrate that not all supposedly good, educational software is used as intended. And even when it is, the predefined objectives often are far from being achieved. Not enough is known about how learning takes place. Educational software offers us new means of investigating this complex process. Site of the Research The research was done at Old Orchard Junior High School (OOJH) in Skokie, Ill., during 1992 and 1993. OOJH, the only middle school in District 68, is recognized nationally for excellence. It serves grades 6 through 8. One Apple lab is used solely for two keyboarding classes and two "tool" classes (spreadsheets and databases) during four periods each day. A second "lab" consists of 25 Tandy laptop computers that are transported to classes as needed for word processing. The third, or main, lab contains 30 Apple color computers, some of which are linked to printers. Other than language arts, music and word processing, rarely is any other subject taught in the main lab. The main lab has an extensive library of educational and entertainment software. For the lunch hours (study hall period), a list is posted of the 40 or so games available to students. The most popular programs are: Shifty Sam Jeopardy Fraction Munchers Number Munchers Where in the World (or the U.S., or in Time) is Carmen Sandiego? Dazzle Draw Game Show Word Challenge Odell Lake Microzines Paper Plane Pilot Wood Car Rally Oregon Trail Pizza to Go Dog Sled Ambassadors. Students at OOJH are provided with some very good educational software. Our focus is on the way the programs are being used. Popular Programs and Why During the school's three lunch periods, students can come to the main lab to play educational games for 20 minutes instead of going to study hall. Observations of students were conducted in this lab during the lunch hours as they interacted with a particular package. A few students were chosen for in-depth interviews on a voluntary basis. Each lunch period, different grade-level students attended but, consistently, most were males. Students were left on their own, except for the presence of a lab supervisor, whose sole function was to distribute software and maintain orderly conduct in the lab. From our own observations and those of lab supervisors, games used during the free lunch periods divide into the following five categories: Games girls like most (frequently played): Jeopardy, Fraction Muncher, Number Muncher, Game Show and Word Challenge. Games girls like less (played some times): Shifty Sam, Carmen Sandiego, Dazzle Draw and Oregon Trail. Games girls don't like (never played): Paper Plane Pilot, Wood Car Rally, Microzines and Odell Lake. Games boys like most: Jeopardy, Oregon Trail, Carmen Sandiego, Odell Lake and Fraction Muncher. Games boys don't like: Game Show and Word Challenge. It was noticed that students did not want to venture onto new software, as their time was limited. When they came to the lab for the first time, they usually played games recommended by their friends. Software documentation indicated that many programs were designed for a classroom environment with trained teacher guidance, which was not the case during these lunch periods. A supervisor was there to answer basic inquiries from students, however, she was not prepared for all questions. How Each Program Is Used Many children play Number Muncher and Fraction Muncher just to attain the highest score. Its arcade-like quality makes it very attractive despite the fact that there is not much a junior high school student can learn from the game. One may argue that some mathematics skills are strengthened, however the majority of students play this game because they know their facts so well that they can proceed at a high speed without much thought. Jeopardy is liked by many students since they are challenged to give the correct answer, win against the computer and score high. The girls that play it are usually enrolled in Advanced Studies; according to the lab supervisor, average female students do not play this game. This may be explained by its level of difficulty and girls' concern about being embarrassed in front of others.1 Self-confidence is also necessary to attempt this game as a great deal of trivia must be memorized. Indeed, these two items have been cited as a possible cause for a general lack of girls in computer labs.2 Where in the World is Carmen Sandiego? is played by both boys and girls and is considered quite challenging. Since it requires a long time for many people to obtain results, some students don't make it their first choice. Dazzle Draw is an attractive program for artistic creation. It allows for figurative and abstract designs not easily done by hand. For example, it offers functions such as mirror symmetry, spray paint, squiggly lines and filling enclosed areas with patterns. Paper Plane Pilot and Wood Car Rally were played by boys only. Players were not following directions and also weren't putting much thought into selecting values for the variables involved. Children seemed to improve their aiming skills, but more by trial-and-error strategies than careful calculation. Odell Lake is a science simulation to identify fish and their habits in Odell Lake, Oregon. It presents concrete details that allow a learner to move from specific to general, from the concrete to the abstract. General principles of predator/prey relationships and food chains are to be discovered. However this goal is rarely achieved. As this lab is not part of a science class, there is no mandate for a child to learn about fish behavior or their food chain. Only boys use this program. Girls' absence may be explained by the possibility that the gratification offered is not one they enjoy. The program's designers exploit aggressive traits, to the extent that some boys care only about swallowing fish and the accompanying sound effects. This is an example of unfocused experience: It is not a child's intention to learn about food chains and thus no effort is made to focus attention on such details. According to the documentation, one possible way to start this game is with no prior guidance. Children are invited to explore Odell Lake on their own, followed by a regular class session. However, such a sequence is not guaranteed, at OOJH or at other schools. If the goal of the program is to teach about predator/prey relationships, it may only be accidentally achieved. A Particularly Rich Example One educational program that is well known and recognized for its "effectiveness" and success with children is Oregon Trail. This is a history simulation with the goal of introducing children to the life of covered wagon travelers on their way to Oregon across North America in 1848. During the simulation, students are able to visit and learn the history of forts and other landmarks, and "talk" to others on the trail about their surroundings and different cultures. They must manage with little money and food, raging diseases and bad water. Stopping at any of the forts can replenish supplies, but prices become more expensive as they approach the West since food and other commodities become rarer. In this program, students must bear the full consequences of the decisions they make. Oregon Trail has much to offer: It requires making intelligent decisions based on several, but not all, available facts. It teaches students to collect, organize and retrieve information for interpretation. Many problem-solving situations, like river crossings, are presented. Every solution has implicit consequences that are accumulated to determine the end result of the game. The game is played in an interactive mode via menus and a few prompting questions. Explanations are provided for almost every choice through sub-menus. As one continues on the 2,400-mile trail, conditions such as weather, health, pace, rations, next landmark and miles traveled from the Missouri River to Oregon, are displayed. The more complex a program -- many different features and paths from which to choose -- the greater the chance it will appeal to a larger audience.3 Oregon Trail is fairly complex in that it has many different kinds of challenges, some of which are ignored by some children, but acted on by others. Characteristics of this game that we observed as most appealing to students include: reaching the end of the trail, scoring higher than before, making a Top-Ten list, hunting and killing animals, digging graves and successfully crossing rivers. Oregon Trail's designers have at least partly succeeded, in that children come back to the program. However, the question of whether a student has learned that which was intended is far less certain. The primary learning objective of this game is to develop decision-making skills in the face of changing and sometimes unforeseen circumstances. One has to consider alternative solutions and consequences, arrive at conclusions and act accordingly. Another objective is to develop intellectual skills by learning to compare and classify things, ideas, events and situations on the basis of similarities and differences, and group them into categories. The richness of the program's subject matter and its inter-disciplinary nature lend it to many individual and group activities. A manual is provided with worksheets for students and guidelines with suggested activities for teachers. From our observations, Oregon Trail is liked mostly by boys who enjoy shooting animals for food. Girls may like this game too, but for other reasons -- reaching the destination, writing epitaphs on tombstones, etc. To some degree, the educational objectives of the game are missed. Attributes that were intended to attract children to the game actually divert their attention from the objectives. One such attribute is competitiveness. For example, part of the challenge is to reach the end of the trail as fast as possible. Children are doing so, but without regard for their companions or oxen. The goal becomes so important that players neglect the health of other travelers and their own lives. Another example, shooting animals for food, was designed to teach children about different animals in different terrain, as well as be part of the reality of life on the trail. However, "shoot-em' up" has become a focus of attention for many students, (mostly boys). Unfortunately, besides eye-hand coordination, not much else is learned. And eye-hand coordination is not one of the stated objectives of this game. Due to limited time, an absence of teacher guidance and the desire to use the software solely for amusement, children tend to play the game differently from one another, based on their personality. They enjoy its different aspects, yet learn only a little from it. Educators who have studied Oregon Trail are usually impressed by the well-written program. It works for them since they can take the time necessary to read all the information and gather the data to make decisions. These educators are also trained in organizing their thoughts and in how to use their skills and other tools to retrieve information from the game at the appropriate time. Yet Oregon Trail has become an arcade game at OOJH. Some learning may take place, such as fording a shallow river and floating across a deep one, but it is occasional and not necessarily registered for later recall. Oregon Trail is a software program with tremendous educational potential; however, the way it is currently used at OOJH is for amusement only. Results and Recommendations From our observations and interviews at OOJH, it was concluded that children do not utilize educational software as intended by its designers. The lunch-time computer lab was set up to provide students with exposure to various stand-alone educational software. It was not the administration's intention to provide students with fun activities. After all, in the equivalent study hall period, one is not permitted to play games. Each child used his or her limited time to fulfill some need for relaxation and fun; it was better to be at the lab and "play" than to be at the study hall and study. Each of the students picked a program they found challenging. Students also stopped using some programs, deeming them "boring," "too slow" or "not challenging enough." All students were attracted at first to a particular software game because it was readily available, already known to them, recommended by a friend or easy to learn. Each came back to a particular game because some characteristic -- such as scoring high or great graphics -- were appealing. These characteristics were put there by designers to encourage children to explore further, yet did not guarantee learning would result. Ideally, a teacher should be "bundled" with the software. At the site where the program is purchased, that teacher could give a short workshop on how to use the program in a classroom. Including such information in the written documentation is, in many cases, not enough; few teachers read all the documentation. It is faster and easier to learn by demonstration, and then browse through the manuals. However, because teachers are not "bundled," software designers' expectations cannot be satisfied. A longer time span, at least 40-50 minutes, is also required to get much educational value out of a program. Thus, in a less than perfect setting, not much of the intended learning can usually be accomplished. Teachers can go a long way toward remedying this situation. First, they must read the package's documentation and implement the software as intended. If, for example, the program is designed to integrate into a curriculum in which teacher instruction and small-group participation is the norm, do not place it in a study hall in which no teacher/mentor is present and student interaction is discouraged. For designers, it is also not enough to embed attractive characteristics in the software because these can easily become diversions from the real goals. Educational objectives cannot be met using "glossy packaging" alone. And it sells short the capacity of computers to aid learning. A software program, however, can be designed to store in memory, for later retrieval, all moves made. It could prompt each individual student according to her record of past behavior, for example, then ask questions or give hints -- something that even a very attentive teacher cannot do. Children, left on their own and wanting basically to have fun, will naturally divert their attention to the features that appeal to them most and thus bypass many of the program's learning opportunities. Until software designers are certain of what features do both -- attract children and teach -- they should adopt the pedagogical methods that teachers use in a hands-on environment or for manipulatives: Remind students of what the goal is and point out inconsistencies in students' actions. While integrating those techniques, designers should avoid the most uninteresting method -- lecture. Designers can also make the program "behave" as a teacher, rather than counting on teachers being present in the room. If educational software can assume a teacher's role and still be motivating to children, it would accomplish something dearly needed: Specific attention provided to each child, immediate feedback and individual guidance. Our study focused on the simple setting of individual computers with limited RAM in an ordinary school, and the learning that is not taking place because of the way software is used, but that could take place given an alteration in the design of the software or in the environment. It is these two areas that educators need to address if the potential of educational software to teach our young people is fulfilled. n Born in Israel, raised in France, Netiva Caftori received her three degrees in mathematics and computer science from the University of Illinois at Chicago. Currently an assistant professor of computer science, Caftori has taught at NEIU since 1982. Her main concern is responsible use of computers in education and society. E-mail:
[email protected] References: 1. Huff, C. and Cooper, J., "Sex Bias in Educational Software: Effect of Designers' Stereotypes on the Software They Design," Journal of Applied Social Psychology, 17(6), pp. 519-532, 1987. 2. Krendl, K. A., Broihier, M. C. and Fleetwood, C., "Children and Computers: Do Sex-Related Differences Persist?", Journal of Communication, 39(3), pp. 85-93, Summer 1989. 3. Wishart, J., "Cognitive Factors Related to User Involvement with Computers and Their Effects Upon Learning from an Educational Computer Game," Computers Education, 15(1-3), pp. 145-150, Great Britain, 1990. Products mentioned: Software products mentioned are available from the Minnesota Educational Computing Consortium, (MECC) an educational software developer and distributor in Minneapolis, Minn. (800) 685-MECC.
This article originally appeared in the 08/01/1994 issue of THE Journal.