An Evaluation of Alternative Technology Based Instructional Formats
##AUTHORSPLIT##<--->
Students have always brought their personal learning frameworks to colleges and universities. As members of the Academy, we have encouraged and often applauded diversity in learning styles and background. However, that same diversity of learning styles can cause problems for students, institutions and faculty when basic student skills are uneven. This is especially true in computer literacy courses. Some students may have great mastery of the material from work or other venues, while others, often older students, may have little or no familiarity with computer operations or terminology. Introductory skill courses such as those designed around MS Office may include students who have never used a mouse as well as those who have mastery of 50% or more of the required skills. Those who do not know how to use the mouse require individual attention while those with extensive skills are bored and often disrupt the class with attempts to demonstrate their knowledge.
Costs and Resources
Limited faculty resources and the high costs of computer equipment exacerbate these classroom management problems. Alternatives discussed and used at many institutions include larger sections or using junior faculty or teaching assistants. These solutions may reduce costs but may also reduce quality. If quality is reduced, successive courses that rely on the skill courses to develop student proficiency may find that extra class hours are required to develop missing skills. In such cases, arguments are sure to arise whether the material was covered (in the skill course), covered well, or conveniently forgotten by the student.
Assessment
A partial solution to student knowledge level is assessment. Assessment, when properly applied, can help an institution, instructor, and student determine if specified skill sets have been mastered. Assessment can take place before enrollment in a course, during the course, or as part of the subject mastery evaluation at the end of the course.
Part of the assessment process requires setting objectives and then assuring that exams and related materials focus on, and link to, those objectives. A key to successful assessment is developing objectives that are both measurable and acceptable. The objectives must be acceptable to faculty, administration and other publics. Those publics include students, parents, legislative bodies and accreditation agencies. External validation has become a phrase that underlies assessment.
Proper assessment procedures also provide guidelines for pre-assessing students and then tailoring a course or curriculum to deficiencies identified by assessment tools. Such a process can be labor intensive and may conflict with institutional needs to control costs.
Other Academic Institutions
Courses and curricula are not developed in a vacuum. Students are becoming less place bound and often seek educational alternatives. State mandates for "equivalent" education at all state universities can mean a loss of institutional identity. Mature students may also "shop" for the institution with the most convenient class and registration schedule. The rapid deployment of distance learning, especially Internet-based courses and degree programs, adds an additional competitive element to the educational options available to students.
Overview of Alternatives
The authors teach at Columbus State University, a regional institution with a mixture of traditional and non-traditional students; most students are primarily non-residential and part-time. Both authors have taught introductory classes in computer skills and have worked with students possessing all levels of computer competencies.
This paper discusses three alternative approaches to traditional classroom instruction. The guidelines and comments provided in each section and at the end of the paper are designed to help other faculty faced with finding instructional solutions within environments similar to the ones described above.
Three alternative approaches are currently in use at Columbus State University:
- Vendor (NETg; www.netg.com) developed Internet-delivered courses
- Vendor (Course Technology; www.course.com) developed tutorials and tests
- Instructor developed and vendor-assisted Web-based instruction
The first two alternatives discussed are self-paced courses. While the last alternative could be self-paced, this alternative has been used more as a support device rather than the primary delivery vehicle. Several options are discussed within this alternative.
NETg
The depth and scope of the technology revolution has caused a dramatic increase in the number of course delivery alternatives. Many colleges and universities are using the Internet to deliver degrees, courses or course components. If colleges and universities are to remain competitive in the higher education market, it is imperative that professors and administrators consider this option. Developing and implementing alternate methods of course delivery, such as distance learning via the Internet, helps meet student needs by providing flexible scheduling and 24 by 7 access to course material. Internet courses can also reduce travel time and minimize conflicts between job and home. However, the traditional model of a University, where the professor lectures and the student listens, is not congruent with the Internet model of content delivery. Therefore new course management tools are needed.
Columbus State University currently provides instruction for selected courses via the Internet. These courses include Technological Solutions and components of certain courses, such as the Introduction to Computer Information Systems. Software developed by the National Education Training Group (NETg) was purchased for use in this endeavor.
NETg Components
NETg is a Web-based interactive, self-paced tutorial in a simulated environment. Courses available include application packages such as Word 97 and programming languages such as C++. Pre-assessment and end-of-unit "mastery" instruments are available for each learning unit. The material presented in each tutorial is modified based upon the student's weaknesses identified in the pre-assessment process. Course information is disseminated via a Semester Class Schedule Booklet; hardcopy and Web page versions (www.colstate.edu) are available. Students also learn of course offerings via "word-of- mouth" from advisors and other students. Students are often excited about the time independence provided by the NETg courses and request course access before classes are scheduled to begin.
The Class Schedule Booklet clearly identifies each NETg course and provides additional information including the class URL. Students contact the course professor either by e-mail or telephone to request the proper procedures for accessing the course and logging on. Student access accounts are created and passwords are distributed by e-mail or during the course orientation, held the first week of the term.
Student Interaction with NETg
Students are able to complete each unit of course material at their own pace, receiving immediate feedback to responses made through simulated conditions. Students may work through the tutorials as many times as needed and repeat the mastery tests as many times as they wish, for the purpose of improving their scores.
Student progress is tracked electronically. Therefore, progress and performance data are accessible any time by the student or professor. Access is controlled via password. The Course Manager software registers the student's mastery scores, as well as the number of times the lesson was accessed and the amount of time spent on the lesson.
Unfortunately, NETg's reporting scheme can frustrate students. Following the mastery test, the immediate score will be different from the one displayed in the Student Report. Test scores are not updated until the subsequent logon.
Students with partial or complete knowledge of the programs claim that taking the pre-assessment tests is a waste of time. Students with no prior knowledge dislike the constant and negative feedback and low scores received from the pre-assessments. When students find this annoying, they simply discontinue taking pre-assessments and work through the entire tutorial.
Mid-term and final examinations are given online and students complete tasks in live, not simulated, programs. Working in the active environment challenges many students, especially those who have not practiced this type of application. In addition to the active environment, the examinations are problem-oriented, not task-oriented. In view of this, professors should provide problem-solving situations throughout the semester or quarter, preferably in a live program project.
If a student withdraws from the NETg Internet course, a questionnaire is sent to identify the reason(s) for withdrawing. The most often cited causes for course withdrawals were:
- lack of adequate computer hardware at home
- Job or family responsibilities
- underestimating the amount of time needed to complete the tutorials
Required Computer Skills Class
The College of Business at Columbus State University requires that all students be computer literate and proficient in Windows 95, the MS Office 97 Suite, Internet and Web use, and automated library software. For a number of terms, this course was taught using a traditional lecture-discussion-lab format. In this format, instructors worked from a common foundation syllabus, text and course objectives, but developed and administered projects and tests independently of each other.
Earlier versions of the course used Symphony, WordPerfect, Office 4.2 and other software packages. In those earlier terms, most students were unfamiliar with basic computer concepts and skills. Hence, students entering the course generally had a uniform starting point: zero skills. In recent years, students came to the class with a wide range of skills, frustrating attempts to define a common starting point. Attempts were made to separate the course into its various components and award credit based on skill development. This approach was a scheduling and administrative nightmare soon abandoned. In addition, testing methods were primitive, time consuming, and test materials were not as congruent with learning objectives as desired by instructors. Simultaneously, program growth was draining resources as more and more faculty were being allocated to the skills course and away from other tasks and courses. It was in this environment that alternative methods and tools were sought.
A number of textbook vendors were contacted. Before contacting vendors, criteria were established to evaluate alternatives. The abbreviated criteria list included:
- The course would be self-paced using extensive tutorials that relied on a live rather than a simulated environment. We believed that students should be taught using the environment they would use in other classes, not an artificial environment or simulated desktop. In this environment, the faculty would establish testing times, include additional projects, and provide support and coaching while reducing the amount of time lecturing.
- Testing would incorporate a live environment rather than a simulated environment and testing modules would be of sufficient breadth and depth to permit testing at various time periods without compromising test integrity. In addition, the system should have some form of pre-assessment methodology.
- Materials provided by the vendor would have to include the full MS Office 97 Suite and Windows 95.
Course Technology e-Course
Course Technology's solution, e-Course, using the textbook, Microsoft Office 97 Certified with Microsoft Windows 951 (ISBN 0-7600-7224-8) was selected. The textbook includes a CD-ROM with tutorials for each of the MS Office 97 products and Windows 95. The tutorials on the CD can be loaded on a local machine or run from a server. If loaded on a local machine, the tutorials can either be installed with minimum disk space (which requires that the tutorial CD be present in the CD-ROM drive) or full install, which places all files on the hard drive. Similar functionality is available for server installations. We selected server installation with minimum disk space requirements. This means that students must place their tutorial CD in the drive to access and use the program. The startup screen is shown in Figure 2.
Clearly, e-Course satisfied criteria one and three listed above. Criterion two proved to be serious problem. An optional module with the text package is a product called SAMs (Skills Assessment Manager). SAMs, as demonstrated by a vendor representative, included a live testing environment. Unfortunately, what was demonstrated in mid 1998 was not fully functional at the start of the fall term. We therefore turned to the alternative offered by Course Technology: e-test.
Course Technology's e-Test uses a simulated environment that has the look and feel of a real desktop environment. The testing module has adequate security features and tests are immediately evaluated. Regrettably, those taking the test cannot use the Help Functions in the software package and the instructor cannot turn off the exam once a student has started it. Those wishing to limit the amount of time a student can spend taking an exam cannot do so. Each exam however was usually completed within 40 minutes.
Course Technology promises to have SAMs operational in early 1999. This version of SAMs is claimed to have additional features and improved testing options.
Evaluation
Initially, problems were encountered while installing e-Course on the Columbus State University network. These problems were more related to the CSU system than to problems with e-Course. Those selecting e-Course should work closely with computer center personnel and Course Technology technical specialists to minimize problems.
The students seem enthusiastic about the self-paced approach. The course was structured so that the first three weeks of class were an introduction to class procedures, computer use and instructor-led exercises. This guaranteed that all students received the same instruction and had an opportunity to ask questions. One of the essential tasks in the early part of the course was how to logon and how to effectively use the university's e-mail system. Students were informed that it was their responsibility to master the materials and check their e-mail on a regular basis. They could then work elsewhere on their assignments or come to the lab, where the instructor would be present.
The syllabus specified testing dates and due dates for instructor-developed projects. As the course moved through the term, attendance at labs dwindled as students found that they could work at home or places of employment. Those who attended the labs could be characterized as those who either needed additional help or who wanted the structure of a fixed time period. Anecdotal information indicates that the shift from attendance to performance requirements encouraged many who did not already own a computer to either purchase one or to use one belonging to a friend. Several indicated that their spouse was also "taking" the course though not officially enrolled.
One of the primary difficulties with the new course structure was the shift for instructors from lecturers in control to coaches; the tutorials now did the "teaching." A problem with coaching is the need to be proficient on any topic in Office 97 at any moment, not just the topic prepared for that day's lecture/presentation.
Students really seemed to appreciate the ability to proceed at their own pace and the immediate feedback. e-Course tutorials include self-testing and tracking software. Since the exams are closely linked to the tutorials, and the tutorials are focused on MS MOUS certification, assessment has become more direct and objective. In addition, students can test out of the class early as they master topics.
While the structure and content of the course has changed from the traditional lecture format, it appears that students are achieving more. In addition, the drop rate is less and students have taken the extra effort to make positive statements concerning both the course and the text.
Lost CDs
One problem not anticipated were students forgetting to retrieve their CDs from the drive when they left the labs. Unfortunately, not all CDs were recovered. Since this was a "test" mode of the product with Course Technology, the company provided several extra CDs that were loaned to students for the remainder of the term. Institutions adopting this text should warn students to remove CDs or be prepared for similar problems. Students should use non-alcohol-based pens to write their names on the CD. While this d'es not guarantee that CDs will be returned, it d'es minimize ownership arguments.
Tracking Progress
Tracking software included with e-Course monitors a student's progress through the tutorials. A "tracking" disk kept by the student records the amount of time spent on each tutorial and the scores on self-assessment tests. When a student comes to an instructor for additional help, the first step is to ask for the tracking disk. On many occasions, the response from the student is, "I guess I have not been spending as much time on the tutorials as I should." For those students who have spent time, the tracking disk can help determine those areas that need remediation. Initially students were required to present a printout of the tracking disk summary at test time. This encouraged them to complete the tutorials and gave an overview of the relationship between test performance and the amount of time spent on the tutorials. As was expected, there was a positive relationship between test results and tutorial effort. Maintaining the printouts of each student's tracking disk became an encumbrance and was discontinued after the first exam.
Projects
In addition to the concern with e-Test and SAMs, the material d'es not lend itself to projects that integrate all the Office 97 functionality. Likewise, the tutorials, while excellent, emphasize skills over critical thinking. Since e-Course cannot cover all topics, instructors should still incorporate projects and examples that emphasize those curriculum goals not covered in the tutorials. Projects should be assigned so students will learn various tools and techniques via the Help Function. One project used in the course required students to create a form letter in Word that retrieved data from an Access database. A query linking tables had to be created in Access so that Word would retrieve the correct data. The mail merge in Word further limited the data selection. The query from Access was exported to Excel where additional analyses and charting were performed. One of the Excel charts was linked and embedded in a PowerPoint presentation.
21104-24362
Evaluation
Unlike NETg, e-Course d'es not tailor the tutorials based on pre-assessment. Hence, all students, regardless of their prior knowledge, are almost required to take the full tutorial. The text and the CD do provide an overview of the tutorial contents so that a decision can be made whether or not to take a tutorial, but that is a subjective rather than an objective decision. Students can skip to the self-assessment at the end of the tutorial if desired.
e-Test is being supplanted by SAMs2. Either of these testing modes can be used to determine whether a student has mastered the course material. The use of the tutorials and testing modules has allowed us to devote more time to those students who need more individualized instruction.
Those who might adopt this approach are cautioned to set a structure for testing and mandatory class meetings. Without a structure for testing, students may either postpone all tests to the end of term, or attempt to take all of them early in the term. Without procedures, testing will subtract from the time available to coach students who need additional assistance. All exams were closely proctored to assure that adjacent students did not benefit from the success or failure of adjacent students. Questions in each test are not randomized, so adjacent students take the same questions in the same order. While e-Test provides four tests on each subject, we believed it too cumbersome to open and assign different tests to each student. We also wished to reserve tests for those who had excused absences or who wished to take an exam early.
After students had completed the first tutorial, one of the four exams was "unlocked" to allow students to preview testing in a simulated environment. The score on this exam was not recorded. Separate e-Test exams were administered for Word, Excel, Access, and PowerPoint. Students were permitted to retake one of these exams under certain restrictions. One restriction was that the last score, whether higher or lower, would be recorded.
Students were required to check their e-mail twice a week, however, this was an inefficient and inconsistent method for communicating information to all students. Following this, part of the scheduled testing day had to be reserved for general announcements. Next term, mandatory attendance days will be included in the syllabus and e-mail usage will be included in the grading scheme.
Internet-Based Instruction
NETg may represent one end-point in instructional use of technology: vendor supplied and developed Internet-delivered courseware. e-Course is another end point since it is also vendor developed and delivered courseware. The primary difference is the delivery method; e-Course is directed toward campus-based instruction. In between these two alternatives are options where technology is more of an assistant rather than the vehicle. This section discusses some of those alternatives used by the authors.
CyberClass
CyberClass is a software package developed by the HyperGraphics Corporation (www.cyberclass.com) to support Internet-enabled delivery of course material. Internet enabled delivery includes delivery of all course materials at one extreme (which makes it similar to NETg) or it can be used to provide additional course materials for more traditional, instructor-led courses.
To create an online presence with CyberClass, the instructor logs onto the site (guest registrations are available at the site) and creates the course. The course, which resides on the HyperGraphics' server, can include bulletin boards, syllabus, assignments (readings, turn in, online), links and testing. The template is easy to use and content can easily be modified. Our first experience required about one hour to create a working Web site. Students (and the instructor) need a password to access the created course site.
HTML Coding
Other course sites have been created using HTML coding and code generators such as Adobe Pagemill and FrontPage. CyberClass provides an easy-to-use template for creating common course elements and no coding experience is required. However, the structure and formatting is less elegant than could be created through HTML or other page generators. In many ways, CyberClass is similar to WebCT (Web Course Tools; University of British Columbia; homebrew1.cs.ubc.ca/webct/webct.html). WebCT provides a template that course developers modify to reflect course content and instructor style.
CyberClass has a number of publishing partners. Depending on the publisher and text, it is possible to include vendor-developed tests on the Web site. Student reactions to the site created by one of the authors received mixed reviews. Much of the initial criticism centered on a desire to have a hard copy of the material listed on the site. Students clearly did not want to continuously log on to review calendars and assignments. Part of the criticism was satisfied in later classes by more carefully reviewing the appropriateness of material included on the site. A calendar and grading information were distributed in hardcopy format. This seemed to satisfy the need for transportable information and reduced the amount of printing from the site.
One of the options with CyberClass is online submission of class work. While this provides a clear time-date stamp on all work, grading e-mail assignments was difficult and inconvenient. No matter how good the monitor and e-mail editor, it still seems more efficient to work with a hard copy that can be easily marked and returned. In addition, students seemed to expect that homework would be graded instantaneously when submitted via e-mail. Students often called and e-mailed asking if the assignments had been graded. While some inquiries are to be expected in all classes, it appears that electronic submission engenders a higher inquiry rate.
For those unfamiliar with HTML coding, code generators or WebCT, CyberClass is an excellent vehicle for supporting a class. For those who do not have space on a server, it may be the only alternative. CyberClass combines ease of use and functionality in an easy-to-learn package. Those wishing more robust or complete Internet classes should investigate WebCT or similar packages.
Summary
All of the methodologies presented and evaluated in this paper have advantages and disadvantages. None of the packages met all instructional needs. The same comment could also be made for the use of chalk or transparencies. All of the products and methods discussed do provide alternative instructional formats that meet specific pedagogical needs. All involve tradeoffs. While e-Course has great tutorials but the tutorials cannot be modified based on assessment. NETg has great pre-assessment, but ease of use and modality are concerns.
The key to successful selection, adoption, and use of any of the tools discussed begins with a clear assessment of institutional needs, student capabilities, resource requirements and instructional objectives. The second key is continual monitoring of student use and success rates. The third key is to maintain contact with both vendors and your local computer center to assure that support is adequate and that new releases and patches are made available on a timely basis.
1 Course Technology states in the book, "If you take this entire e-Course, you will be prepared to take exams for Word Proficient, Excel Proficient, Access Expert, and PowerPoint Expert and become certified as a Microsoft Office User Specialist." 2 Course Technology's e-test is available on the instructor resource disk provided by the company. It can be installed on a single machine or implemented via network installation. We selected the network installation. An instructor logs on to the network and unlocks one of four exams available with each module. After a short time period, the instructor locks the exam. Students are told to set the disk and print options. The print option prints the results and the disk option saves the results to a floppy disk. The disk not only provides additional backup in case of printer failure, but also provides additional testing detail. There is no charge for e-test. SAMs is an added cost and has different security features. If an institution uses SAMs, the cost of SAMs can be part of the text. SAMs includes as security certificate and other software to help manage a course.