Evaluating Distance Education Across Twelve Time Zones

• 02/01/01

On the evening of September 9, 1999, the Singapore-MIT Alliance (SMA) launched the first of nine new, highly collaborative engineering subjects for the fall semester. By the end of the first year, 17 new subjects would be offered to the first class of SMA students. Over the next three years, Massachusetts Institute of Technology (MIT), the National University of Singapore (NUS), and the Nanyang Technological University (NTU) plan to develop and offer five interdisciplinary graduate engineering degree programs. The goals for SMA are high: to create highly visible, world class graduate education and research programs in areas of strategic importance to Singapore and the United States, and to form a new paradigm for distance collaboration in education, research, and "technopreneurship."

SMA is a unique and ambitious program for many reasons, one of which is that it brings graduate students from opposite sides of the globe together in one virtual classroom by crossing 12 time zones through an Internet2 connection. Through research and development from the MIT Center for Advanced Educational Services, the NUS Centre for Instructional Technology and the NTU Centre for Educational Development, the SMA program blends the use of state-of-the-art asynchronous and synchronous technology to create a dynamic, virtual learning environment.

SMA has been many years in the making. In the mid-1990s, the government of Singapore invited an MIT assessment team to review the engineering programs at their two universities. In November 1998, the Singapore-MIT Alliance was signed as the result of key recommendations from that team. The SMA program is co-directed by Professor Merton C. Flemings (MIT) and Professor Hang Chang Chieh (NUS), with Deputy Directors Professor Anthony T. Patera (MIT) and Professor Chua Soo Jin (NUS). It includes MIT, NUS and NTU faculty from the disciplines of Mechanical Engineering, Electrical Engineering and Computer Science, Aeronautics and Astronautics, Chemical Engineering, Materials Science, and Management. The first two programs, Advanced Materials and High Performance Computation for Engineered Systems, began in July 1999. The third program, Innovation in Manufacturing Systems and Technology, began in July 2000, and the final two programs, Chemical Engineering and Computer Science, will begin in July 2001.

Each SMA program has M.S. and Ph.D. tracks. Students accepted into the program receive degrees from NUS and NTU. Although courses are delivered asynchronously and synchronously over the Internet, an important feature of this program is "summer immersion." In July, many MIT faculty members travel to Singapore to meet their new students and, along with the Singaporean faculty, begin teaching an intensive pre-curriculum. SMA master's students then accompany their SMA faculty advisors to MIT for two weeks in August to continue their instruction and work in their MIT faculty advisors' labs. Ph.D. students follow in the fall and work in their MIT advisors' labs for the entire semester. It is this combination of both face-to-face time and Web-based content delivery that makes this distance learning program a unique hybrid of brick-and-mortar and virtual classrooms.

Another unusual characteristic of the SMA program is the highly collaborative nature of the degree programs and the coursework. Each course is team-taught by at least two faculty members and, in some cases, as many as six. Almost all courses have at least one instructor from Singapore and one instructor from MIT.

Driven by Technology

Although SMA faculty travel back and forth frequently between Singapore and the United States, most of the classroom experiences involve faculty and students on opposite sides of the planet using state-of-the-art technology. SMA faculty understand their students' need for face-to-face interaction with their professors, making synchronous delivery of course content imperative. Classes therefore meet early in the morning and in the evening to accommodate the time zone differences.

The use of Internet2, a very high-speed adaptation of the Internet, is important to the success of the program because it allows for high quality videoconferencing. Faculty and students can move around without causing distortion to the receiving site, talk to one another with only a two-second time-delay, and view graphics that are more detailed and sophisticated than those possible using ISDN video-conferencing.

Each live lecture is taped and then digitized and archived on the course Web site for later viewing. Other features of the Web site include a chat room, a calendar, homework assignments and reading material.

Technical delivery of the SMA program has been a highly collaborative effort as well. Technical teams at the Center for Advanced Educational Services (MIT), the Centre for Instructional Technology (NUS) and the Centre for Educational Development (NTU) have faced many of the same challenges as the SMA faculty: working with different delivery systems, time zones, and physical distance.

It was important, therefore, to conduct an assessment of the technical delivery (both synchronous and asynchronous) at a very early stage of the program. We therefore decided to survey students on their experiences with the program during the spring 2000 semester. Due to the formative nature of the program, the survey focused entirely on technical delivery and specifically excluded questions regarding faculty teaching in the distance learning environment.

Students' Use of the SMA Instructional Delivery System

Of the 70 students enrolled in the program, 61 (87 percent) responded, the majority (87 percent) of whom are studying at NUS. About half (52 percent) are enrolled in the Advanced Materials program, with the other half in the High Performance Computation for Engineered Systems program. Most (92 percent) are in at least their second semester of the SMA program, and this is the only distance education course ever taken by almost all (97 percent) of the students.

Most students (89 percent) attend classes in person on campus in Singapore, and 87 percent stated that they "always" or "almost always" attend the videoconferencing lectures. Only 23 percent, however, stated that they "always," "almost always," or "usually" reviewed the videos of the lectures on the Web. Another 31 percent stated that they have reviewed at least half of the videos, while 39 percent stated that they have reviewed only a few. Five percent of students stated that they never review the videos.

Students access the course Web site quite often, as 61 percent reported that they do so at least three to five times per week. Most (92 percent) access the site at high speed via a corporate or university LAN, and only seven percent reported that their type of connection prevented them from using the Web site as they would otherwise. Ninety three percent access the course Web site from their university or workplace, so it d'es not appear that Web access is any barrier to our students.

Only seven percent of students reported that they access the Discussion section of the course Web site at least once a week. Twenty three percent reported that they have not even looked at that section. Likewise, 15 percent reported that they access the Calendar section at least once a week, while 25 percent say they have never looked at it.

When students were asked to pick the three features of the online materials that they found most valuable, four features stood out prominently:

• 79 percent selected Lecture Notes
• 75 percent selected Videos
• 54 percent selected Assignments
• 43 percent selected Solutions

The next closest feature, Reading, was chosen by only 13 percent of students.

Students' Opinions on the Technical Aspects of the SMA Instructional Delivery System

Questions 16-25 asked students to rate various aspects of the instructional delivery system as "unacceptable," "poor," "OK," "good," or "excellent."

Aspects pertaining to the live instructional delivery system

16. your ability to hear the professor clearly over the audio system

17. the professor's ability to hear you clearly over the audio system

18. your ability to hear questions and comments from the remote location

19. your ability to see the lecturer remote site clearly through the video system

20. your ability to see slides or projected displays through the projection system

21. your ability to get the attention of the professor at the remote site to ask a question

Aspects pertaining to the archived instructional delivery system

22. your ability to hear the professor clearly

23. your ability to hear questions and comments from others

24. your ability to see the lecturer clearly

25. your ability to see slides or projected displays

To analyze responses to these questions, we first converted students' responses to a numeric scale as follows:

1 = unacceptable

2 = poor

3 = OK

4 = good

5 = excellent

We then computed the mean and standard deviation of these numeric representations. When the results are expressed as a box-and-whisker chart, as shown in Figure 1, it can easily be seen that, in general, students rated the various aspects of the instructional delivery system between "OK" and "good." These evaluation and data display techniques are useful for identifying those areas in which the development team needs to focus its efforts to improve student satisfaction with the system.

professional - - - - - - amateurish

Questions 26-30 asked students to further evaluate various aspects of the instructional delivery system by indicating how they felt about five carefully-worded statements using a Likert Scale. For each question they were to indicate whether they "strongly agreed," "agreed," had "no opinion," "disagreed," or "strongly disagreed" with the statement. These questions' results are shown in Table 1.

In four of the five questions, responses spanned the entire range from "strongly agree" to "strongly disagree," and the range in the other question spanned four of the five possible responses. Thus, we may conclude that there is either a wide range of opinions about these statements or a wide range of misunderstandings about what these statements meant. We would therefore caution against analyzing these responses too critically, but one can see that students tended to agree with each statement.

The next set of questions attempted to get at students' thoughts about the SMA Instructional Delivery System using a Semantic Differential. As in the Likert Scale questions, student responses to the questions in this section spanned a wide range, but they tended to be grouped a little more strongly than they were for the Likert Scale questions. We may be able to put a little more weight on these responses, but again we would caution against interpreting these results too strongly.

The survey presented six Semantic Differential items, each with six possible positions between them.

As before, we converted these responses to numbers, computed the means and standard deviations, and plotted them in Figure 2. It can be seen that these responses had smaller standard deviations and therefore tended to be grouped a little more strongly than responses for the Likert Scale questions. It can also be seen that, as hoped, students tended to characterize the system as "professional," "polished," "effective," "easy," "clear," and "helpful," as opposed to their semantic opposites.

The final set of questions gave students a chance to express their feelings about the system in a freeform manner. We asked two questions to stimulate their responses. Thirty-eight students (62 percent) responded to Question 37, and 22 (36 percent) responded to Question 38. Following is a sampling of representative student responses.

How did taking this course via distance learning affect the quality of the course?

• It would [be] more interactive if the professors [were] present in person, but I do not think it affected the quality of the course.
• I asked [fewer] question[s] via distance learning than on-site learning.
• Personally, I feel face-to-face interaction is always better. After class, we can approach [the] lecturer with ease.
• ...distance learning facilitated more lectures from MIT, which [is] good. But at the same time, the professor is not able to judge from our faces, if we have understood something or not, in video conferencing...
• It's not as effective as being taught locally, but given all the constraints and limitations, SMA Instructional Delivery System is quite impressive to me.
• The lecturers were fine, but consultation via e-mail is not as effective as through video-conferencing. Maybe we can arrange for that.

Please share with us any ideas, suggestions, or concerns you would like us to know about.

• Use a more distinct mouse pointer.
• Lectures should be [delivered at] a slower pace (maybe over a longer period of time).
• If we [could] have head-phones for [listening to the] lecturers, we [could] adjust the volume according to our requirements.
• It would be good if the class videos [were] put on the Web [earlier].
• As for [using] PowerPoint slides [versus] blackboard, using [the] blackboard leaves more time [for] thinking.
• Arrange some class times that are not at either extreme end [of the day], and if only local [Singapore] students attend MIT classes, evening is preferable to early morning classes. And post announcements or notes at least a day in advance.
• Reschedule the nighttime lecturers because I find very low efficiency in night classes.
• Some means of communicating with the students at MIT is [desirable].
• Ought to promote more [use of] net meeting for better or more interaction with lecturers.

Conclusions

We have cautioned against placing too much emphasis on the analytical parts of this survey, but students' free responses do seem to support the main points born out of that analysis. With those reservations stated, we offer the following conclusions:

• Students are conscientious about attending the live lectures, but their use of the course Web site is more sporadic. They find the lecturers acceptable, but we recommend making training available to lecturers so that they can improve their abilities to deliver instruction using this medium.
• Certain features of the course Web site are not used much at all. Efforts should therefore be directed toward improving the more heavily used features, particularly the lecture notes, videos, and assignments.
• Students are more pleased with the system's video than its audio component.
• Students generally find the system acceptable, but even small glitches seem to have a significant impact on their perception of the efficiency of the learning process.

As we look ahead, we hope to use SMA students' suggestions to strengthen our delivery. Faculty members are discussing the possibility of pre-recording their lectures and putting them on the Web to view prior to synchronous class time. Live time together can then be better spent in discussion and problem solving between the students and faculty. The joint technical team from MIT, NUS, and NTU is working to create new tools to facilitate more effective learning during the synchronous class time.

For more information about SMA and technology-enabled learning developments from MIT, NUS and NTU, please visit the following Web sites:

Singapore-MIT Alliance: http://web.mit.edu/sma/

Center for Advanced Educational Services, MIT: www-caes.mit.edu/

Centre for Instructional Technology, NUS: www.cit.nus.edu.sg/

Centre for Educational Development, NTU: www.ntu.edu.sg/ced/ced.htm

Special Note of Thanks: We gratefully thank our colleagues at NUS for their participation in conducting this survey. Through their diligence, we received an outstandingly high percentage of completed surveys (87 percent). Without their efforts, this evaluation could not have happened. We also thank the SMA leadership for affording us the opportunity to survey their students.

Jesse M. Heines is an associate professor of Computer Science at the University of Massachusetts, Lowell. He specializes in the implementation and evaluation of interactive, user-centered programs with rich graphical user interfaces (GUIs), and their related technologies. Heines spent 10 years with Digital Equipment Corporation, where he founded the Computer-Based Course Development Group and developed a large variety of CBT courseware. He holds a B.S. in Earth Sciences from the Massachusetts Institute of Technology, a M.S. in Science Education from the University of Maine, and an Ed.D. in Educational Media and Technology from Boston University. He has done post-doctoral work at The Open University in Great Britain, Brown University in Rhode Island, and the Massachusetts Institute of Technology, where he currently holds an appointment as a Visiting Scholar in the Center for Educational Computing Initiatives.

E-mail: [email protected]

Melinda Cerney has worked in education, both secondary and post-secondary, in a variety of roles including teacher, administrator and project manager. During the planning and implementation stages of the Singapore-MIT Alliance, Cerney served as project manager. She currently has a dual role at the MIT Center for Advanced Educational Services as project manager for a new series of online courses being developed for medical professionals and as interim manager of external relations. Her previous experience includes the administration of the MIT Advanced Study Program, an on- and off-campus continuing education program and administration of the education program for the MIT Department of Chemistry. She holds a bachelor's in Education from Michigan State University and a master's in Counseling Psychology from Lesley College.

E-mail: [email protected]