Higher Ed Is My Co-Pilot

A Massachusetts middle school shares the benefits of a tech-based initiative with a team of Ivy League researchers.

• By Jennifer Demski
• 02/11/11

Ann Koufman-Frederick isn't one to pass up an opportunity to put her district at the forefront of emerging educational technologies. As the superintendent of Watertown Public Schools, in Watertown, MA, she encourages her teachers to participate in tech-based pilot and research programs run by higher ed institutions. And why shouldn't she, when her district enjoys such notable neighbors as Harvard University, the Massachusetts Institute of Technology, and Boston College?

"These pilot programs are fantastic professional development opportunities for my teachers," Koufman-Frederick says. "They're learning so much about what's on the horizon and what's possible in teaching and learning."

A former instructor of educational technology, Koufman-Frederick sees value in these K-12-higher ed experiments that flows the other way as well. "It's very important for higher ed researchers to be connected to real students and teachers in schools so that they can see where schools are right now and where they need to go," she says. "Some of these research projects are very forward-looking. By participating, we help higher ed researchers understand what the real possibilities are in schools."

Watertown has taken part in a variety of pilot programs through the Harvard Graduate School of Education, thanks to its close proximity to the university as well as to the professional network Koufman-Frederick formed while teaching courses for Harvard's extension school and for WIDE (Wide-scale Interactive Development for Educators) World, the university's online professional development program.

One such pilot the district is now involved in is Harvard's Virtual Assessment Project, whose goal, as stated on the program's website, is to develop three "single-user, immersive, three-dimensional environments to assess middle school students' science inquiry skills." The three 90-minute assessments are aligned with National Science Education Standards and designed to run on schools' current computing environments. Another carrot dangled by the researchers is maintenance-free participation: The assessments "will require little preparation for users and no additional paper-based materials."

A Better Measure of Skills
The project also has a larger aim in mind. The researchers hope to modernize how schools assess students' abilities in science by showing that these immersive environments are superior to conventional paper-and-pencil tests for measuring science inquiry learning. Each of the three assessments requires students to navigate their avatar through a virtual environment to explore the cause of a health issue among a species within its own ecosystem.

As an example, "Save the Kelp!" brings students to an Alaskan bay to investigate the decline in the kelp population. As the student directs the avatar to interact with the virtual environment, the software automatically creates a detailed, time-stamped event log of the student's actions. The Harvard team theorizes that those actions will more reliably and accurately reflect the proficiency of a student's inquiry skills than a traditional, lab-based test can.

The assessments "are very gamelike," says the project director of the pilot, Jody Clarke-Midura, a research associate at the Harvard Graduate School of Education. "They have the look of a video game, but the students are solving a problem, investigating, and engaging in inquiry."

When Harvard approached Koufman-Frederick with the details of the project, she saw how the technology could have a real effect on K-12 education.

"We must do better assessments, especially for science," she says. "Even though the assessment is virtual, it's hands-on. It gets at more than just assessment of content. It gets at how good kids are at critical thinking, figuring out puzzles, doing research, and collaborating with a team -- even though the team is all virtual. On paper-and-pencil tests in science, that never happens."

Equally enthusiastic about participating in the project were two science teachers at Watertown Middle School, Elizabeth Kaplan and Karen MacAulay. "What really got my attention wasn't the technology, but what the technology was assessing," Kaplan says. "I love that it assesses the students' inquiry skills. Every single year, I'm trying to teach them how to be scientists, and we struggle to have them carry those skills on from year to year."

MacAulay was intrigued by the potential broader impact the pilot could have on her students, who would be interacting one-on-one with the Harvard researchers as they took the assessments, in addition to being exposed to the methods and processes used by actual scientists working out in the field.

"Those one-on-one sessions let our students see how real science is done, how a science evaluator or researcher looks at the world and collects data, and then collates, analyzes, and evaluates that data," says MacAulay.

The buy-in from faculty members is important to Koufman-Frederick. It's one of the conditions she factors in when determining whether or not a pilot would be beneficial to her schools ("Should You Get on Board?"). "The students are always enthusiastic about participating," she says, "but do I have teachers who are interested in the topic?"

One reason Koufman-Frederick was able to get her teachers' support for the Virtual Assessment Project was that participation in the pilot didn't require them to dedicate any lesson time or adjust their curriculum. Students take the assessments apart from the classroom.

"We used the professional development time built into our weekly schedule to organize and arrange for students to get pulled from their classes in order to take the assessment," MacAulay says, "and we made sure that the students who participated knew they'd be responsible for making up any classroom instruction that they missed while they were taking the assessment. It hasn't been burdensome for us at all."

'Invaluable' Feedback
The Virtual Assessment Project is being piloted in three phases. The development phase took place throughout the 2009-2010 school year, during which the Harvard researchers carried out the one-on-one work with the students -- "think-alouds," the scientists called them -- interacting with each student as he or she worked through the assessment.

"As they go through the assessment," Clarke-Midura explains, "they tell us what they're doing and what they're thinking, and we ask them probing questions about their methods and their decisions."

The approach allows the researchers to draw information out of even the quietest students or those who have difficulties articulating their thoughts. "Some students will be totally candid and say, 'Oh, this is boring because of x, y, and z,'" Clarke-Midura says, "whereas with other students, for example, if you see them struggling during the assessment, you really have to ask them about it. It's just a matter of developing a rapport with them."

Clarke-Midura considers the on-the-spot student feedback an "invaluable" contribution to the design process because it enables the researchers to see whether the technology is achieving the desired results.

"It's always helpful to get students' perspectives," she says, "because as a designer, when you're designing a new tool or technology, how you're implementing it might not be how it actually plays out in practice. It's like a cognitive test analysis. We want to make sure that the students are interpreting the assessment, and the design of the assessment, as we intended."

By the end of the school year, a think-aloud session had been conducted with nearly every eighth-grader at Watertown Middle School.

The researchers used the students' input to make improvements to the interactive palettes of the assessment, incorporate more drag-and-drop functionality rather than relying on text-based interactions, and add a series of quests to provide more of a "game" feel. MacAulay has noticed the difference.

"The refining that was done has really brought [the assessment] up to a level that will engage the kids even more," she says. "The new version also has a high level of scientific questioning and technical expertise, so I'm hoping that it really does the job that it set out to do."

The new and improved assessment made its debut in the second phase of the pilot, which began in January with a new crop of Watertown eighth-graders. The pilot will conclude in May with a large-scale third phase of testing performed on a single day. To get a fresh crop of students who have yet to be exposed to the immersive environments, the researchers will have every seventh-grader in the district, along with students at schools in the states of Ohio, New York, and Massachusetts, take two of the project's three assessments.

The objective is twofold, according to Clarke-Midura. The first is to confirm the validity of the tests by checking for consistent student performance on both of them.

"Second," Clarke-Midura says, "we're studying the assessments' ability to scale up to the classroom level." That, she notes, is the project's ultimate purpose -- to discern the feasibility of using these immersive environments in the classroom setting in place of paper-and-pencil tests to measure students' science inquiry learning.

Whatever that determination turns out to be, it won't influence the positive feelings Kaplan and MacAulay both have about their experiences participating in pilot programs with higher ed institutions.

"Anytime there's an opportunity for a partnership with a university, or with any company that does K-12 outreach," Kaplan says, "if the opportunity is good and useful you've got to jump on it."

Both teachers say that their exposure to the immersive environments created by the Harvard team has inspired them to consider ways to incorporate similar open-ended scientific inquiry processes into their own classroom activities.

MacAulay also describes a more subtle benefit -- the opportunity to remain excited about the teaching profession. "A teacher can go stale in the classroom because we are so isolated," she says. "You need to be in touch with what's going on out there in the research community, beyond the classroom, so you stay stimulated.

"I think classroom teachers need more connections with higher learning," she adds. "That's where policy change occurs, and I want to be in touch with what's happening and the direction that education is taking."

FOR MORE INFORMATION on the Harvard Graduate School of Education's Virtual Assessment Project, and to view the immersive environments used in the project, visit here.