360 Degrees of Reflection

Some cutting-edge technologies are trying to take teacher evaluation to the next level. But their real power may be as self-evaluation tools.

• By John K. Waters
• 05/02/11

Classroom observation is--at best--an inexact science, historically plagued by the limitations of the human being doing the observation: There's only so much that eyes can see, ears can pick up, or pens can record in a given moment. And then there's the subjectivity problem: What looks like "exceeds expectations" to one observer may look like "needs improvement" to another. 

Pioneers in "systematic" classroom observation, like Tom Good and Jere Brophy, tried to address these limitations by creating "objective" instruments that would enable administrators to easily and quickly record observable, measurable, instructional data--such as how frequently a teacher checks understanding--and that would limit reliance on human judgment or inference. 

Unquestionably, such systematic approaches have made classroom observation a much more useful and impartial method for understanding teachers' strengths and weaknesses. Applied well, these techniques can be powerful aids to professional growth.

But the truth is, systematic instruments, as good as they are, are still plagued by the limitations of the person using them. No one can see everything going on in a classroom, so most observation protocols narrow the phenomenon being observed (teacher's use of time, for example) to the exclusion of other instructional behaviors in order to enable the observer to capture all that happens in a given period of time around that particular teaching skill. But even this may not be enough. Charlotte Danielson's much beloved observation protocols, for instance, depend on continuous note taking--not a simple task for supervisors who might struggle with graphomotor skills or active working memory.

And as much as the designers of these instruments have tried to limit the role of human judgment--to make the process truly objective--the data is still being recorded by a human being with all the preconceptions and prejudices that flesh is heir to.

Indeed, the holy grail of classroom observation, the complete and totally objective capture of all teacher-student interactions in a learning setting, has never been reached--until now?

There's reason to believe that technology can play a critical, perhaps indispensable, role in freeing classroom observations from human limitations. No one would suggest that the use of technology will make classroom observations infallible, but there is movement afoot on different fronts that suggests that the use of certain digital tools could elevate classroom observation to whole new levels of comprehensiveness and objectivity. And while the proponents of these tools say they can immensely improve the teacher evaluation process, their real value may lie in the information they can give teachers themselves for self-reflection and professional growth.

Beyond Lecture Capture
In 2009, the Bill & Melinda Gates Foundation launched the Measures of Effective Teaching (MET) project to "uncover and develop a set of measures that work together to form a more complete indicator of a teacher's impact on student achievement." In other words, to nail down a set of objective and reliable gauges of effective teaching. The project's much-talked-about first report, released last year, focused on "value added modeling," a controversial approach to judging the effectiveness of teachers based on the learning gains of students. 

Much less talked about--hardly mentioned, in fact--was a technology by-product of that first year of research: a classroom observation system that combines cutting-edge hardware and software to provide a unique tool for observing teachers and students in action--no human observer necessary.

Dubbed "Reflect," the system marries a 360-degree digital video camera with an online software package. The result is a classroom observation tool that captures a panoramic view of a lesson and makes it available on the web via password-protected video sharing and management software.

The Reflect system was engineered by Teachscape, a San Francisco-based provider of professional development technology and services for K-12 and higher education, under the auspices of the MET project.

"This was one of the biggest strictly technical challenges the MET researchers faced," explains Matthew Nathan, director of the strategic marketing and development group at Teachscape. "Classroom observation was one of the five types of data they wanted to collect. The other types--student achievement gains, student and teacher perceptions, and teachers' knowledge of their subjects--could all be collected with existing tools. But what they wanted from the classroom observation would require a new piece of technology."

The limitations of the existing tech were obvious. You really can't follow peripatetic teachers and lively students with a stationary device set up like a nanny cam, and if you employ a camera operator, he is bound to be intrusive and distracting. Moreover, the researchers wanted to record a class in toto.

The image-capture solution Teachscape developed comprises a lightweight, low-profile combination of two high-definition digital video cameras and two wireless microphones. One camera focuses on what's going on at the front of the classroom--at the board--while the other one grabs a 360-degree panoramic video of the entire classroom. The teacher wears one of the microphones (a lavaliere); the other is an ambient mic designed to pick up student discussions and student-teacher interactions.

The Reflect hardware resembles a high-tech overhead projector, and it's designed to sit at about the same place in the classroom as an overhead, "in the middle of the action," Nathan says. The digital recording is saved initially to a laptop specially configured to handle the huge video files. And the whole setup fits on a cart that can be rolled from classroom to classroom.

But cool as the hardware is, it's really the software that makes this technology so potentially useful as a tool for measuring teacher effectiveness. The video that is captured initially in the laptop is uploaded to a secure website hosted by Teachscape. There, the captured lesson is presented in two windows: one showing a stationary shot of the front of the classroom and the other displaying a panoramic view. The two views are synchronized, so that what you're seeing in one window happened at the same time as what you're seeing on the other. A slider allows viewers to advance, rewind, and pause the video, while another one allows users to adjust the relative volume between mikes. Users can insert the slider along the scroll bar to facilitate quick jumps to different moments of the lesson. And there's a tool for rotating the 360-degree view to any angle on the classroom, to tilt up and down, and to zoom in and out.

Beneath the two video windows is a discussion board where teachers and administrators can post comments on the video. This feature is billed by the company and the MET project as a tool for sharing feedback, comments, and reflections about the lesson.

As of this writing, the Teachscape Reflect system hardware is a prototype. The MET project worked with nearly 3,000 teacher volunteers in six predominantly urban school districts across the country to capture and score more than 12,000 lessons. Nathan says he expects to see commercial implementations of the technology in K-12 school districts later this year.

All Data All the Time
An inevitable criticism of a classroom video capture system like Teachscape Reflect is that it feels like a tool of Big Brother.

Tony Davis, principal consultant at Colorado-based Mid-continent Research for Education and Learning (McREL), acknowledges that these technologies "might seem a little bit Big Brother-ish," but they are "part of the emerging conversation" about measuring teacher effectiveness.

McREL has its own technology-based classroom observation tool, Power Walkthrough, a handheld device that facilitates the recording of classroom observation data. The organization has collected data on more than 30,000 classroom observations, so Davis speaks with some authority when he predicts "these kinds of technologies are going to be very attractive, given the demands on administrators these days. How can you as an individual get into all those classrooms, or even those buildings, and provide a fair assessment of a person's work? Classroom capture technology could well be the answer."

Nathan sees an additional advantage. "If you're a district that's considering moving toward a more rigorous program of teacher evaluation, you have to be fair to the teachers. You have to let them know what it is you expect--what the teaching you expect from them looks like. You need to give them a model to aspire to and advance toward, so the evaluation means something. These tools allow you to capture exemplars that make highly effective teacher evaluation possible."

Yet, as a performance evaluation tool, the Teachscape Reflect technology is still somewhat a prisoner of the same human limitations that held back its non-digital predecessors: a supervisor has to watch the video, and can only look for one, maybe two, teaching phenomena at a time. Certainly the manipulative capabilities of the software help the supervisor to zero in on the skills they are observing for, but repeated video viewings or recordings are still necessary to take note of other teaching skills. So, it may not be a time-saver.

In addition, there's that pesky human subjectivity--whoever is watching the video is interpreting what he or she sees. Yes, having video-based evidence is much more reliable than real-time observation or memory, but at the end of the day, human judgment will have some impact--good or bad--on the observation process. 

But what if there were no observation in teacher observations? What if the human element were removed completely?

Mickey Revenaugh, cofounder of Connections Academy, says that the tools for measuring effective teaching are built into systems for online learning and that they may be able to do more for capturing teacher performance data than any human observer can do.

Connections Academy is a provider of full-time virtual public education, which is available in 22 US states. The company recently launched Connections Learning, which brings the same virtual curriculum, technology, and instructional resources to schools and individuals not affiliated with the full-time Connections Academy program.

Online systems, Revenaugh says, almost automatically weave together all of the teaching and learning that happens online, making it possible to keep track of very granular data about both student performance and teacher activity.

"It's actually very simple to correlate teachers with how their kids are doing on our system," she says. "We engineered it that way from the ground up, because we realized that, in online learning, the substitute for the eyeball-to-eyeball contact you get in a traditional classroom is data. Our teachers depend on lots of data--how the students are progressing through their lessons, where they're having trouble, what interventions seem to work for them and which ones don't, and ultimately their academic success at the end of the year."

In Connections Academy's comprehensive management system, called Connexus, student performance on every lesson is captured by the system, Revenaugh points out. "Not just every lesson," she says, "every piece of every lesson."

The Connexus system is a digital learning platform designed specifically for K-12 instruction and administration. It combines three management systems--learning, content, and student information. It also includes a full communications suite and asset management system. It's used to run the company's virtual school programs, as well as online courses a la carte.

Along with the typical student performance metrics, the Connections Academy system measures what Revenaugh calls the nuts and bolts of good practice--things like how long it took a particular teacher to grade a student's work and get it back to him, how quickly another teacher responds to her e-mails, and how often a teacher is holding web conferences.

"All of our best practices are laid out for the teachers at the beginning of the year," she says. "And our principals can see the data about that anytime they want to, not just at the end of the year or when they're observing a teacher's class, but in real time."

This kind of continuous data capture is particularly important to online schools, says Revenaugh, "because the technology allows us to collect so much data, we catch problems early on. It also helps us to foster a really close supervision and coaching relationship between the instructional leader and the teachers."

Those relationships are enhanced by a physical proximity that is rare among virtual faculty; Connections Academy teachers and principals actually work together in an office space during the day, which allows for just-in-time professional development, Revenaugh says.

"What's actually most exciting to me about this system and systems like it," she says, "is that it puts powerful analytical tools into the hands of teachers. We work hard to create an environment in which people are reflecting on their own practice all the time, and working toward continuous improvement. And here, that's based not just on how they feel, or how pretty their classroom is, or what a superstar presenter they are, but on real data."

The Power of Reflection
Indeed, Manuel J. Rivera, CEO and cofounder of Global Partnership Schools, a New York-based educational consultancy, sees these cutting-edge systems as potentially game changing technology, but less for teacher evaluation than for supporting teacher reflection.

"I'm assuming that this is the first of a generation of these kinds of tools," he says of the Teachscape Reflect product. "And I'm hoping that they'll be used primarily to help teachers to grow and develop. That's what we ought to be capturing. The question becomes, how do we use this technology to track those areas of development that the teacher needs to work on as a professional educator? And then how will that educator best maintain those artifacts in their portfolio as evidence of what they've done to address their individual needs as a professional?"

Teachscape insists that "self-reflection and evaluation" is the primary use case for the Reflect technology. "It's virtually impossible for teachers to see themselves teach," Nathan notes. "Videotaping happens; it's something the profession recognizes is important for new teachers when they're in their early days of service. But then it sort of goes away. Because we store captured lessons in an online library, people get to see themselves teaching throughout their careers."