Closing the Data Gap

There is a veritable chasm between the shiploads of data collected on student performance and the teachers who could use that data to make instructional decisions. How do districts bridge that divide?

• By Dian Schaffhauser
• 01/07/13

The education mantra of data-driven decision-making has morphed into "accountability for results." Teachers are charged with using data to improve their instructional practices, which in turn is expected to improve student outcomes. What's not so clearly laid out is how to do this.

Today, if a teacher wants to understand how a particular student is doing against learning standards, it often can involve trips to multiple data streams: one to the student information system (SIS) to get basic enrollment data, another to tap into past performance data maintained in the state reporting system, a trek into the learning management system (LMS) for recent work, a quick foray into the grade book, slogs through content-specific programs, and on and on. Each system may have its own password, not to mention its own way of reporting data. Compiling the data for a cohesive view may require copying it into some kind of spreadsheet concoction that has to be updated manually by the teacher who wants the information.

Is it any wonder that even the most committed educators blanch when talk turns to using data in the classroom? Policymakers and leaders at the national, state, and district levels may believe the use of data can help solve instructional problems, but "teachers are not so sure," says Bill Rust, a research director for Gartner.

Nicole Catapano, former coordinator for data analysis with an educational cooperative in New York and now a predictive analyst and solutions architect for IBM, sums up the challenge this way: "Time is precious, resources are few. So how do you maximize the time and get the most efficient information in the hands of the teachers so they can do something with it as soon as possible?"

In other words, how do we close the data gap?

Why the Data Gap Even Exists
A 2012 Gartner project, "Closing the Gap: Turning Data Into Action," decided that to address the so-called data gap, a good place to start was with the LMS and SIS. After all, notes Rust, those are the two data sources used most often by schools. The challenge, he says, is "to prevent those systems from becoming vacuum cleaners without a bag. We collect a lot of data about the kids, but unless we use it for something, it's just bits and bytes."

Gartner's research of 716 districts, schools, and technology leaders and 1,010 teachers found that as districts choose SIS and LMS applications, their leaders don't emphasize the use of data in the classroom as part of the selection and implementation process. So, not surprisingly, teachers turn out not to use the functionality of the systems as fully as they could.

The lack of use is partly due to a professional development problem: Almost three- quarters (70 percent) of responding teachers rated their SIS PD content as "weak." Furthermore, not even a quarter of teachers (23 percent) said that the data from their SIS helped them plan classroom activities; approximately seven in 10 teachers did not believe that their SIS helped them solve "important problems in my classroom."

There was slightly more enthusiasm for the LMS systems teachers were using. However, even in that regard, only half reported that they could use the data to increase student achievement.

Part of the problem, Rust believes, is that SIS- and LMS-related activities are currently viewed in the districts as being "owned" by IT. So it's little wonder that districts aren't exploiting their LMS and SIS in the classroom. After all, it used to be "all that information lived in the guidance office and administrative suite," he points out. Now, there's a different set of users: teachers, students, and parents. "The assumption is, 'Wow! We can throw that data right on that screen and the teacher will know how to make adjustments for every child in the classroom.'" But that, of course, doesn't happen.

Rust has made a study of the districts that use SIS and LMS systems most effectively in the classroom. Their best practice follows the recommended change management pattern: They hand the running of the LMS or SIS evaluation, selection, and deployment project over to the "education side," whether that be the office of accountability or a chief academic officer. They put together a cross-functional team with representatives from every group of stakeholders. And they turn user training not into an event but a process to help users mature in their use of the applications over time and to address the inevitable system changes that are bound to happen.

A True Data Infrastructure
While academic ownership of the SIS and LMS is clearly a necessary condition for data-driven decision-making, both IT and education-side leaders can make the mistake of thinking that the systems, in and of themselves, are all the infrastructure a district needs to ensure data integration. As officials in Everett Public Schools have learned, it's not that simple.

Of 6,800 students at the quickly growing Massachusetts district, about a thousand are considered special needs. Explains Assistant Superintendent Tom Stella, as many as 50 to 60 percent come from families where English is not the first language. Rather than pulling kids out into small classrooms catering to their unique learning difficulties, the district has chosen to keep them in the regular classrooms. "They're all together; they all access the curriculum in a different way at a different rate; and that presents a significant challenge to teachers."

To deal with the multiple needs of students, Stella estimates that the district has access to 20 to 25 different assessment programs used in specific grade levels or special groups of students. To keep up with demands of such a diversified data stream, Everett has been involved in "every major initiative" that offers the promise of data integration and is pushed by the state department of education. That's how the district became involved in a pilot of the latest data initiative from the Shared Learning Collaborative (SLC), which Stella hopes with time will enable all data used in the classroom to be fed in an "understandable and digestible form" for use by teachers.

The SLC is an ambitious attempt to create the plumbing (the "shared learning infrastructure" or SLI, as it's called) required to ensure that the Common Core State Standards and their attendant assessments succeed in helping teachers create a learning environment that is personalized for every student. Massachusetts is one of nine states with districts participating in the pilot program, which will soon be opened up to other states as well.

To date, the number of elements in play that supply the necessary data that would help teachers truly differentiate learning are so numerous and complex that no one organization or company has been in a position to help schools take on this holy grail of instruction. That's where SLC comes in.

As senior program officer Henry Hipps explains, "SLC is trying to take one thin slice of the work"-- data and application integration and dashboards--"that's very broad and very complicated and not very value added, so that it makes life easier for both providers of various applications and tools and for the folks that want to buy those applications and tools and services with the aim of impacting students and teachers."

Actually, it's more than one slice--the SLC work also requires tackling the identity management and reporting problems that have vexed teachers and prevented them from getting at and using student data effectively.

It also happens to be very pricey work but with deep-pocketed supporters. Like Gartner's "Closing the Gap" research, SLC is also funded by the Bill & Melinda Gates Foundation. The project includes participation from the Council of Chief State School Officers, Alliance for Excellent Education, and Carnegie Corp. of New York. A first release is expected in December .

A shared learning infrastructure consists of a set of components: a shared service for data storage, which will be available as separate and secured spaces for each state and district to maintain data about schools, employees, and students; an application programming interface to enable data from various applications to mesh with systems being used by the teachers; a user identity directory, to tackle the multi-password hassle; and dashboards to allow users to gain integrated views of data.

Early on in the SLC effort, Hipps recalls, teachers in focus groups talked about their frustrations with "information being all over the place." But as the educators better understood what the SLC was trying to build, they began getting ambitious for the project, in particular for how it might enable them to create individual education plans (IEPs) not just for special education students, but for every student. In a world where the walls between data and applications are coming down and data can be delivered and acted on as soon as it's entered into a program, the teachers asked, "Why wouldn't we have an IEP for every kid?"

A Data Success Story
Not all districts are waiting for this national initiative to pave the way for them; some have forced the issue with their own solutions. One example is Fairfax County Public Schools in Virginia, which in 2007-2008 piloted the use of an integrated system that offered a single point of access to integrated online curriculum, resources, and formative assessment tools. Electronic Curriculum Assessment Resource Tool or eCART was made available to all schools in the county in the following year.

Joy McManus, who has taught chemistry for 23 years, 18 of them at Mount Vernon High School, was "skeptical" about eCART when the project was first introduced at her school. Up to that point in her long career, she'd had no relationship with student data that might help steer her instruction. But once she figured out how the eCART data could be used, it just made "logical sense" to her: "We're scientists. It makes sense to look at and analyze data and have data-driven decisions."

ECART consists of a curriculum repository that contains links to the Virginia state Standards of Learning (SOL) and district-approved curriculum materials. It also offers Horizon, an assessment tool developed in partnership with the county by Northrop Grumman that includes practice test questions and correlated test items that can be used to create assessments. Plus, it has tools to provide real-time assessment reports and to connect longitudinal data with current data to monitor student progress.

The early days of using eCART were difficult, McManus says. The team had to filter through an existing set of chemistry questions already in the system--each cluster of questions tied to a specific objective in the SOL--and determine which ones were worth keeping and what topics needed to be improved on. That process took a long time, and it's an activity that's repeated every year.

However, once the roster of questions is created, each team member tests her or his students for a given topic on the same day (to reduce the chance that students have shared information or that the results would somehow be affected by other factors), and the analysis of results can begin. If a large number of kids miss most questions on a given standard, a teacher may decide to review and test on that specific topic again to get that percentage of success higher.

The current system relies on the teacher to sort out which lesson, activity, or other resource will best suit a particular intervention need. "When somebody goes into the system to look for a resource, they can search by many different factors," explains eCART manager Aron Sterling. That includes filtering by standard, instructional strategy, audience, or type of activity. "Is there a pop-up a page that says, 'Look at this?' No. It relies on the teacher making the best decision."

Technology-Supported Decision-Making
Of course, since the 1970s, technology-based learning systems have been able to make those resource decisions in place of teachers, matching students to appropriate interventions, based on performance data. The use of such systems can save a tremendous amount of instructional prep time as well as lead to improvements in student performances.

Farmington Elementary, a K-5 school in Culpep er, VA, with 600 students, experimented last year with i-Ready, an adaptive diagnostic and instruction service from Curriculum Associates that focuses on, among other areas, Common Core standards in math, language arts, and reading. The school has tried a number of assessment products from different companies, but, says Principal Gail Brewer, "We were looking for more than just testing. We wanted something deeper than that." The i-Ready solution, Brewer says, places the student "where they're weak and goes over those skills with them."

Based on student performance, the program pulls up intervention activities--the kind that formerly had to be selected by the teacher. This automation allows Farmington to differentiate "as much as we want to," Brewer says, acknowledging how hard it is, without the aid of technology, for an individual teacher "in the time of the day to make sure that you have covered everyone's individual needs."

The pilot tested, among others, fifth-grade students who had failed the previous year's state summative tests. These students showed an astonishing 88 percent recovery rate in reading and about 75 percent in math after 35 to 45 minutes of daily i-Ready use. That improvement led Brewer to purchase a number of annual subscriptions to the service and she hopes to expand that with more user seats.

Beyond higher test scores, Brewer is excited that students reported to their teachers that they liked how the program allowed them to be independent with their learning and that it allows them to track their own progress, empowering them to be data driven themselves. "They know when they falter," Brewer explains. "They're cognizant enough to know, 'I need to work more on that. I didn't read the paragraph carefully.'"

Enabling students to take some responsibility for their own learning is a very powerful tool, says Gartner's Rust, and is suggestive of what could be the ultimate level of maturity in the use of data in the class: when the teachers are no longer the only ones using it. The teacher's role, he muses, would change from "owning" instruction to becoming a navigator and guiding students, who would themselves take on responsibilities for their achievements. In other words, students would use the data to take charge of their own learning plans.

A Culture of Learning and Assessment
Educational experts will say that almost any significant change in the classroom is ultimately a change in culture, which is clearly what happened with the implementation of eCART at Mount Vernon High. According to teacher McManus, using the program changed "the learning and culture of assessment" for her students.

"I can tell you numerous personal anecdotes of students who say, 'I've never been good in science.' Just mastering one standard in a test (was) a victory," she says. Now, most students end up mastering two or three standards in every test, which really spurs them to focus. "Kids don't want to do work that they don't think is necessary. And if you're making them do a whole set of test corrections when they only missed one section of the test or one topic, they see that as a waste of their time, and I agree."

The result is that she gets buy-in from her class for putting more effort into a particular standard that wasn't well understood, "which helps adjust the mental attitude of the students." Those students who do well on the standards act as "masters" or experts and help the other kids get through the work.

Although McManus knows eCART has played a role in helping the school's chemistry students do better on their summative assessments (two years ago students hit 85 percent, a "significant jump," which has since leveled off a bit), she's hesitant to point to the use of eCART as the sole reason. At the same time eCART was introduced, other instructional changes were made. For example, the course used to be "topic-driven," and now it's done in a "spiraling fashion," where topics are introduced, and the class returns to them to layer on more information.

But perhaps the biggest change came about with the introduction of common planning time at the school. As McManus explains, all five teachers in the chemistry department have an hour each day when they can collaborate. That provides the time they need to review the results of the data being generated by eCART assessments and set a plan for instruction to address the gaps. "I don't think you could find any teachers who will argue that looking at data isn't a valid thing to do," she points out. But prior to eCART, "The question would be, how long is it going to take to look at this data? And how accessible is the data?"

The school accommodates a common planning hour by scheduling no science classes for that time period anywhere in the building, which means the students are attending other classes. This arrangement does lead to overcrowding some of the other classes, but the burden is shared equally across all disciplines.

For SLC's Hipps, common planning time for looking at data is the first step in his vision of a transformed school culture. He imagines groups of teachers from the same grade or same subject sitting down and saying, "Okay , here is how the kids did. Are there global changes that we need to make in the curriculum? Things that we came up short with in particular topic areas? Do we need to strengthen these areas? And then they will actually get down into the weaknesses that may exist in (individual) student learning."

Brewer believes her school is almost there. "We work hard on bonding kids and building a learning community. Kids own a piece of the responsibility in their learning. Teachers own a piece. When you get that sort of team effort going on, they like that. Each week the children can review the reports and look at what they did well on and what they didn't do well on and check their progress. The teachers can become facilitators, cheerleaders, and still offer a teaching piece. We still come back to the core curriculum, but we always want to make sure we're closing the gap."