The Math of Khan

Khan Academy has inspired both unconditional love and virulent criticism. But the controversy around the videos has sparked something truly valuable: a national conversation about math instruction and the role of technology, data, and teachers in helping students learn.

• By Dian Schaffhauser
• 02/06/13

It's like a dream come true for educators: free resources--all you and your students could consume in a lifetime of classes--available wherever a web browser can operate and with nary an iota of advertising. What's not to like? Apparently, plenty. Khan Academy has spawned critics like flies at a feast. Among the more mild opinions recently expressed: "Someone needs to take khan academy [sic] and push it down the well." What is it about this organization, whose mission is "to provide a free, world-class education for anyone anywhere," that has provoked such vitriol?

Part of it is that when your videos have been watched some 217 million times, a certain category of YouTube viewer will simply be intent on taking you down. Those who are better informed take exception to the pedagogy--and even the accuracy of the instruction. After all, Sal Khan, who stars in many of the 3,647 Khan videos, isn't a full-fledged teacher. Others say the output of Khan Academy simply updates the repetitive practices of "drill and kill." But perhaps the most serious concern is this: that so many important people--Arne Duncan, Bill Gates, the producers at 60 Minutes--view the work of the Khan Academy as so remarkable that they may convince us to let it lead the way in transforming education. That, the naysayers shudder, is chilling, indeed.

Revolutionary Setbacks
Peter Kelman is a quasi-historian of technology in teaching. Although he's retired now, he ran the master's program in teaching at Wesleyan University, trained teachers for certification at Dartmouth with a specialty in math and sciences, and was a publisher of math and science textbooks as well as educational software. He's a person who has seen the term "revolutionary" too loosely applied to multiple generations of educational breakthroughs, as it was by USA Today in May 2012, when it headlined an article "Sal Khan's 'Academy' sparks a tech revolution in education."

The problem, as Kelman sees it, is that "breakthroughs" can actually hold progress back. "As each technology emerges, it starts out with what I call 'amateur educators,' and it starts to evolve, and then something else comes along and we take 10 giant steps back. HyperCard took us back to BASIC. When we got to HTML, that took us 10 steps back to where we were with HyperCard. Then Java took us back, just as people were beginning to figure out really interesting things to do with HTML.

"Now that we have streaming video, which is the technological basis for the Khan stuff, that takes us back," Kelman observes. It's simply a return to what an ordinary teacher can do, he suggests. Except with Khan, "You can't ask questions. The teacher can't see the look on your face."

Adam Bellow, a former director of educational technology at the College Board, has been quoted as saying, "I don't want to pick on Khan Academy, but straight lectures don't get to the misconceptions; they just explain broad concepts that students don't grasp very well through video without practice."

Not Problem-Solving
Two math education professors, John Golden and David Coffey at Grand Valley State University in Michigan, have a similar critique: Khan videos offer the same limitations as textbooks. Except the Khan critique they produced--in video form--was quickly picked up by the education media, which tweeted and blogged and spread the link, starting what Coffey called "a serious conversation on Khan Academy's role in education reform." It also sparked a mini-industry of other satirical takeoffs on Khan videos, as well as videos by defenders of Khan, and even a sequel by Golden and Coffey titled "Meta Mystery Teacher Theatre 2000," in which they review their own earlier effort.

As Golden explains, the problems displayed by Khan are "asking students to repeat what they've been told, versus problem-solving." What's the difference? "You only learn to problem-solve when you're working on something you don't already know how to do. If you've already been told the techniques for doing something, you might fine-tune how to execute it, you can practice it, but that's not problem-solving."

The understanding of math, Golden says, can be "instrumental," where the learner knows the rules of procedures; or "relational," where the learner understands some of the reasoning behind how to solve the problem and then connects it to previous material learned and "forward material." The videos by Khan "typically use very instrumental instruction," he notes. "I don't know whether that means that's how he understands the material or if that's just what he thinks is effective for students."

The approach to instruction that Golden prefers is "think aloud," which he explains as "kind of opening up your head to share your thinking with your students." Instead of telling students what to do, the teacher shares how he or she thinks about a problem and the various connections and patterns that come up. "As you demonstrate that, you ask [the students] to be active observers: What do they see you do? What do they hear you say? What things do you write down? That makes them intelligent consumers of what you're sharing. Instead of looking for a list of instructions, they might learn how to problem-solve."

Consider Caitlin Grubb. A former student of Golden's, she decided to create her own math videos rather than using Khan's to encourage greater student engagement. Grubb recently finished her student teaching at Creekside Middle School in Zeeland, MI. At one point she flipped the classroom by creating an iBook for the students to download to their iPads. The digital content had a page for each lesson with learning goals, guiding questions for the summary they were required to write, and a 10- to 12-minute video Grubb created. She also created mini quizzes so students could assess their own understanding before class and come ready to dive deeper into the topic.

"After implementing this method of teaching, I found immense improvement in the engagement of my students," Grubb says. "It allowed me to conference with small groups on a daily basis in order to better assess their understanding, rather than lecturing at the front of the room."

But surely the Khan videos cover much of the same ground, so why not just use them? For several reasons, Grubb explains. "I feel as though the students respond better to watching videos from the teacher that they are familiar with," she says, adding that she was able to make the videos more specific to what she wanted students to do at home. That included reminding them to pause the video and try a problem or two. Finally, the homemade videos enabled her to "bring misconceptions to light that I would like to talk about more in class."

Student Ownership
The kind of initiative shown by Grubb would probably be applauded by Khan (although not likely imitated by most teachers who lack the time or the training to create such sophisticated learning materials). Like Grubb, Khan wants to put students at the center of their own learning process. In his new book, The One World Schoolhouse: Education Reimagined, he writes about his first year at MIT, where he and his buddy Shantanu Sinha (now president and COO at Khan Academy) became "class skippers." Guided by upperclassmen, Khan and his roommate figured out that they could take "something close to double course loads" by avoiding seat time and pursuing "whatever actually helped us learn." That may be one of the reasons why Khan found the time to acquire three degrees from MIT; Sinha earned four.

But it may also be one of the reasons why Khan is so intent on helping students discover their own motivations. "We want them to take ownership of their learning," he says. "A lot of schools are focused on almost spoon-feeding students so that they can get to college. And then they pat themselves on their back: 'Look! All of our students went to college,' or 'We have a really high graduation rate,' or whatever it might be. But then the students go to college and they struggle because college is very self-directed." A big part of that struggle, he says, is that they didn't have ownership of their own learning.

At Khan Academy, that ownership starts when user accounts are set up, not by schools or districts, but by the students. The academy sends out a confirmation e-mail, the user logs in, and he or she is presented with a "Knowledge Map" from which to choose a topic. A set of problems for the topic appear, one after the other. When students get stuck, they can choose to watch a related video or start showing hints, either of which lays out how to work through a problem step by step.

At each decision point, the system is collecting data: how many problems are solved, how many hints are taken, how long the student takes to solve the problem, what videos are watched, how much time is spent on each video. Students can add "coaches," such as the teacher and other students, who can view the same data stream.

It's that data stream that Khan would actually like us to think about when we hear his name, not the videos. And that's where the investment needs to be, he says. "It seems, anecdotally, that people find value in [the videos]; but if we can use that as a way to bring them and engage in this deeper experience and then leverage the data that we can collect to make anything we create better, that'd be a win."

It's the Data
Kami Thordarson understands the power of the data behind the videos. Los Altos School District, where she works, has been experimenting with Khan Academy resources for a couple of years. Thordarson was chosen as one of the test pilots in those efforts because, she explains, "I had been somewhat of a risk taker and I love technology; I love bringing new, innovative things into the classroom."

Recently, Thordarson became the California district's innovative strategies coach, a new position where she spends her time helping teachers come up with new ways to deliver instruction. But she has her teaching chops. She'd spent four years at Los Altos teaching fifth and sixth grades; prior to that she'd taught in Colorado for a decade.

The innovation of Khan, she insists, isn't in the videos. "To be honest, that is such a small part of how we use Khan Academy and where our focus is. That it's not really our main concern. The videos are there, and they're an awesome resource. But that isn't what drives us to use the product."

The value of Khan lies with its lesser-known components: open-ended and interactive math exercises and the data those produce. "Khan Academy for us is a tool that helps us drive curriculum decisions. It generates data unlike any other tool that we've got. I can get immediate feedback on how kids are performing on certain skills that I can't get from other assessments. It's real time."

The basic idea isn't original: Goals are set, exercises are worked through to prove proficiency, and progress is monitored. The data is generated automatically, capturing details at the student and classroom level on how students complete the exercises, what activities they have spent time on, and what areas are causing the biggest struggles.

Students have access to the same performance data as the teacher, which has led to discussions in Thordarson's classes about what it means to set goals and how to interpret the data. For example, a focus graph showing how a student has spent his or her time is "a real eye-opener" for the students. "'You're having trouble with fractions, but you spent 80 percent of your time, according to your focus graph, doing this other thing. Maybe we could make a connection there. If we had put a little more time on fractions, that might look different.' We don't give them that kind of feedback very often."

Because the data is generated in real time, Thordarson can walk around the classroom with an iPad to get a "read" on exactly what the kids are doing on their Google Chromebooks. "I don't have to wait and grade a paper. I don't have to gather data off another system. It's just all right there."

Generating detailed data is clearly something that standard video materials don't do, so saying that Khan is just video instruction is hardly fair or accurate. But the Khan data is not as sophisticated as what would be generated by true diagnostic software, which can analyze students' underlying skill deficiencies and branch them to appropriately relearning activities before moving them on in the skill sequence.

Unlike expensive individualized learning systems, though, everything Khan Academy offers is free and will always be free, according to the organization. And the use of their materials does not require an infrastructure investment, Thordarson points out. "I know a lot of people say, 'We can't do Khan Academy because we're not 1-to-1.' You don't need to be," she says. "When I started Khan Academy, we had one laptop cart that we shared among our grade level. It can be a great tool that works as long as you can get access at some point."

A Blended Model
What the California district has found is that Khan Academy fits nicely with the blended model of instruction it's experimenting with. "It's not like we're putting kids in the classroom with a computer and then walking out the door," says Thordarson. The teacher remains at the heart of the classroom as the designer of curriculum. "You have all of these resources--Khan Academy, your textbook, projects, different tools that you can use at your disposal--and based on the needs of your students, you are picking and choosing what works for you in that moment."

Problems arise, she points out, when the instructor doesn't keep the blended model in mind--when videos substitute for instruction and students are forced to work through never-ending Khan exercises as some kind of timed class requirement. In other words, even something as well-intentioned as Khan Academy can be abused by unprepared teachers.

Also, Khan has yet to prove its value with solid improvements in student test scores. But that's antithetical to the Khan way, in which students are free to learn anytime, anywhere. As Khan states in his book, tests don't always capture the full picture. "We were trying to enable learning in a different and, we believed, more organic way, a way aimed at conceptual understanding rather than test prep. Because we encouraged students to progress at their own pace, we had some very advanced fifth graders already working on algebra and even trigonometry. But this impressive advancement would go unrecognized on the CSTs, which only tested proficiency in the usual fifth-grade material."

The organization is still new enough that performance results from end-of-year assessments are rare. The only solid data released by the organization to date shows definite improvements in seventh-grade math scores at Los Altos--but because other teaching changes were introduced simultaneously, the district couldn't give the Khan resources sole credit. Other pilots are just beginning to deliver testing data, Khan reports.

Indeed, Khan Academy's ultimate critic may be Khan himself. Even as parts of the Khan playbook have been deployed by so many educators, for good or bad, Khan says the academy is really just getting started. And it can't move fast enough to fix all of the weaknesses it identifies within its own offerings, he adds. "There are some interesting things that we've done that seem really promising, at least in my opinion. We're getting traction where maybe people haven't seen traction before. We think we're at a very early, early, stage. Boy, if I could tell you the stuff I want to have done by tomorrow, it'd make your head spin."