Sending AI Off to School
Can machine learning really free up time dedicated to grunt work, giving your educators more time for working with students?
- By Dian Schaffhauser
of discussions about the use of artificial intelligence talk about
how AI could help educators by shrinking the amount of time they have
to spend on the trivia that pervades their work and freeing them up
to focus on the job of teaching. In the latest CoSN
IT leadership survey,
more than half of respondents (55 percent) said that AI would have a
significant or even transformational impact on teaching and learning
within the next five years, if privacy issues can be addressed to
session delivered during the CoSN2020 virtual conference,
Girard Kelly gave a preview of what the use of AI could look like by
using an example from his own employer. Kelly works as counsel and
director of the privacy program at nonprofit Common
division of Common Sense
serves as a vetting tool to help teachers choose educational software
based on learning ratings, community ratings and privacy ratings.
It's that last category that's especially difficult to nail down,
Problem with Privacy Policies
problem Common Sense faced was in trying to help teachers and
families make better decisions about the software they use, based on
the privacy policies published by the companies that produce those
programs. The privacy evaluation process is intense and has both
"qualitative and quantitative" aspects. The organization
has to evaluate whether it exists in the first place, whether the
privacy practices are transparent, whether there's advertising or
user tracking in the product, whether the company does data
collection and sells that data, and a whole bunch of other issues.
Sense uses a three-tier rating system: Blue means pass (about 20
percent of products fall into this rating); red means fail (about 10
percent); and orange means proceed with caution (about 70 percent).
existence of the privacy rating in itself has already made an impact,
according to Kelly. Of the 750 products already evaluated, about half
of the companies have updated their policies based on the ratings
they've initially been given.
big part of the job of developing each rating involves having legal
the companies. Many of the policies run "dozens of pages long"
— some 50 to 60 pages — and they're full of legalese, Kelly
noted. "We found that nobody actually reads the privacy policies
because it's really hard and it takes a really long time."
Sense wondered whether machine learning might help speed up the
Machine Learning Terms Worth Understanding
Kelly explained, the very concept of AI covers a lot of ground,
encompassing machine learning, knowledge graphs and expert systems.
In the case of ed tech, machine learning dominates by a wide margin.
What distinguishes it from the other flavors of AI is "its
ability to modify itself when exposed to more data." The more
data that's fed into a machine learning model, the more accurate its
recommendations or findings.
learning itself has subsets too, primarily broken down into how
categorization — the task to be done — takes place.
learning" references the use of prior knowledge of the output.
The model already knows what output is desired, and AI is used to
sort out or classify the input that will lead to that output. Once
the model learns the formula, it can stop learning and do its job.
According to Kelly, supervised learning crops up in identity fraud
detection, image identification and weather forecasting, among other
applications. When you identify the squares in a reCAPTCHA that show
the crosswalk, for instance, you're helping the model learn how to
identify certain imagery in the data.
"unsupervised learning," you have the input data but you
don't know what the output should look like. The goal for the model
is to learn more about the data and reveal patterns to you. This
proves useful in recommendation systems, targeted marketing and big
data visualization, as examples.
a gray area between those two approaches, which is called
"semi-supervised" machine learning. You have a lot of data
for input, and some of the data is identified. Human experts help the
model evolve by undertaking identification until it can learn to a
sufficient level of competency.
third category of machine learning is called "reinforcement
learning." This is the process by which a learning algorithm
(the "agent") learns how to learn, based on whether it gets
a positive or negative response: It loses the chess game or it wins.
There's no training data plugged in; the training happens as the
process runs. According to the experts, this most closely resembles
how humans learn too. (Think: small child and hot stove.)
Reinforcement learning shows up in gaming and products doing
real-time decision-making, performing robot navigation or undertaking
Learning to the Rescue?
are two challenges to the effective use of machine learning for the
kind of work Common Sense had in mind:
even humans "go back and forth" on various issues of
can't expect machines to help us," noted Kelly.
privacy policies don't follow a consistent order. Companies' privacy
practices aren't laid out in a standard format.
organization turned to the use of natural language processing and
"transformers" to do an initial assessment of privacy
policies. Transformers, pre-trained models, parse sentences and
paragraphs to highlight what would be relevant for human analysis.
For example, if the evaluators want to see specific language in a
category of the scoring rubric, possibly machine learning could help
them make that first pass. Then the humans could focus on the "hard
parts," not the straightforward contents.
the results haven't always met the bar. As Kelly asserted, a lot of
the machine learning technology currently available relies on
keyword-based pattern matching, which isn't entirely accurate and
generates "too many false negatives."
data when it actually does:
do not sell your data, except if you give us your consent by creating
an account with the service."
transformer might pick up the relevant part ("we do not sell
your data") and give a passing score, thereby ignoring the
caveat that a human evaluator would immediately recognize ("except
if you give consent...").
it's possible that the policy doesn't say whether data is sold, even
though the company does sell it:
only share your data with third parties for legitimate interest
that case, suggested Kelly, what's a "legitimate interest"?
For companies, that's "to make money." While the human
evaluator would pick up on that, it's "something where automated
keyword-based systems won't catch it."
for the Future
said the systems that Common Sense has tried out have had some
success, and the process of "growing the training data"
continues. There are a few cases where the machine learning is good
enough to answer some questions about the privacy policies under
effective date? Because that's the most structured (the policy
usually references "version" or "effective date"),
it's easy to detect.
of the promise offered by AI, the organization intends to continue
its work with machine learning. That will probably involve developing
a "hybrid approach" that could reduce the amount of time
humans spend on the evaluation by automating some aspects of the
there still needs to be the human touch," Kelly emphasized.
"There's got to be some human intervention," to continue
training the AI model."
offered a list of questions for school leaders to ask companies that
tout their programs' AI capabilities:
does the software really do, and where's the evidence that can
support that claim?
general or specific does the AI get? For example, can it "measure
general student outcomes or just a slice of them"?
AI can do the work better than humans, which humans are we talking
about, and how much better? What's the cost-benefit equation?
are the privacy or ethical risks to students involved in the use of
populations don't exist in the training data, which could make for
less effectiveness or downright bias?
there a demo that lets the school try the software using its own
the software be trained by the users? Can it work with other data
training sets or does the company have control over those aspects?