30 JANUARY 2015 • VOL 347 ISSUE 6221 493 SCIENCE sciencemag.org
is, or soon will be, out of the bottle,” says
Brian Mennecke, an information systems
researcher at Iowa State University in
Ames who studies privacy. “There will be
no going back.”
SIMPLY DETECTING FACES is easy for a
computer, at least compared with detecting common objects like flowers, blankets,
and lamps. Nearly all faces have the same
features—eyes, ears, nose, and mouth—in
the same relative positions. This consistency provides such an efficient computational shortcut that “we’ve been able to
detect faces in images for about 2 decades,”
LeCun says. Even the puny computers in
cheap consumer cameras have long been
able to detect and focus on faces.
But “identifying a face is a much harder
problem than detecting it,” LeCun says.
Your face uniquely identifies you. But
unlike your fingerprints, it is constantly
changing. Just smile and your face is transformed. The corners of your eyes wrinkle,
your nostrils flare, and your teeth show.
Throw your head back with laughter and
the apparent shape of your face contorts.
Even when you wear the same expression,
your hair varies from photo to photo, all
the more so after a visit to the hairdresser.
And yet most people can spot you effortlessly in a series of photos, even if they’ve
seen you in just one.
In terms of perceiving the
world around us, facial recognition may be “the single most
impressive thing that the human brain can do,” says Erik
Learned-Miller, a computer
scientist at the University of
Massachusetts, Amherst. By
contrast, computers struggle
with what researchers call the
problem of A-PIE: aging, pose,
illumination, and expression.
These sources of noise drown
out the subtle differences that
distinguish one person’s face
Thanks to an approach called
deep learning, computers are
gaining ground fast. Like all
machine learning techniques,
deep learning begins with a
set of training data—in this
case, massive data sets of la-
beled faces, ideally including
multiple photos of each per-
son. Learned-Miller helped
create one such library, called
Labeled Faces in the Wild
(LFW), which is like the ulti-
mate tabloid magazine: 13,000
photographs scraped from the
Web containing the faces of 5749 celebri-
ties, some appearing in just a few photos
and others in dozens. Because it is on-
line and free to use, LFW has become the
most popular benchmark for machine vi-
sion researchers honing facial recognition
To a computer, faces are nothing more
than collections of lighter and darker pixels. The training of a deep learning system
begins by letting the system compare faces
and discover features on its own: eyes and
noses, for instance, as well as statistical
features that make no intuitive sense to
humans. “You let the machine and data
speak,” says Yaniv Taigman, DeepFace’s
lead engineer, who’s based at Facebook’s
Menlo Park headquarters. The system
first clusters the pixels of a face into elements such as edges that define contours.
Subsequent layers of processing combine
elements into nonintuitive, statistical features that faces have in common but are
different enough to discriminate them.
This is the “deep” in deep learning: The
input for each processing layer is the output of the layer beneath. The end result of
the training is a representational model
of the human face: a statistical machine
that compares images of faces and guesses
whether they belong to the same person.
The more faces the system trains on, the
more accurate the guesses.
The DeepFace team created a buzz in the
machine vision community when they described their creation in a paper published
last March on Facebook’s website. One
benchmark for facial recognition is identifying whether faces in two photographs
from the LFW data set belong to the same
celebrity. Humans get it right about 98% of
the time. The DeepFace team reported an
accuracy of 97.35%—a full 27% better than
the rest of the field.
Some of DeepFace’s advantages are from
its clever programming. For example, it
overcomes part of the A-PIE problem by
accounting for a face’s 3D shape. If pho-
tos show people from the side, the pro-
gram uses what it can see of the faces to
reconstruct the likely face-forward visage.
This “alignment” step makes DeepFace far
more efficient, Taigman says. “We’re able
to focus most of the [system’s] capacity on
the subtle differences.”
“The method runs in a fraction of a sec-
ond on a single [computer] core,” Taigman
says. That’s efficient enough for DeepFace
to work on a smart phone. And it’s lean,
representing each face as a string of code
called a 256-bit hash. That unique repre-
sentation is as compact as this very sen-
tence. In principle, a database of the facial
identities of 1 billion people could fit on a
But DeepFace’s greatest advantage—
and the aspect of the project
that has sparked the most
rancor—is its training data.
The DeepFace paper breezily
mentions the existence of a
data set called SFC, for Social
Face Classification, a library of
4.4 million labeled faces harvested from the Facebook pages
of 4030 users. Although users
give Facebook permission to
use their personal data when
they sign up for the website,
the DeepFace research paper
makes no mention of the consent of the photos’ owners.
“JUST AS CREEPY as it sounds,”
blared the headline of an article in The Huffington Post
describing DeepFace a week
after it came out. Commenting
on The Huffington Post’s piece,
one reader wrote: “It is obvious that police and other law
enforcement authorities will
use this technology and search
through our photos without
us even knowing.” Facebook
has confirmed that it provides
law enforcement with access
Is that really you?
Just glance at these photos and it is immediately obvious that you’re
looking at the same person (computer scientist Erik Learned-Miller).
To a computer, however, almost every parameter that can be measured
varies from image to image, stymieing its ability to identify a face. A
technique called deep learning squelches noise to reveal statistical
features that these visages have in common, allowing a computer to
predict correctly that they all belong to the same individual.