2020 is a watershed year for machine learning. It has seen the true arrival of commodized machine learning, where deep learning models and algorithms are readily available to Internet users. GPUs are cheaper and more readily available than ever, and new training methods like transfer learning have made it possible to train powerful deep learning models using smaller sets of data.
But accessible machine learning also has its downsides. A recent New York Times article by Kashmir Hill profiled clearview.ai, an unregulated facial recognition service that has downloaded over 3 billion photos of people from the Internet and social media and used them to build facial recognition models for millions of citizens without their knowledge or permission. Clearview.ai demonstrates just how easy it is to build invasive tools for monitoring and tracking using deep learning.
So how do we protect ourselves against unauthorized third parties building facial recognition models that recognize us wherever we may go? Regulations can and will help restrict the use of machine learning by public companies but will have negligible impact on private organizations, individuals, or even other nation states with similar goals.
The SAND Lab at University of Chicago has developed Fawkes1, an algorithm and software tool (running locally on your computer) that gives individuals the ability to limit how their own images can be used to track them. At a high level, Fawkes takes your personal images and makes tiny, pixel-level changes that are invisible to the human eye, in a process we call image cloaking. You can then use these "cloaked" photos as you normally would, sharing them on social media, sending them to friends, printing them or displaying them on digital devices, the same way you would any other photo. The difference, however, is that if and when someone tries to use these photos to build a facial recognition model, "cloaked" images will teach the model an highly distorted version of what makes you look like you. The cloak effect is not easily detectable by humans or machines and will not cause errors in model training. However, when someone tries to identify you by presenting an unaltered, "uncloaked" image of you (e.g. a photo taken in public) to the model, the model will fail to recognize you.
Fawkes has been tested extensively and proven effective in a variety of environments and is 100% effective against state-of-the-art facial recognition models (Microsoft Azure Face API, Amazon Rekognition, and Face++). We are in the process of adding more material here to explain how and why Fawkes works. For now, please see the link below to our technical paper, which will be presented at the upcoming USENIX Security Symposium, to be held on August 12 to 14.
The Fawkes project is led by two PhD students at SAND Lab, Emily Wenger and Shawn Shan, with important contributions from Jiayun Zhang (SAND Lab visitor and current PhD student at UC San Diego) and Huiying Li, also a SAND Lab PhD student. The faculty advisors are SAND Lab co-directors and Neubauer Professors Ben Zhao and Heather Zheng.
1The Guy Fawkes mask, a la V for Vendetta.
In addition to the photos of the team cloaked above, here are a couple more examples of cloaked images and their originals. Can you tell which is the original? (Cloaked image of the Queen courtesy of TheVerge).
Publication & Presentation
Fawkes: Protecting Personal Privacy against Unauthorized
Deep Learning Models.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y. Zhao.
In Proceedings of USENIX Security Symposium 2020. ( Download PDF here )
Downloads and Source Code - Version 0.3 (July 2020)
- NEW! Fawkes v0.3
v0.3 has an updated target selection algorithm that significantly reduces the likelihood of perturbation artifacts after cloaking. This version was released after the NYTimes interview, and eliminates most of the distortions described in the NYTimes article. If you downloaded and experimented with v0.2 or v0.1, we strongly suggest you try v0.3.
- Download the Fawkes Software:
Fawkes.dmg for Mac (v0.3)
DMG file with installer app
Compatibility: MacOS 10.13, 10.14, 10.15 Fawkes.exe for Windows (v0.3)
Compatibility: Windows 10
Setup Instructions: For MacOS, download the .dmg file and double click to install. If your Mac refuses to open because the APP is from an unidentified developer, please go to System Preference>Security & Privacy>General and click Open Anyway.
- Download the Fawkes Executable Binary:
Fawkes binary offers additional options on selecting different parameters. Check here for more information on how to select the best parameters for your use case.
Download Mac Binary (v0.3)
Download Windows Binary (v0.3)
Download Linux Binary (v0.3)
- Fawkes Source Code on Github, for development.
If you have any issues running Fawkes, please feel free to ask us by email or raising an issue in our Github repo. Check back often for new releases.
Media and Press Coverage
- The Verge, James Vincent: Cloak your photos with this AI privacy tool to fool facial recognition
- New York Times, Kashmir Hill: This Tool Could Protect Your Photos From Facial Recognition
- UChicago CS, UChicago CS Researchers Create New Protection Against Facial Recognition
With more commentary and responses to Clearview.ai CEO's comments in NY Times article.
- Tech by Vice, Researchers Want to Protect Your Selfies From Facial Recognition
- TechSpot, Adrian Potoroaca: University of Chicago researchers are building a tool to protect your pictures from facial recognition systems
- The Register (UK), Sick of AI engines scraping your pics for facial recognition? Here's a way to Fawkes them right up
- Gizmodo, Shoshana Wodinsky: This Algorithm Might Make Facial Recognition Obsolete
- South China Morning Post (HongKong), Xinmei Shen: Anti-facial recognition tool Fawkes changes your photos just enough to stump Microsoft and Amazon
- ZDNet, Fawkes protects your identity from facial recognition systems, pixel by pixel
- Schneier on Security, Fawkes: Digital Image Cloaking
- MIT Technology Review China, article
- OneZero on Medium, This Filter Makes Your Photos Invisible to Facial Recognition
- Built in Chicago, Nona Tepper: UChicago Researchers Made a Photo-Editing Tool That Hides Your Identity From Facial Recognition Algorithms
- El universal (Mexico), Esta herramienta protege tus fotos del reconocimiento facial
- Radio Canada, This tool can keep your photos safe from facial recognition algorithms
- Sina Tech (China), Put this "cloak" on your photos to foil state of the art facial recognition systems
- Instalki.pl (Poland), Ten algorytm pozwoli Ci uchronic sie przed systemami rozpoznawania twarzy
- Manual do Usuario (Brazil), Rodrigo Ghedin: O algoritmo anti-reconhecimento facial
- News18 (India), Shouvik Das: Privacy vs Facial Recognition: Fawkes Aims to Help Protect Your Public Photos From Misuse
- WION News (India), Engineers develop tool called 'Fawkes' to protect online photos from facial recognition
- IGuru (Greece), Fawkes: protect your photos from face recognition
- Golem.de (Germany), Fawkes soll vor Gesichtserkennung schutzen
- Heise Online (Germany), Verzerrungs-Algorithmus Fawkes will Gesichtserkennung verhindern
- Developpez.com (France),
Des chercheurs mettent au point Fawkes, un <
Frequently Asked Questions
- How effective is Fawkes against 3rd party facial recognition models like ClearView.ai?
We have extensive experiments and results in the technical paper (linked above). The short version is that we provide strong protection against unauthorized models. Our tests against state of the art facial recognition models from Microsoft Azure, Amazon Rekognition, and Face++ are at or near 100%. Protection level will vary depending on your willingness to tolerate small tweaks to your photos. Please do remember that this is a research effort first and foremost, and while we are trying hard to produce something useful for privacy-aware Internet users at large, there are likely issues in configuration, usability in the tool itself, and it may not work against all models for all images.
- How could this possibly work against DNNs? Aren't they supposed to be smart?
This is a popular reaction to Fawkes, and quite reasonable. We hear often in popular press how amazingly powerful DNNs are and the impressive things they can do with large datasets, often detecting patterns where human cannot. Yet the achilles heel for DNNs has been this phenomenon called adversarial examples, small tweaks in inputs that can produce massive differences in how DNNs classify the input. These adversarial examples have been recognized since 2014 (here's one of the first papers on the topic), and numerous defenses have been proposed over the years since (and some of them are from our lab). Turns out they are extremely difficult to remove, and in a way are a fundamental consequence of the imperfect training of DNNs. There have been multiple PhD dissertations written already on the subject, but suffice it to say, this is a fundamentally difficult thing to remove, and many in the research area accept it now as a necessary evil for DNNs.
The underlying techniques used by Fawkes draw directly from the same properties that give rise to adversarial examples. Is it possible that DNNs evolve significantly to eliminate this property? It's certainly possible, but we expect that will require a significant change in how DNNs are architected and built. Until then, Fawkes works precisely because of fundamental weaknesses in how DNNs are designed today.
- Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to
the image to destroy image cloaks?
As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.
- How is Fawkes different from things like the invisibility cloak projects at UMaryland, led by Tom Goldstein, and
other similar efforts?
Fawkes works quite differently from these prior efforts, and we believe it is the first practical tool that the average Internet user can make use of. Prior projects like the invisibility cloak project involve users wearing a specially printed patterned sweater, which then prevents the wearer from being recognized by person-detection models. In other cases, the user is asked to wear a printed placard, or a special patterned hat. One fundamental difference is that these approaches can only protect a user when the user is wearing the sweater/hat/placard. Even if users were comfortable wearing these unusual objects in their daily lives, these mechanisms are model-specific, that is, they are specially encoded to prevent detection against a single specific model (in most cases, it is the YOLO model). Someone trying to track you can either use a different model (there are many), or just target users in settings where they can't wear these conspicuous accessories. In contrast, Fawkes is different because it protects users by targeting the model itself. Once you disrupt the model that's trying to track you, the protection is always on no matter where you go or what you wear, and even extends to attempts to identify you from static photos of you taken, shared or sent digitally.
- How can Fawkes be useful when there are so many uncloaked, original images of me on social
media that I can't take down?
Fawkes works by training the unauthorized model to learn about a cluster of your cloaked images in its "feature space." If you, like many of us, already have a significant set of public images online, then a model like Clearview.AI has likely already downloaded those images, and used them to learn "what you look like" as a cluster in its feature space. However, these models are always adding more training data in order to improve their accuracy and keep up with changes in your looks over time. The more cloaked images you "release," the larger the cluster of "cloaked features" will be learned by the model. At some point, when your cloaked cluster of images grows bigger than the cluster of uncloaked images, the tracker's model will switch its definition of you to the new cloaked cluster and abandon the original images as outliers.
- Is Fawkes specifically designed as a response to Clearview.ai?
It might surprise some to learn that we started the Fawkes project a while before the New York Times article that profiled Clearview.ai in February 2020. Our original goal was to serve as a preventative measure for Internet users to inoculate themselves against the possibility of some third-party, unauthorized model. Imagine our surprise when we learned 3 months into our project that such companies already existed, and had already built up a powerful model trained from massive troves of online photos. It is our belief that Clearview.ai is likely only the (rather large) tip of the iceberg. Fawkes is designed to significantly raise the costs of building and maintaining accurate models for large-scale facial recognition. If we can reduce the accuracy of these models to make them untrustable, or force the model's owners to pay significant per-person costs to maintain accuracy, then we would have largely succeeded. For example, someone carefully examining a large set of photos of a single user might be able to detect that some of them are cloaked. However, that same person is quite possibly capable of identifying the target person in equal or less time using traditional means (without the facial recognition model).
- Can Fawkes be used to impersonate someone else?
The goal of Fawkes is to avoid identification by someone with access to an unauthorized facial recognition model. While it is possible for Fawkes to make you "look" like someone else (e.g. "person X") in the eyes of a recognition model, we would not consider it an impersonation attack, since "person X" is highly likely to want to avoid identification by the model themselves. If you cloaked an image of yourself before giving it as training data to a legitimate model, the model trainer can simply detect the cloak by asking you for a real-time image, and testing it against your cloaked images in the feature space. The key to detecting cloaking is the "ground truth" image of you that a legitmate model can obtain, and unauthorized models cannot.
- How can I distinguish photos that have been cloaked from those that have not?
A big part of the goal of Fawkes is to make cloaking as subtle and undetectable as possible and minimize impact on your photos. Thus it is intentionally difficult to tell cloaked images from the originals. We are looking into adding small markers into the cloak as a way to help users identify cloaked photos. More information to come.
- How do I get Fawkes and use it to protect my photos?
We are working hard to produce user-friendly versions of Fawkes for use on Mac and Windows platforms. We have some initial binaries for the major platforms (see above). Fawkes is also available as source code, and you can compile it on your own computer. Feel free to report bugs and issues on github, but please bear with us if you have issues with the usability of these binaries. Note that we do not have any plans to release any Fawkes mobile apps, because it requires significant computational power that would be challenging for the most powerful mobile devices.
- We are adding more Q&A soon. If you don't see your question here, please email us and we will add it to the page soon.