Catalina Cangea , Laurynas Karazija , Edgar Liberis , Petar Veličković
Team FaceJack working hard at Hack Cambridge ’17.
Machine learning with deep neural networks (commonly dubbed "deep learning") has taken the world by storm, smashing record after record in a wide variety of difficult tasks from different fields, including computer vision, speech recognition and natural language processing. One such computer vision task is facial recognition—identifying a person using a picture of their face. Many social networks and Computer Vision API services in this area are built using neural networks.
Neural networks may be used in security-sensitive applications, such as to authenticate a person based on a picture of their face in a smart home system. Despite the seemingly superb accuracy of such models, it is fairly easy to construct inputs which trick the network into authenticating a stranger. These are known as “adversarial examples”.
We are a team of students from University of Cambridge who have built an application to demonstrate this concept at the Hack Cambridge hackathon this January . We aim to highlight:
Adversarial inputs are inherent to neural networks. Broadly speaking, a network is a sequence of transformations (layers) parametrised by weights , which are applied to the input to obtain a prediction. These transformations are designed to be differentiable, so that a network can be efficiently trained by:
In a conventional training setting, we consider inputs and outputs of the network to be fixed and modify the weights to minimise the loss on these inputs. Here, however, we consider a fixed choice of weights and modify the input to produce a desirable output:
Figure 1. On the left: adversarial setting, where network parameters (weights) are fixed and inputs are adjustable. On the right: conventional training setting, where inputs and outputs are fixed, and parameters are adjustable.
To obtain an adversarial example, we set the network’s optimisation target (the loss function) to maximise the probability of the desired output class, e.g. that a face does belong to an administrator. With this set-up, we run the training process to “optimise” the input image for misclassification. As a result, feeding this modified input into the original face recognition network causes an avalanche of errors across the layers, which produces a wrong output.
Deep neural networks are particularly vulnerable to this for three main reasons:
· Computing an adversarial example usually only requires a crude approximation of the derivative (gradient) of the desired output with respect to the input image. Often, only the sign of the gradient for each input pixel is sufficient.
· Adversarial examples are often imperceptibly similar to the original input. In fact, Szegedy et al. (2013) demonstrated there is an entire space of adversarial inputs surrounding any correctly classified image.
· Even worse—what's adversarial for one network architecture will very often be adversarial for a completely different network as well —as they are often trained on the same datasets!
This shows that using neural networks in secure contexts requires particular care, because adversarial inputs can be used as an exploit.
During the hackathon we built an app (called FaceJack ) to demonstrate this weakness of neural networks. The system consists of multiple components:
The visualisation system runs on a Flask web-server which receives the data from the classification process and displays it as a web page.
Using the Azure services for deployment was very quick and convenient. We created the virtual machine using the azure-cli Azure command line interface and, after installing CUDA and cuDNN, we deployed our software and went live within minutes.
FaceJack was a success during the hackathon and managed to impress members of the audience with its novelty—many participants were not familiar with adversarial examples. It has achieved its objective of highlighting the security issue in a clear and concise fashion. As one of the judges jokingly put it: “Now I’ll be afraid to set-up face unlock on my phone”.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.