As part of the 2023 OpenDP Community Meeting, a breakout session centered around privacy attacks and auditing attracted leading experts in the room and online from industry, academia, non-profits, and government entities alike to share their perspectives on this new topic. It was exciting to see broad recognition that attacks are and will continue to be an important part of privacy research, and in complementing the adoption of differential privacy. The animating question for the breakout session was how OpenDP can develop new functionalities to make it easier to incorporate attacks into the process of adoption, developing, and testing deployments of differential privacy.
Yves-Alexandre de Montjoye (Imperial College London) and Jonathan Ullman (Northeastern University) first probed the audience on the role of privacy attacks in supporting the development of differential privacy as a technology. The discussion kicked off with a few survey questions to gauge the audience’s views. One pattern that emerged is that there are several broad categories of attacks with very different objectives:
-
Attacks against alternative methodologies (e.g. anonymization) that motivate the adoption of differential privacy.
-
Attacks that are used to evaluate and communicate the qualitative and quantitative privacy risks of differential privacy when used on real-world datasets
-
Attacks that are used to identify problems with the implementation of differentially private algorithms, such as coding errors, side channels, or adverse interactions between components in the pipeline.
The discussion then shifted to how researchers can help develop a systematic toolkit for attacks to make them easier for practitioners to use. The audience identified several major questions for study:
-
Privacy attacks differ on so many different axes---what the data looks like, what we are trying to release, what kind of algorithm is being used for the release, what kind of attacker we are considering, and what sort of information the attacker is extracting, and more. How can we systematize the space of attacks into something that would help a practitioner know what attacks to try?
-
Privacy is inherently quantitative, and systems are not necessarily “broken” or “unbroken,” which makes it subtle to interpret attacks and leads to confusion over whether an attack is meaningful or not. Can we develop a clean set of standards for what constitutes a successful or unsuccessful attack?
-
What makes the presentation of an attack compelling to different audiences? How can we design tools to support communicating the results of attacks to policy makers, software developers, privacy experts, and others?
-
Are there specific attacks that are used repeatedly and can be turned into general purpose tools? What is the right interface for these attacks?
It was exciting to see such broad recognition that privacy attacks are not only important for encouraging the adoption of privacy technologies but continue to be important in their development and deployment. One analogy that came up repeatedly is to attacks in cryptography, where attacks on cryptographic systems have always been used in all phases of the development of new cryptographic technologies. Although privacy is less mature, the participants were optimistic that privacy attacks could grow to play a similar role.
Interested in joining the conversation and learning more?
We will start scheduling monthly syncs with the OpenDP Community and all are welcome to join - we would love to include as many perspectives and experiences as possible! Complete this form to register your interest by February 23rd, 2024 to be added to the mailing list and included for the first meeting to be scheduled in March. You can also join the dedicated slack channel to introduce yourself and chat with other members of the working group.
