The OpenDP team is proud to announce the release of OpenDP Library v0.7!
The OpenDP Library is a modular collection of algorithms for building privacy-preserving applications, with an extensible approach to tracking privacy, and a vetted implementation. It is available as binaries for Python on PyPI, for Rust on crates.io, or in source form on GitHub.
This release focuses on architectural additions, enabling some exciting new capabilities in OpenDP, and laying the groundwork for future enhancements.
Interactive Measurements
Interactive measurements are a form of private algorithm where one can ask a sequence of questions interactively. (Contrast this with simple measurements, which are one-shot randomized functions where the output is a single answer). This interactive nature provides the option to choose questions adaptively, allowing data scientists more flexibility when performing an exploratory analysis. Interactive measurements also give us a way to represent algorithms that don’t fit into the model of a one-shot function, such as the Sparse Vector Technique.
The theory behind interactive measurements is laid out in the OpenDP Programming Framework, the conceptual underpinning of our work. We’re excited to bring a robust implementation of interactive measurements to the OpenDP Library.
In OpenDP Library v0.7, we’ve established the core elements for interactive measurements, by creating an extensible representation of queryables (the kind of object returned by an interactive measurement). We also needed to create facilities for queryables to message one another internal to the system, and a way for queryables to transparently “wrap” other queryables (necessary for situations when a queryable returns another queryable, and wants to mediate access to that new child queryable). Building all this required a bunch of clever engineering to get the needed dynamism from the Rust language, while still benefiting from its strict type and memory constraints. It took many iterations to get things right, but we’re very proud of the result!
As for user-facing functionality, the initial release of interactive measurements in OpenDP Library v0.7 starts with a constructor for sequential composition. We’ll expand these with more interactive measurements in the future; you can watch the milestone on GitHub to see progress.
User-defined Functions
Another new architectural element in OpenDP Library v0.7 is a facility for user-defined functions. (This was actually quietly introduced in OpenDP Library v0.6, but we’ve been waiting to accumulate some experience before spreading the news.) User-defined functions allow you to construct a transformation or measurement where OpenDP calls back into your own code for the actual implementation. This gives you the flexibility to experiment with new mechanisms in dynamic Python, rather than having to compile your algorithm into the Rust portion of OpenDP. It also allows you to bridge to implementations of DP algorithms in libraries outside of OpenDP, expanding the range of algorithms available.
Because user-defined functions don’t undergo the same vetting as built-in library code, using them requires enabling the "honest-but-curious" feature flag. This flag signals to the system that you’re on your honor when it comes to the claimed privacy properties of your user-defined function. However, assuming a correct implementation on your part, user-defined functions will operate the same as built-in functions, benefitting from OpenDP’s ability to chain and/or compose elements and automatically determine the total privacy loss!
Modeling of Domains, Metrics, Measures
The third core change in OpenDP Library v0.7 is the way that we model domains, metrics, and measures. Before now, we’ve made heavy use of the Rust type system and generic type arguments to model the parameters of measurements and transformations. This was nice in that it allowed some constraints to be enforced at compile time, and Rust’s automatic type inference often made it transparent to user code. But there was still some runtime validation, for instance in the size parameter of SizedDomain. It also turned out to be problematic in Python, which lacks the strong typing and inference of Rust. We worked around this by creating multiple constructors for the different variations of types, but this was becoming a limitation.
Moving forward, we’ve switched to capturing more information in the runtime parameters of domains, modeled by a new, more expressive set of Domain types that can be used in multiple contexts. Hand in hand with this, we’re adopting a new convention of all constructors having explicit arguments for input domains and metrics/measures.
This may seem like a subtle change, but it has some important implications. First, there’s the immediate benefit of making the Python interface more expressive, allowing you to interact with domains/metrics/measures directly. It also allows us to explicitly validate the pairing of a specific input domain with an input metric/measure, which is a constraint that could be omitted in the previous implementation. Most importantly, it enables the creation of a new high-level API for OpenDP Library that will leverage this and partial construction. This high-level API is intended to allow a much more fluid and natural style of writing OpenDP applications in Python. (You’ll hear more about this in the future, but please track the milestone to follow along!)
We've built the infrastructire for explicit domain and metric/measure arguments in our FFI layer, and are in the process of converting constructors to the new convention. A few have been converted in OpenDP Library v0.7 and we'll be converting the rest in future releases. Progress on this can be tracked in another milestone.
(One side effect of this change you’re likely to encounter in user code is that AllDomain has been deprecated and replaced by AtomDomain; existing Python scripts will get a warning with a suggested conversion.)
Other Goodies
Beyond the architectural work, we’ve made the usual quality-of-life improvements and bug fixes in OpenDP Library 0.7. A few highlights:
-
Simplified representation of postprocessors as just a Function struct (from the previous full Transformation)
-
Switching to the new platform implementation of backtraces, and fixing some related corner cases, which provides much faster response when you encounter an error backtrace
-
Adding a new opendp.extrinsics Python module, providing a place for code contributions and proofs outside of Rust
-
Performing a whole-codebase reformat using rustfmt to get everything to a clean state and minimize spurious churn in future contributions
-
An improved version of the User Guide based on reader feedback
-
A new First Look at DP notebook, which is a great resource to understand the basic concepts of DP in the context of a concrete example
Take It for a Spin!
Further details can be found in the repository CHANGELOG. We're excited to have you try OpenDP Library 0.7! You can find it on PyPI, crates.io, or GitHub.
We welcome your feedback and participation in the OpenDP Project. To learn more, please visit the OpenDP website or join our Slack workspace.
