The OpenDP team is excited to bring you our latest release, OpenDP Library 0.8!
The OpenDP Library is a modular collection of algorithms for building privacy-preserving applications, with an extensible approach to tracking privacy, and a vetted implementation. It is available as binaries for Python on PyPI, for Rust on crates.io, or in source form on GitHub.
This release features a number of improvements to OpenDP APIs, with the aim of streamlining user programs. It also includes an early preview of some new functionality, plus some quality-of-life improvements.
Partial Constructors and Simplified Chaining
In OpenDP Library 0.8, we’ve tweaked the way constructors look, making it easier to build chains of operations. First, the signatures of most constructors have been standardized, so that the first two arguments are always the input Domain and input Metric. Then, we’ve added an alternate version of every constructor, where those two arguments are removed, and the name is changed from make_* to then_*. (You’ll see why shortly.)
For example:
make_laplace(input_domain, input_metric, scale, QO='float')
then_laplace(scale, QO='float')
These then_* constructors are essentially the partial application of regular constructors: They return a partially-constructed Transformation or Measurement, which then requires supplying an input Domain and input Metric to be fully-constructed. (For the curious, the then_* partial constructors are generated automatically as part of a code generation pass in our build system.)
Finally, we’ve added an overload of the chaining operator >> that knows how to join everything when building a chain. It takes the output Domain and Metric of the preceding Transformation (or a bare tuple that you supply as the input space), and uses them to populate the input Domain and Metric of the following Transformation or Measurement.
Putting it all together, chains now look like the following:
>>> dp_sum = (
... (input_domain, input_metric) >>
... dp.t.then_clamp((0.0, 5.0)) >>
... dp.t.then_sum() >>
... dp.m.then_laplace(5.0)
... )
In particular (thanks to work in OpenDP 0.7, where domain instances have metadata), you’ll notice that you no longer have to repeat the same bounds argument to multiple constructors! It’s a subtle change, but we’ve found that it’s cleaned up our code nicely. You’ll see more examples of this pattern throughout our updated documentation.
Preview of Context APIs
When writing an OpenDP program, you use a “function-centric” style: You call a sequence of constructors to produce a Measurement (which is essentially a kind of function). Then you invoke that Measurement, supplying your data as an argument. The advantage of this approach is that it closely matches the OpenDP Programming Framework, making it easy to reason about privacy.
However, we’ve received feedback that this style of code can be awkward for some application developers. So we’ve been exploring some alternative approaches, more in line with the data-centric style of popular dataframe libraries like Pandas. In these libraries, calculations typically start from a collection of data, with operations chained onto the collection object as a sequence of dot-separated function calls (what’s often called “builder notation”).
In OpenDP Library 0.8, we’re piloting a flavor of this, in something we’ve named the Context API. This aspect of our API still adheres to the OpenDP privacy calculus internally, but wraps it in a more user-friendly, data-centric abstraction.
The Context API lets you write analyses that look like this:
>>> context = dp.Context.compositor(
... data=...,
... privacy_unit=dp.unit_of(contributions=1),
... privacy_loss=dp.loss_of(epsilon=3.0),
... split_evenly_over=3
... )
>>> context.query().clamp((0.0, 5.0)).sum().laplace().release()
5.778190935559005
There are some nice properties of this approach:
-
You set privacy properties up front and explicitly, when creating the Context.
-
The system will automatically solve for an unbound parameter (typically noise scale) to satisfy the given privacy properties.
-
It requires less housekeeping of intermediate variables, as operations are created in-line with dot notation.
A more thorough sample can be found in this example notebook.
This is just a preview, as we’re still gaining experience with the approach. The details likely will change, but we’re excited about our early experiences, and think that this could be the way forward for most OpenDP programs. We’d love for you to give it a try and send us your feedback!
Grab Bag
It wouldn’t be an OpenDP Library release without some miscellaneous goodies! Here’s what we have this time:
Unified Constructors
As a follow-on to work in OpenDP Library 0.7, where domain instances carry more metadata, we’ve unified a number of common constructors under a single entry point, letting the system figure out the right thing and simplifying user code. Instead of having to name a specific constructor, you can use a general form, and the system will examine the input domain and/or input metric, choosing the appropriate implementation under the hood.
For instance, make_base_laplace() and make_base_discrete_laplace() have been merged into make_laplace(). Similar simplifications were done to create make_gaussian(), make_sum(), make_mean(), make_variance().
Additional Proofs
Continuing our efforts to expand the mathematical verification of OpenDP, we’ve added proofs for make_clamp() and make_row_by_row() (an internal building block used throughout the library).
Nightly builds
As the OpenDP Library has evolved over the past few years, its build system has had to come along for the ride. This resulted in a growing amount of complexity: Compiling for multiple operating systems and CPU architectures, generating language bindings, rendering LaTeX proofs, merging documentation trees, publishing to multiple repositories, etc. All of these steps made producing a new release a time-consuming and error-prone process for the OpenDP team.
For OpenDP 0.8, we’ve rewritten the CI pipeline from scratch, and it is much more robust now. This removes much of the toil and stress of creating new OpenDP releases. The new CI system also makes it feasible for us to support nightly releases, which are now available for OpenDP! This will allow you to access the latest and greatest OpenDP functionality, between formal releases, without having to compile from scratch yourself.
To access nightlies for Python, you can use the --pre flag to pip, which will enable installation of pre-release versions:
% pip install --pre opendp
For Rust projects, reference a version that includes the nightly marker (currently at 0.9.0):
[dependencies]
opendp = "0.9.0-nightly"
Please note that nightly builds don’t undergo full release validation, so you’ll need to decide if they’re appropriate for you. But if you run into problems, please let us know, and we’ll do our best to help out!
Getting OpenDP Library
Further details can be found in the repository CHANGELOG. We're excited to have you try OpenDP Library! You can find it on PyPI, crates.io, or GitHub.
We welcome your feedback and participation in the OpenDP Project. To learn more, please visit the OpenDP website or join our Slack workspace.
