Introducing OpenDP Library v0.6

 

Dear OpenDP Community,

 

The OpenDP team is delighted to announce the release of OpenDP Library 0.6!

 

The OpenDP Library is a modular collection of algorithms for building privacy-preserving applications, with an extensible approach to tracking privacy, and a vetted implementation. It is available as binaries for Python on PyPI, for Rust on crates.io, or in source form on GitHub.

This release is an important milestone for us, as it lays the groundwork for incorporating mathematical proofs of OpenDP code, as well as adding some nice documentation and usability improvements.

Proofs, Proofs, Proofs!

The key enhancement in OpenDP 0.6 is the integration of mathematical proofs into our code review and documentation process. Mathematical verification is one of the key differentiators of OpenDP; our goal is to have proofs accompany all relevant aspects of the library code. We've been hard at work developing document conventions and a logistical framework for these proofs, and this release is the first reflection of that.

 

For an example, have a look at the documentation for the make_count() constructor. You'll see how this incorporates a link to the proof (still undergoing vetting), as well as references to a paper with relevant theory and the underlying Rust function. These documents and links are all generated automatically from LaTeX sources and other metadata incorporated directly into our repository, right alongside the code. Our continuous integration system discovers new and updated proofs, associates them with a specific commit hash for revision tracking, attaches them to the relevant Pull Request for review, and stitches everything together when publishing to the documentation site. This was a huge amount of work to engineer! But we think the effort was worthwhile because it greatly reduces the friction in creating and maintaining proofs.

 

This release is our first step in this direction, and as such, most proofs are still under development and not yet present in the published version. Future releases will see us adding more proof documents as they are authored and reviewed. But this infrastructure is essential for giving us a blueprint for proof structure and a process for incorporating new work. Please take a look; we'd love to hear your feedback about the form and function of these proofs!

Documentation Updates

To accompany the integrated proofs, we've made some helpful improvements to other aspects of the library documentation. Our growing collection of example notebooks are now rendered to HTML and incorporated directly into the docs website. This way, you can browse the examples without having to fire up Jupyter. We've also improved the API documentation throughout the library. Inline Rust documentation has been greatly enhanced with explanations of behavior and mathematical constraints, as well as identification of all arguments and generic types. (See SmoothedMaxDivergence for a nice example leveraging our new KaTeX capability.) For contributors, there are updates to the Developer Guide, including an introduction to using Rust in OpenDP and guidelines for writing proofs.

Usability Improvements

Our efforts haven’t been limited to proofs and documentations, so there are plenty of other goodies in the grab bag this time around:

• Additional combinators for converting privacy measure:

  • make_pureDP_to_fixed_approxDP()

  • make_pureDP_to_zCDP()

• Additional accuracy functions for discrete noise mechanisms:

  • discrete_laplacian_scale_to_accuracy()

  • discrete_gaussian_scale_to_accuracy()

  • accuracy_to_discrete_laplacian_scale()

  • accuracy_to_discrete_gaussian_scale().

• A transformation to compute a Lipschitz b-ary tree of counts:

  • make_b_ary_tree()

• Utilities for dataframe subsetting:

  • make_subset_by()

  • make_df_is_equal()

  • make_df_cast_default()

• A combinator for using transformations as a post-processing function:

  • make_chain_tm()

• Cleanup of module names to make things more consistent and legible:

  • meas → measurements

  • trans → transformations

  • comb → combinators

• A new feature flag, honest-but-curious, to allow opting into mechanisms which require honest behavior on the part of the caller for correct operation (for example, properly stating the population size in the make_population_amplification() combinator).

• Various smaller bug fixes.

Check It Out!

Further details can be found in the CHANGELOG. We're excited to have you try OpenDP Library 0.6! You can find it on PyPI, crates.io, or GitHub. (And for the eagle-eyed: Yes, the latest version is actually 0.6.1, which fixes a small documentation issue discovered shortly after the release of 0.6.0.)

We welcome your feedback and participation in the OpenDP Project. To learn more, please visit the OpenDP website or join our Slack workspace.