Privacy
YouTube search... ...Google search
- Risk, Safety, Compliance and Regulation ... Ethics ... Privacy ... Law ... AI Governance ... AI Verification and Validation
- Cybersecurity ... OSINT ... Frameworks ... References ... Offense ... NIST ... DHS ... Screening ... Law Enforcement ... Government ... Defense ... Lifecycle Integration ... Products ... Evaluating
- Policy ... Policy vs Plan ... Constitutional AI ... Trust Region Policy Optimization (TRPO) ... Policy Gradient (PG) ... Proximal Policy Optimization (PPO)
- Blockchain
- Data Science ... Governance ... Preprocessing ... Exploration ... Interoperability ... Master Data Management (MDM) ... Bias and Variances ... Datasets
- OpenMined
- Facial recognition has to be regulated to protect the public, says AI report | Will Knight - MIT Technology Review
- Other Challenges in Artificial Intelligence
- Data Security and Privacy | IBM Research
The availability of massive amounts of data, coupled with high-performance cloud computing platforms, has driven significant progress in artificial intelligence and, in particular, machine learning and optimization. Indeed, much scientific and technological growth in recent years, including in computer vision, natural language processing, transportation, and health, has been driven by large-scale data sets which provide a strong basis to improve existing algorithms and develop new ones. However, due to their large-scale and longitudinal collection, archiving these data sets raise significant privacy concerns. They often reveal sensitive personal information that can be exploited, without the knowledge and/or consent of the involved individuals, for various purposes including monitoring, discrimination, and illegal activities. The AAAI Workshop on Privacy-Preserving Artificial Intelligence
|
|
|
|
|
|
|
|
Privacy Preserving - Machine Learning (PPML) Techniques
Youtube search... ...Google search
- Secure and Private AI by Facebook AI | UDACITY ... introduction of three cutting-edge technologies for privacy-preserving AI: Federated Learning, Differential Privacy, and Encrypted Computation. Learn how to extend PyTorch with the tools necessary to train AI models that preserve user privacy.
- Perfectly Privacy-Preserving AI | Patricia Thaine
Many privacy-enhancing techniques concentrated on allowing multiple input parties to collaboratively train ML models without releasing their private data in its original form. This was mainly performed by utilizing cryptographic approaches, or differentially-private data release (perturbation techniques). Differential privacy is especially effective in preventing membership inference attacks. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Multiparty Computation (MPC) enables computation on data from different providers/parties, such that the other participating parties gain no additional information about each others’ inputs, except what can be learned from the public output of the algorithm. In other words, when we have the parties Alice, Bob and Casper, all three have access to the output. However, it is not possible for, e.g., Alice to know the plain data Bob and Casper provided. Secure Multiparty Computation — Enabling Privacy-Preserving Machine Learning | Florian Apfelbeck - Medium
- DeepSecure DeepSecure: Scalable Provably-Secure Deep Learning | B. Rouhani, M. S. Riazi, and F. Koushanfar
- SecureML SecureML: A System for Scalable Privacy-Preserving Machine Learning | Payman Mohassel & Yupeng Zhang
- MiniONN Oblivious Neural Network Predictions via MiniONN transformations | J. Liu, M. Juuti, Y. Lu and N. Asokan
- ABY3 ABY3: A Mixed Protocol Framework for Machine Learning | Payman Mohassel & Peter Rindal
Cryptographic Approaches
Youtube search... ...Google search
When a certain ML application requires data from multiple input parties, cryptographic protocols could be utilized to perform ML training/testing on encrypted data. In many of these techniques, achieving better efficiency involved having data owners contribute their encrypted data to the computation servers, which would reduce the problem to a secure two/three party computation setting. In addition to increased efficiency, such approaches have the benefit of not requiring the input parties to remain online. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Homomorphic Encryption
Youtube search... ...Google search
Fully homomorphic encryption enables the computation on encrypted data, with operations such as addition and multiplication that can be used as basis for more complex arbitrary functions. Due to the high cost associated with frequently bootstrapping the cipher text (refreshing the cipher text because of the accumulated noise), additive homomorphic encryption schemes were mostly used in PPML approaches. Such schemes only enable addition operations on encrypted data, and multiplication by a plain text. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Garbled Circuits
Youtube search... ...Google search
Assuming a two-party setup with Alice and Bob wanting to obtain the result of a function computed on their private inputs, Alice can convert the function into a garbled circuit, and send this circuit along with her garbled input. Bob obtains the garbled version of his input from Alice without her learning anything about Bob’s private input (e.g., using oblivious transfer). Bob can now use his garbled input with the garbled circuit to obtain the result of the required function (and can optionally share it with Alice). Some PPML approaches combined additive homomorphic encryption with Garbled circuits. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Secret Sharing
Youtube search... ...Google search
A method for distributing a secret among multiple parties, with each one holding a “share” of the secret. Individual shares are of no use on their own; however, when the shares are combined, the secret can be reconstructed. With threshold secret sharing, not all the “shares” are required to reconstruct the secret; but only “t” of them (“t” refers to threshold). In one setting, multiple input parties can generate “shares” of their private data, and send these shares to a set of non-colluding computation servers. Each server could compute a “partial result” from the “shares” it received. Finally, a results’ party (or a proxy) can receive these partial results, and combine them to find the final result. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Secure Processors
Youtube search... ...Google search
While initially introduced to ensure the confidentiality and integrity of sensitive code from unauthorized access by rogue software at higher privilege levels, Intel SGXprocessor are being utilized in privacy-preserving computation. Ohrimenko et al.14 developed a data oblivious ML algorithms for neural networks, SVM, k-means clustering, decision trees and matrix factorization that are based on SGX-processors. The main idea involves having multiple data owners collaborate to perform one of the above mentioned ML tasks with the computation party running the ML task on an SGX-enabled data center. An adversary can control all the hardware and software in the data center except for the SGX-processors used for computation. In this system, each data owner independently establishes a secure channel with the enclave (containing the code and data), authenticates themselves, verifies the integrity of the ML code in the cloud, and securely uploads its private data to the enclave. After all the data is uploaded, the ML task is run by the secure processor, and the output is sent to the results’ parties over secure authenticated channels. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Perturbation Approaches
Youtube search... ...Google search
- Dimensionally reduction (DR) perturbs the data by projecting it to a lower dimensional hyperplane to prevent reconstructing the original data, and/or to restrict inference of sensitive information. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
- Differential privacy (DP) techniques resist membership inference attacks by adding random noise to the input data, to iterations in a certain algorithm, or to the algorithm output. While most DP approaches assume a trusted aggregator of the data, local differential privacy allows each input party to add the noise locally; thus, requiring no trusted server.
Dimensionality Reduction (DR)
Youtube search... ...Google search
perturbs the data by projecting it to a lower dimensional hyperplane. Such transformation is lossy, and it was suggested by Liu et al.24 that it would enhance the privacy, since retrieving the exact original data from a reduced dimension version would not be possible (the possible solutions are infinite as the number of equations is less than the number of unknowns). Hence, Liu et al.24 proposed to use a random matrix to reduce the dimensions of the input data. Since a random matrix might decrease the utility, other approaches used both unsupervised and supervised DR techniques such as principal component analysis (PCA), discriminant component analysis (DCA), and multidimensional scaling (MDS). These approaches try to find the best Projection matrix for utility purposes, while relying on the reduced dimensionality aspect to enhance the privacy. Since an approximation of the original data can still be obtained from the reduced dimensions, some approaches, e.g. Jiang et al.25, combined dimensionality reduction with DP to achieve differentially-private data publishing. While some entities might seek total hiding of their data, DR has another benefit for privacy. For datasets that have samples with two labels: a utility label and a privacy label, Kung26 proposes a DR method to enable the data owner to project her data in a way that enables maximizing the accuracy of learning for the utility labels, while decreasing the accuracy for learning the privacy labels. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Differential Privacy (DP)
Youtube search... ...Google search
Differential privacy is a powerful tool for quantifying and solving practical problems related to privacy. Its flexible definition gives it the potential to be applied in a wide range of applications, including Machine Learning applications. Understanding Differential Privacy - From Intuitions behind a Theory to a Private AI Application | An Nguyen - Towards Data Science
Differential privacy is a system for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset. Another way to describe differential privacy is as a constraint on the algorithms used to publish aggregate information about a statistical database which limits the disclosure of private information of records whose information is in the database. For example, differentially private algorithms are used by some government agencies to publish demographic information or other statistical aggregates while ensuring confidentiality of survey responses, and by companies to collect information about user behavior while controlling what is visible even to internal analysts. Roughly, an algorithm is differentially private if an observer seeing its output cannot tell if a particular individual's information was used in the computation. Differential privacy is often discussed in the context of identifying individuals whose information may be in a database. Although it does not directly refer to identification and reidentification attacks, differentially private algorithms probably resist such attacks. Differential privacy was developed by cryptographers and thus is often associated with cryptography, and draws much of its language from cryptography. Wikipedia
Local Differential Privacy (LDP)
Youtube search... ...Google search
When the input parties do not have enough information to train a ML model, it might be better to utilize approaches that rely on local differential privacy (LDP). With LDP, each input party would perturb their data, and only release this obscure view of the data. An old, and well-known version of local privacy is randomized response (Warner 1965), which provided plausible deniability for respondents to sensitive queries. For example, a respondent would flip a fair coin: (a) if “tails”, the respondent answers truthfully, and (b) if “heads”, then flip a second coin, and respond “Yes” if heads, and “No” if tails. RAPPOR 22 is a technology for crowdsourcing statistics from end-user client software by applying RR to Bloom filters with strong 𝜀-DP guarantees. RAPPOR is deployed in Google Chrome web browser, and it permits collecting statistics on client-side values and strings, such as their categories, frequencies, and histograms. By performing RR twice with a memoization step in between, privacy protection is maintained even when multiple responses are collected from the same participant over time. A ML oriented work, AnonML23, utilized the ideas of RR for generating histograms from multiple input parties. AnonML utilizes these histograms to generate synthetic data on which a ML model can be trained. Like other local DP approaches, AnonML is a good option when no input party has enough data to build a ML model on their own (and there is no trusted aggregator). Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University
Differential Privacy Mathematical Framework
Differential privacy mathematical framework is used to share statistical patterns about a group of people while withholding information about any specific individual in the dataset. The U.S. Census Bureau introduced a differential privacy-based disclosure avoidance system for the 2020 Census to protect respondent identities against modern database reconstruction attacks while still publishing useful demographic data.
Think of differential privacy like looking at a pointillist painting from across a room. When you stand twenty feet back, you clearly see the park, the trees, and the crowd on the grass. If you walk right up to the canvas with a magnifying glass to inspect a single dot, the artist has blurred the individual strokes just enough that you can't tell what color any solitary dot originally was. You get the big picture, but the individual contribution remains hidden.
How It Works
The central idea behind differential privacy is to introduce carefully calibrated statistical noise into data outputs. This allows researchers and policy analysts to identify broad population patterns without reliably determining facts about a specific person or household.
In practice, a differentially private statistical pipeline follows five core stages:
- Collect actual data: The organization gathers raw individual and household records into a secure, confidential database.
- Calculate statistics: The system runs queries on the raw records to compute baseline population counts, averages, and demographic summaries.
- Add mathematical noise: A randomized algorithm modifies the calculated counts using a known probability distribution, obscuring any single person's contribution.
- Enforce consistency: A postprocessing step adjusts the noisy results so they obey logical rules, such as preventing negative population counts and ensuring that local totals add up to state totals.
- Publish useful statistics: The organization releases the cleaned, privacy-protected tables so researchers can analyze trends without ever touching the underlying personal records.
Illustrative Example
Suppose a rural census block contains 100 residents, including 12 people in a specific age bracket. If the agency publishes the exact count of 12, someone with outside voter registration records or commercial data might cross-reference the block's traits and lock onto a specific resident's identity. By applying differential privacy, the system nudges the number slightly before publication.
| Metric | Resident Count | Notes |
|---|---|---|
| Total Block Population | 100 | Held consistent at the block housing level or adjusted within geographic bounds |
| Actual Count (Target Age Group) | 12 | Confidential figure stored in the internal database |
| Illustrative Published Count | 14 | Includes calibrated statistical noise (illustrative, not an actual Census calculation) |
An analyst looking at the published table still sees that this age group makes up a small slice of the neighborhood (roughly 12% to 15%). However, because the published count of 14 includes deliberate randomness, an outside snooper can't prove whether a specific neighbor falls into that age bracket.
The Mathematics of Privacy
Differential privacy doesn't rely on vague promises to anonymize data by stripping out names and Social Security numbers. Instead, it provides a formal, mathematical guarantee that holds up regardless of how much outside information an attacker already possesses.
A randomized algorithm <math>M</math> satisfies <math>(\varepsilon, \delta)</math>-differential privacy if, for all datasets <math>D</math> and <math>D'</math> that differ by at most one person's record, and for all possible output sets <math>S</math>:
<math>P[M(D)\in S]\leq e^\varepsilon P[M(D')\in S]+\delta</math>
Each variable in this inequality plays a specific role in governing privacy:
- <math>D</math> and <math>D'</math> (Neighboring Datasets): Two versions of a database that are identical except that one specific person's data is included in <math>D</math> and removed (or altered) in <math>D'</math>.
- <math>M</math> (The Randomized Mechanism): The algorithm that calculates the statistic and injects mathematical noise (typically drawn from a Laplace, Gaussian, or discrete Gaussian distribution).
- <math>\varepsilon</math> (Epsilon, the Privacy Loss Budget): The dial that controls how much an output probability is allowed to shift when one person enters or leaves the dataset. A smaller <math>\varepsilon</math> forces the outputs of <math>M(D)</math> and <math>M(D')</math> to look nearly identical, providing stronger privacy protection at the expense of statistical accuracy.
- <math>\delta</math> (Delta, the Failure Probability): A tiny allowance (often set smaller than one divided by the total population size) for the probability that the strict <math>\varepsilon</math> bound might not hold in extreme tail events. When <math>\delta = 0</math>, the system achieves "pure" differential privacy.
You can think of <math>\varepsilon</math> like a financial budget. Every time you publish a new table or answer a query from the same confidential dataset, you spend a slice of your privacy budget. Once your total spending hits the limit you set, you can't release more slices of the data without weakening your privacy guarantee.
The U.S. Census Bureau's TopDown Algorithm
When the Census Bureau designed its 2020 Disclosure Avoidance System (DAS), engineers realized they couldn't just add independent random numbers to every single census block and add them up. If you add random noise to millions of tiny blocks and sum them together to get a state's population, the errors either compound into massive distortions or allow attackers to compare hierarchy levels and strip the noise away. Even worse, raw random noise frequently produces impossible numbers, like "-3 residents" or "4.7 children" in a neighborhood.
To solve this, the Census Bureau built the TopDown Algorithm (TDA). Instead of building from the block up, TDA starts at the national level and works its way down the geographic ladder: Nation to State, State to County, County to Census Tract, Tract to Block Group, and finally down to the individual Census Block.
At each rung of that geographic ladder, the TopDown Algorithm executes two distinct technical phases:
1. The Measurement Phase (Noisy Measurements)
First, the algorithm takes the confidential internal dataset, known as the Census Edited File (CEF), and calculates a massive set of detailed demographic histograms. It then injects random noise drawn from a discrete Gaussian distribution into those counts.
The amount of noise depends on how much of the global privacy-loss budget (<math>\varepsilon</math>) is assigned to that specific geographic level and table type. Because the algorithm splits its total privacy budget across all six geographic levels and multiple demographic cross-tabulations, queries that need high precision receive a larger share of the <math>\varepsilon</math> budget (meaning less noise), while less critical cross-tabulations receive a smaller share (more noise).
2. The Estimation and Postprocessing Phase
Once the noisy measurements are generated for a geographic level, they are messy and inconsistent. A county's noisy demographic subtotals might not add up to the parent state's population, and rural blocks might show negative headcounts.
To clean this up without looking back at the raw confidential data (which would violate differential privacy), TDA runs a constrained convex optimization problem using least-squares (<math>L_2</math>) minimization, followed by controlled integer rounding. This mathematical solver adjusts the noisy numbers just enough to satisfy two strict rulebooks:
- Structural Inequalities: Population counts must be non-negative integers. You can't have fewer than zero people in any demographic bucket.
- Invariants (Zero-Noise Constraints): Certain numbers are locked in as exact truths by policy or constitutional requirement. For the 2020 Census, total state populations (used for congressional apportionment), the total number of housing units on a census block, and the number and type of group quarters facilities (like dorms or nursing homes) on a block were held invariant with zero noise added.
Think of the TopDown Algorithm like a corporate CFO finalizing an annual budget across regional branches. The national total is locked in at headquarters first. When the state offices submit their noisy estimates, the math forces all fifty state totals to equal the national figure. Next, each county inside a state has to reconcile its numbers to match its state's locked-in total, continuing step-by-step down to the street block.
Applying Differential Privacy in AI Model Training
Differential privacy has become a cornerstone of modern artificial intelligence because deep learning models have a habit of memorizing their training data. When you train a large neural network on medical charts, financial transactions, or private emails, the model doesn't just learn general rules. Like a student who memorizes the exact wording of a textbook instead of understanding the subject, an unprotected AI can accidentally regurgitate a patient's name or allow an attacker to run a "membership inference attack" to prove that a specific person's record was in the training set.
To let an AI learn general patterns without memorizing individuals, engineers apply differential privacy directly to the training loop.
Differentially Private Stochastic Gradient Descent (DP-SGD)
Standard neural networks learn through Stochastic Gradient Descent (SGD). The model looks at a batch of examples, measures its prediction error, calculates a "gradient" (a directional nudge telling the model how to adjust its internal weights), and updates itself. DP-SGD modifies this learning loop in three specific steps:
- Compute Per-Example Gradients: Instead of calculating one combined gradient for a whole batch of 64 or 256 records at once, the system computes the exact gradient for each individual record in the batch separately.
- Clip Each Gradient (Bounding Sensitivity): If one outlier record in the batch has bizarre or extreme data, it will produce a massive gradient that pulls the model's weights sharply in its direction. DP-SGD puts a hard ceiling (a clipping threshold <math>C</math>) on the length (<math>L_2</math> norm) of every individual gradient. If a record's gradient exceeds <math>C</math>, the algorithm scales it down to <math>C</math>. This acts like a volume limiter on every microphone in a town hall meeting: everyone gets to speak, but no single person can shout loud enough to dominate the room.
- Add Calibrated Gaussian Noise: After clipping all the individual gradients in the batch, the system averages them together and adds random Gaussian noise scaled to the clipping ceiling <math>C</math>. Only after this noise is mixed in does the AI update its weights.
Because an AI model updates its weights thousands of times during training, engineers attach a Privacy Accountant (using techniques like Rényi Differential Privacy or the Moments Accountant) to the training loop. The accountant tracks how much <math>\varepsilon</math> is spent on every batch update and halts training the moment the cumulative privacy budget runs out.
Private Aggregation of Teacher Ensembles (PATE)
Another practical way to train AI privately is PATE. Instead of training one giant model with noisy gradients, you split your sensitive dataset into dozens of separate, non-overlapping partitions. You train an independent "teacher" model on each private partition.
Next, you take an unlabeled, non-sensitive public dataset and ask all the teacher models to vote on the correct label for each example. You add Laplace or Gaussian noise to the teachers' vote tallies and use only the winning noisy labels to train a final "student" model. If all the teachers agree on a general medical rule, the vote margin is huge and the noise doesn't change the outcome. If a decision hinges on a single patient's weird record, only one teacher knows about it, and the voting noise buries that single vote. The student model is then safe to deploy publicly.
Trade-offs Between Privacy and Accuracy
Differential privacy is not a free lunch. Mathematics dictates a fundamental tension known as the privacy-utility trade-off: every bit of privacy protection you gain requires sacrificing some degree of statistical sharpness.
| Dimension | High Privacy (Low <math>\varepsilon</math>, e.g., 0.1 to 1.0) | High Utility (High <math>\varepsilon</math>, e.g., 10.0+) |
|---|---|---|
| Individual Protection | Very strong; an attacker learns almost nothing new about any individual. | Weakened; outliers and unique combinations of traits become easier to reconstruct. |
| Large Population Counts | Remains highly accurate because random noise is tiny relative to millions of people. | Extremely accurate; nearly identical to raw unmasked counts. |
| Small Rural Blocks & Rare Groups | Heavy distortion; noise can easily double or zero out a tiny headcount. | Moderately accurate; preserves small neighborhood ratios better. |
| AI Model Convergence | Slower training; model struggles to learn rare edge cases and minority classes. | Fast convergence; model captures fine-grained nuances across subclasses. |
1. Small Populations and the Signal-to-Noise Ratio
The hardest trade-off in differential privacy hits small geographic areas and rare demographic groups. Because the magnitude of added noise is calibrated to protect one individual (a sensitivity of 1), the raw noise stays roughly the same size whether a group has 10 people or 100,000 people.
Adding or subtracting 3 random people in a city of 100,000 changes the population by 0.003%, which is statistical dust. Adding or subtracting 3 people in a rural subgroup of 6 residents swings the count by 50%. Researchers studying Native American reservations, small rural towns, or rare pediatric diseases have to account for the fact that small counts carry high relative error.
2. Systematic Bias from Postprocessing
When algorithms like the Census Bureau's TopDown system force noisy numbers to be non-negative (turning a noisy "-2" into a "0"), they introduce a subtle upward bias in empty or near-empty categories. If a rural county has dozens of demographic categories with zero actual residents, positive random noise stays positive while negative random noise gets chopped off at zero. To make the county total still add up to the state total, the optimization solver has to trim counts slightly from the largest population groups to pay for the artificial bumps in the tiny categories.
3. Fairness vs. Privacy in Machine Learning
In AI training, clipping gradients in DP-SGD affects minority groups more than majority groups. Common examples in a dataset already produce small, predictable gradients because the model quickly learns to get them right. Rare or underrepresented examples (such as uncommon dialects in speech recognition or rare skin tones in dermatology images) produce larger gradients because the model hasn't seen many of them yet. When DP-SGD clips all large gradients down to a ceiling <math>C</math> and adds noise, it disproportionately mutes the learning signal coming from those minority records. Engineers have to tune the clipping threshold and batch size carefully so that protecting privacy doesn't inadvertently make the AI less accurate for the exact groups that need fair representation.
General Data Protection Regulations (GDPR)
YouTube search... ...Google search
|
|
|
|
|
|