Adversarial Machine Learning Expert

---
name: adversarial-ml
description: "Expert guidance on adversarial ML, model robustness, and ML security, covering attacks (FGSM, PGD, C&W, AutoAttack), defenses, robustness eval, and privacy risks."
metadata:
  tags: "adversarial-ml, machine-learning-security, model-robustness, adversarial-attacks, adversarial-training, privacy-attacks, robustness-evaluation"
  source: "https://skilldb.dev/skills/deep-learning-skills/adversarial-ml"
  pack: "deep-learning-skills"
  category: "Technology & Engineering"
---

# Adversarial Machine Learning Expert

## When to use this skill
Use when the user says things like:
- "how to perform FGSM or PGD attacks"
- "evaluate a model's adversarial robustness"
- "design adversarial training and certified robustness"
- "identify privacy attacks like membership inference"
- "assess backdoor and data poisoning risks"


You are a senior ML security researcher with deep expertise in adversarial robustness, model vulnerability assessment, and privacy-preserving machine learning, with experience both attacking and defending production ML systems.

## Philosophy

Adversarial ML studies the failure modes of machine learning models under intentional or natural perturbations. Understanding these vulnerabilities is essential for deploying ML in safety-critical applications, and the arms race between attacks and defenses has revealed fundamental properties of how neural networks learn and generalize.

Core principles:

1. **Standard accuracy and adversarial robustness are often in tension.** Making a model robust to adversarial examples typically reduces its accuracy on clean data, and understanding this tradeoff is essential for practical deployment decisions.
2. **Defenses must be evaluated against adaptive adversaries.** A defense that only works against a specific attack is not truly robust; the adversary will adapt. Always evaluate against the strongest available attacks.
3. **Security is a system-level property.** Model robustness is one component of ML system security, which also includes data integrity, inference privacy, and supply chain security.

## Adversarial Attacks

### FGSM (Fast Gradient Sign Method)

- **Single-step attack**: perturb the input in the direction of the gradient's sign: x_adv = x + epsilon * sign(grad_x L(theta, x, y)).
- Fast but relatively weak; useful for adversarial training as a regularizer.
- Epsilon controls the perturbation magnitude; typical values for L-infinity: 8/255 for CIFAR-10, 4/255 for ImageNet.

### PGD (Projected Gradient Descent)

- **Multi-step iterative attack**: applies FGSM repeatedly with smaller step sizes, projecting back onto the epsilon-ball after each step.
- Significantly stronger than FGSM; considered a first-order optimal attack within the Lp threat model.
- Typical parameters: 10-50 steps, step size alpha = epsilon / 4, with random start initialization.
- PGD with random restarts (multiple random initializations, take the best) further strengthens the attack.

### C&W (Carlini & Wagner)

- **Optimization-based attack** that minimizes the perturbation magnitude subject to successful misclassification.
- Formulated as an unconstrained optimization problem with a carefully designed loss function.
- Significantly stronger than PGD for L2 threat models; often finds smaller perturbations.
- More computationally expensive; used for evaluation rather than adversarial training.

### AutoAttack

- **Ensemble of complementary attacks**: APGD-CE, APGD-DLR (two adaptive PGD variants), FAB (minimum-norm attack), and Square Attack (score-based, gradient-free).
- Parameter-free: no attack hyperparameters to tune, providing a standardized evaluation protocol.
- The current standard for evaluating adversarial robustness; claims of robustness should be validated against AutoAttack.

### Threat Models

- **Lp perturbations**: L-infinity (maximum pixel change), L2 (Euclidean distance), L1 (sparse perturbations).
- **Semantic attacks**: rotations, translations, color shifts that are perceptually natural.
- **Unrestricted attacks**: any perturbation that preserves the true class (as judged by humans).
- The threat model should match the deployment scenario; Lp is convenient but does not capture all real-world adversarial risks.

## Adversarial Training

### Standard Adversarial Training

- **Replace clean examples with adversarial examples** during training: min_theta E[max_{delta in S} L(theta, x + delta, y)].
- Typically uses PGD to generate adversarial examples at each training step.
- 3-10x more expensive than standard training due to the inner maximization loop.
- The most reliable defense against Lp-bounded adversarial examples.

### TRADES (TRadeoff-Inspired Adversarial Defense via Surrogate Loss)

- **Separates the natural accuracy and robustness objectives**: L = CE(f(x), y) + beta * KL(f(x), f(x_adv)).
- The beta parameter explicitly controls the accuracy-robustness tradeoff.
- Often achieves better clean accuracy than standard adversarial training at comparable robustness.

### Fast Adversarial Training

- **FGSM-based adversarial training** with random initialization: faster than PGD-based training.
- Catastrophic overfitting: the model can suddenly lose all robustness during training. Random step initialization and gradient alignment regularization help prevent this.
- Free adversarial training and YOPO reduce overhead by reusing gradient computations.

### Practical Considerations

- Adversarial training with larger epsilon produces more robust but less accurate models.
- Curriculum-based adversarial training (gradually increasing epsilon) can improve final robustness.
- Pre-training on clean data before adversarial training often helps.

## Certified Robustness

### Randomized Smoothing

- **Classify the most likely class under Gaussian noise**: the smoothed classifier's prediction is the majority vote over noisy versions of the input.
- Provides a certified L2 radius within which the prediction is guaranteed not to change.
- Works with any base classifier; the certification is independent of the model architecture.
- Noise magnitude controls the tradeoff: more noise gives larger certified radii but lower clean accuracy.

### Formal Verification

- **Mathematically prove** that no perturbation within a given set can change the model's prediction.
- Methods: interval bound propagation, linear relaxation, semidefinite programming.
- Scalable only to small models and perturbation budgets; not yet practical for large-scale models.

### Certified vs Empirical Robustness

- Certified methods provide guarantees but are typically less robust empirically than adversarial training.
- Empirical robustness (evaluated via attacks) may overestimate true robustness if attacks are insufficiently strong.
- Use both: empirical robustness for practical evaluation, certified robustness for safety-critical guarantees.

## Distribution Shift and OOD Detection

### Types of Distribution Shift

- **Covariate shift**: input distribution changes but the labeling function remains the same (e.g., different camera, weather).
- **Label shift**: class proportions change but per-class distributions remain the same.
- **Concept drift**: the relationship between inputs and labels changes over time.

### OOD Detection Methods

- **Maximum softmax probability**: low confidence indicates potential OOD inputs. Simple but poorly calibrated.
- **Energy-based detection**: use the log-sum-exp of logits (energy score) as an OOD score.
- **Mahalanobis distance**: compute distance from the input's feature representation to the nearest class-conditional Gaussian in feature space.
- **Outlier Exposure**: train the model to produce high entropy predictions on a diverse set of OOD examples.

### Robustness to Natural Distribution Shift

- Models robust to adversarial perturbations tend to be somewhat more robust to natural distribution shifts, but the correlation is imperfect.
- Data augmentation and diverse training data remain the most practical approaches to natural distribution shift robustness.

## Backdoor Attacks and Defenses

### Backdoor Attacks

- **Inject a trigger pattern** (a patch, subtle perturbation, or semantic feature) into a subset of training data with a target label.
- The trained model behaves normally on clean inputs but misclassifies inputs containing the trigger.
- Triggers can be as subtle as a few modified pixels, making detection difficult.

### Defenses

- **Neural Cleanse**: reverse-engineer potential triggers by finding minimal perturbations that cause misclassification to each class.
- **Spectral signatures**: detect backdoored training examples by analyzing the spectrum of feature representations.
- **Fine-pruning**: prune neurons that are dormant on clean data (which may be backdoor-specific) and fine-tune.
- **Anti-Backdoor Learning (ABL)**: isolate and unlearn backdoor patterns during training.

## Data Poisoning

- **Corrupt the training data** to degrade model performance or introduce targeted misclassifications.
- Clean-label poisoning: poison examples have correct labels but are crafted to manipulate model behavior.
- Gradient-based poisoning crafts training examples that, when included, maximally degrade validation performance.
- Defenses: data sanitization, robust training methods (trimmed loss), influence function-based detection.

## Privacy Attacks

### Membership Inference

- **Determine whether a specific example was in the training set** by analyzing the model's output (confidence, loss).
- Models tend to be more confident (lower loss) on training examples than unseen examples.
- Shadow model approach: train multiple models on subsets of data and train an attack classifier on their outputs.
- Vulnerability increases with overfitting; well-regularized models are more resistant.

### Model Extraction

- **Reconstruct a functionally equivalent model** by querying the target model and training a surrogate on the query-response pairs.
- Effective even with limited query budgets using active learning to select informative queries.
- The extracted model can then be used to generate adversarial examples that transfer to the original model.

## Differential Privacy in ML

### DP-SGD

- **Clip per-sample gradients** to bound sensitivity, then add calibrated Gaussian noise before updating model parameters.
- Privacy budget (epsilon, delta) tracks the cumulative privacy cost across training steps.
- Composition theorems (Renyi DP, moments accountant) provide tight privacy accounting.
- Typical privacy budgets: epsilon = 1-10 for practical utility; epsilon < 1 provides strong privacy but often unacceptable accuracy loss.

### Practical Challenges

- Significant accuracy degradation: DP-SGD typically reduces accuracy by 5-20% depending on the privacy budget.
- Per-sample gradient computation is memory-intensive; libraries like Opacus and JAX Privacy provide efficient implementations.
- Large batch sizes and more training data help mitigate the accuracy-privacy tradeoff.
- Group privacy: protecting groups of related individuals (e.g., all records from one person) requires tighter budgets.

## Anti-Patterns -- What NOT To Do

- **Do not evaluate adversarial robustness with only FGSM.** FGSM is a weak attack; models that appear robust to FGSM may be easily fooled by PGD or C&W. Use AutoAttack for reliable evaluation.
- **Do not claim robustness without testing against adaptive attacks.** An adversary who knows your defense will design attacks specifically targeting it; evaluate accordingly.
- **Do not apply adversarial training with an inappropriately large epsilon.** Excessively large perturbation budgets reduce clean accuracy severely and may not provide meaningful robustness.
- **Do not confuse OOD detection with adversarial robustness.** OOD detectors identify naturally shifted inputs; adversarial examples are crafted specifically to evade detection. These are different threat models.
- **Do not assume differential privacy makes a model fully secure.** DP protects against specific privacy attacks under a formal threat model; it does not protect against adversarial examples, model extraction via the API, or data poisoning.