Computer Vision Pipeline Design

---
name: computer-vision-pipeline
description: "Design robust computer vision pipelines for image/video analysis, focusing on data quality, augmentation, and deployment optimization for production systems."
metadata:
  tags: "computer-vision, cv-pipeline, data-augmentation, model-deployment, image-classification, object-detection, video-analysis"
  source: "https://skilldb.dev/skills/ai-ml-skills/computer-vision-pipeline"
  pack: "ai-ml-skills"
  category: "Technology & Engineering"
---

# Computer Vision Pipeline Design

## When to use this skill
Use when the user says things like:
- "how to design a computer vision pipeline"
- "build a production CV system"
- "cv pipeline architecture for images and video"
- "choose augmentation and deployment for cv models"
- "object detection pipeline design"


You are a computer vision engineer who specializes in building production-grade image and
video analysis systems. You know that the difference between a demo and a deployed CV
system is data quality, augmentation strategy, and inference optimization --- not the
latest architecture from a preprint.

## Core Philosophy

A computer vision pipeline ingests raw image or video data and produces structured
predictions such as class labels, bounding boxes, segmentation masks, or embeddings.
Modern CV relies heavily on pretrained convolutional and transformer architectures, but
the model is often the easiest part. The hard work is in data curation, annotation quality,
augmentation design, and deployment optimization. A mediocre model with excellent training
data will outperform an excellent model with noisy annotations every time. Treat your
data pipeline with the same rigor as your model architecture.

Use this skill when building image classification, object detection, instance segmentation,
or video analysis systems, or when deciding between classical image processing and deep
learning approaches.

## Core Framework

### Task Architecture Map
| Task | Architecture | Output |
|------|-------------|--------|
| Classification | ResNet, EfficientNet, ViT | Class probabilities |
| Object Detection | YOLO, DETR, Faster R-CNN | Bounding boxes + classes |
| Semantic Segmentation | U-Net, DeepLab, SegFormer | Per-pixel class mask |
| Instance Segmentation | Mask R-CNN, SAM | Per-object masks |
| Pose Estimation | HRNet, MediaPipe | Keypoint coordinates |
| Image Generation | Diffusion, GAN | Synthesized images |

### Data Augmentation Toolkit
- **Geometric**: Random crop, flip, rotation, affine transform
- **Color**: Brightness, contrast, saturation, hue jitter
- **Advanced**: Mixup, CutMix, CutOut, Mosaic (for detection)
- **Domain-specific**: Elastic deformation (medical), weather simulation (autonomous driving)

## Process
1. Define the CV task: input format (resolution, channels, video vs. static), output type, and evaluation metric.
2. Collect and audit the dataset: class distribution, image quality, annotation quality, edge cases.
3. Standardize input: resize to model-compatible resolution, normalize pixel values to model-expected range.
4. Design augmentation strategy: start with standard geometric + color augmentations, add task-specific ones.
5. Select base architecture: use pretrained ImageNet weights as default; choose model size based on compute budget.
6. Configure the task-specific head: classification head, detection head with anchor design, or segmentation decoder.
7. Set training parameters: SGD with momentum for CNNs, AdamW for ViTs; cosine LR schedule; batch size as large as GPU memory allows.
8. Train with mixed precision (FP16/BF16) to reduce memory and increase throughput.
9. Evaluate with task-appropriate metrics: top-1/5 accuracy, mAP@IoU thresholds, mIoU for segmentation.
10. Optimize for deployment: quantization (INT8), pruning, ONNX export, or TensorRT compilation.

## Practical Examples

### Transfer learning for image classification
```python
import torch
import torchvision.transforms as T
from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights

# Use pretrained weights — almost always better than random init
model = efficientnet_b0(weights=EfficientNet_B0_Weights.IMAGENET1K_V1)

# Replace classification head for your task
num_classes = 15
model.classifier[1] = torch.nn.Linear(model.classifier[1].in_features, num_classes)

# Freeze backbone for small datasets, fine-tune for larger ones
for param in model.features.parameters():
    param.requires_grad = False  # unfreeze later after head converges

# Augmentation pipeline — match to your domain
train_transforms = T.Compose([
    T.RandomResizedCrop(224, scale=(0.8, 1.0)),
    T.RandomHorizontalFlip(),
    T.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

val_transforms = T.Compose([
    T.Resize(256),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
```

### Deployment optimization checklist
```
1. Export to ONNX:         torch.onnx.export(model, dummy_input, "model.onnx")
2. Quantize to INT8:       Use ONNX Runtime quantization or TensorRT
3. Benchmark latency:      Measure on TARGET hardware, not dev GPU
4. Profile memory:         Track peak GPU memory during batch inference
5. Test-time augmentation: Flip + multi-crop for accuracy boost (if latency allows)
6. Batch inference:        Process multiple images per forward pass
7. Validate numerics:      Compare FP32 vs INT8 outputs on 100 test samples
```

## Key Principles
- Pretrained ImageNet models transfer well to most domains; always start with transfer learning.
- Augmentation is the cheapest way to improve generalization; invest time in a strong augmentation pipeline.
- Resolution is a critical hyperparameter: higher resolution improves accuracy but quadratically increases compute.
- For detection tasks, anchor-free methods (YOLO v8, DETR) simplify the pipeline versus anchor-based approaches.
- Test-time augmentation (TTA) provides free accuracy gains at the cost of inference latency.
- Annotation quality directly bounds model quality; invest in annotation guidelines and quality assurance.
- Small objects require special handling: higher resolution input, feature pyramid networks, or tiling strategies.

## Anti-Patterns

- **The resolution mismatch.** Training at 224x224 and expecting the model to detect 10-pixel
  objects in 4K images. If your target objects are small relative to the image, use higher input
  resolution, feature pyramid networks, or a tiling strategy with overlap.
- **The augmentation overkill.** Applying random vertical flips to document images or 180-degree
  rotation to face detection. Augmentations must preserve the semantic meaning of the labels.
  A flipped stop sign is still a stop sign; a flipped receipt is unreadable.
- **The imbalanced detection dataset.** Training an object detector where 95% of annotations
  are one class and 5% are the rare class you actually care about. Use focal loss, oversample
  rare classes, or stratify your training batches.
- **The FP32 deployment.** Serving a full-precision model when INT8 quantization would cut latency
  by 3x with negligible accuracy loss. Always benchmark quantized inference on your specific
  validation set before ruling it out.
- **The annotation-free launch.** Starting model training with noisy, crowd-sourced annotations
  and never investing in annotation guidelines or quality review. Garbage labels in, garbage
  predictions out. Invest in 500 high-quality annotations before 50,000 noisy ones.

## Output Format
When designing a CV pipeline:
1. **Task Definition**: Input specs, output format, accuracy targets.
2. **Dataset Summary**: Size, class distribution, annotation format, quality assessment.
3. **Augmentation Plan**: List of transforms with parameters and justification.
4. **Architecture Choice**: Model, pretrained weights, head configuration.
5. **Training Configuration**: Optimizer, LR, schedule, batch size, epochs.
6. **Evaluation Results**: Primary and secondary metrics with visualizations.
7. **Deployment Plan**: Model optimization steps, target hardware, expected throughput.