NLP Pipeline Design

---
name: nlp-pipeline
description: "Design and evaluate end-to-end NLP pipelines from ingestion to output for tasks like classification, NER, and QA."
metadata:
  tags: "nlp, pipeline-design, text-processing, transformers, model-selection, task-framing, evaluation"
  source: "https://skilldb.dev/skills/ai-ml-skills/nlp-pipeline"
  pack: "ai-ml-skills"
  category: "Technology & Engineering"
---

# NLP Pipeline Design

## Overview
An NLP pipeline transforms raw text into structured predictions or representations. Modern NLP is dominated by transformer-based models, but effective pipelines still require careful text preprocessing, task framing, and post-processing. The pipeline design must account for language diversity, domain-specific vocabulary, and the tradeoff between pretrained model capability and task-specific fine-tuning.

Use this skill when building text classification, named entity recognition, question answering, summarization, or other NLP systems, or when evaluating whether to use a pretrained model, fine-tune, or prompt.

## Core Framework

### Pipeline Architecture
```
Raw Text -> Cleaning -> Tokenization -> Encoding -> Model -> Post-processing -> Output
```

### Approach Selection
| Approach | When to Use | Data Requirement |
|----------|-------------|-----------------|
| Prompting (zero/few-shot) | Quick prototyping, low data | 0-20 examples |
| Fine-tuning pretrained | Production quality needed | 1k-100k labeled examples |
| Training from scratch | Highly specialized domain | 1M+ examples |
| Classical ML + TF-IDF | Simple tasks, low compute | 100-10k examples |

### Task Taxonomy
- **Classification**: Sentiment, intent, topic, spam detection
- **Token-level**: NER, POS tagging, chunking
- **Span extraction**: QA, keyphrase extraction
- **Generation**: Summarization, translation, dialogue
- **Similarity**: Semantic search, duplicate detection, clustering

## Process
1. Define the NLP task precisely: input format, output format, label schema, evaluation metric.
2. Collect and audit the text corpus: language distribution, average length, domain vocabulary, label distribution.
3. Design text cleaning: lowercasing (if case-insensitive), Unicode normalization, HTML/URL removal, language detection.
4. Select tokenization strategy: WordPiece/BPE for transformers, domain-specific tokenizer if needed.
5. Choose the base model: task complexity and data size determine whether to prompt, fine-tune, or train.
6. Implement the model with appropriate head: classification head, token classification head, or seq2seq.
7. Design the training loop: learning rate (2e-5 to 5e-5 for fine-tuning), warmup steps, early stopping on validation metric.
8. Build post-processing: confidence thresholds, entity merging, output formatting.
9. Evaluate on held-out test set with task-appropriate metrics (F1 for NER, accuracy for classification, ROUGE for summarization).
10. Deploy with monitoring for input drift, prediction distribution shifts, and latency.

## Key Principles
- Pretrained transformers (BERT, RoBERTa, DeBERTa) are the default starting point for most NLP tasks.
- Text cleaning should be minimal for transformer models; they handle noise better than classical methods.
- Tokenizer and model must match; never use a tokenizer from a different model family.
- For multilingual tasks, use multilingual models (XLM-R) rather than translating to English.
- Label quality matters more than label quantity; 1000 clean examples beat 10000 noisy ones.
- Long documents require chunking strategies with overlap or hierarchical models.
- Evaluation must include per-class metrics, not just aggregate scores, to catch systematic failures.

## Common Pitfalls
- Over-cleaning text and removing signal (e.g., removing punctuation that matters for sentiment).
- Fine-tuning with a learning rate too high for the pretrained model, causing catastrophic forgetting.
- Ignoring class imbalance in classification tasks and reporting misleading accuracy.
- Using BLEU/ROUGE as sole metrics for generation without human evaluation.
- Failing to handle out-of-vocabulary tokens and special characters in production inputs.
- Not accounting for maximum sequence length limits and silently truncating important text.

## Output Format
When designing an NLP pipeline:
1. **Task Specification**: Input/output format, label schema, success criteria.
2. **Data Summary**: Corpus statistics, language coverage, quality assessment.
3. **Architecture Decision**: Approach chosen (prompting/fine-tuning/training) with rationale.
4. **Pipeline Diagram**: Each stage with expected input/output shapes.
5. **Model Selection**: Specific model checkpoint and why.
6. **Evaluation Plan**: Metrics, test set composition, baseline comparisons.
7. **Deployment Requirements**: Latency, throughput, model size constraints.