Search Implementation

---
name: search-implementation
description: "Implement and optimize fast, relevant search with indexing, analyzers, and autocomplete across engines like Elasticsearch, Algolia, and Meilisearch."
metadata:
  tags: "autonomous-agents, search, full-text-search, indexing-strategies, relevance-tuning, autocomplete, elasticsearch, meilisearch"
  source: "https://skilldb.dev/skills/autonomous-agent-skills/search-implementation"
  pack: "autonomous-agent-skills"
  category: "Autonomous Agents"
---

# Search Implementation

## When to use this skill
Use when the user says things like:
- "How do I implement search in my app?"
- "I need a fast full-text search with autocomplete."
- "Help me set up Elasticsearch for my product catalog."
- "Which search engine should I use for my app?"
- "I want to tune relevance for product search results."


You are an autonomous agent that builds search experiences users love. Search is often the primary way users find content in an application. A good search implementation is fast, forgiving of typos, and returns relevant results. You understand the trade-offs between search engines, know how to design indexes, and can tune relevance to match user expectations.

## Philosophy

Users expect search to just work. They expect it to be fast, to understand what they meant (even when they misspelled it), and to put the most relevant results at the top. This expectation is set by Google, and while you do not need to match Google's sophistication, you need to meet users where they are. Search is not just a database query — it is a user experience that requires thoughtful design at every layer from indexing to result presentation.

## Techniques

### Full-Text Search Basics
- Full-text search works by building an **inverted index**: a mapping from every word to the documents that contain it.
- **Tokenization** breaks text into individual terms. "The quick brown fox" becomes ["the", "quick", "brown", "fox"].
- **Normalization** lowercases text, removes accents, and standardizes characters so that "Cafe" matches "cafe" and "caf\é."
- **Stemming** reduces words to their root form: "running," "runs," and "ran" all map to "run." Use language-appropriate stemmers.
- **Stop word removal** filters out common words ("the," "is," "at") that add noise without improving relevance. Use cautiously — removing stop words can hurt phrase queries.
- **Analyzers** combine tokenization, normalization, stemming, and filtering into a pipeline. Choose or customize analyzers based on your content language and domain.

### Search Engine Selection
- **Elasticsearch** is the industry standard for large-scale, complex search. It offers powerful aggregations, geo-search, and extensive query DSL. Best for applications with complex relevance requirements and large document volumes.
- **Algolia** is a hosted search service optimized for speed and developer experience. Excellent for e-commerce and consumer-facing search with typo tolerance built in. Best when you want fast time-to-market.
- **Meilisearch** is an open-source alternative to Algolia with similar typo tolerance and simplicity. Good for smaller applications that want Algolia-like experience without vendor lock-in.
- **PostgreSQL full-text search** (`tsvector`, `tsquery`) is suitable for simpler search needs within an existing Postgres database. Avoids adding infrastructure but has limited relevance tuning.
- Choose based on scale, complexity, operational willingness, and budget. Do not use Elasticsearch for a 10,000-document blog. Do not use Postgres full-text search for a product catalog with complex faceting.

### Indexing Strategies
- Index only the fields users will search against. Do not dump entire database rows into the search index.
- Store additional fields for display (title, thumbnail URL, price) without indexing them for search.
- Use separate indexes for different content types (products, articles, users) when they have different schemas and relevance rules.
- Keep search indexes in sync with the primary database. Use change data capture (CDC), database triggers, or application-level events.
- Re-index on schema changes. Use index aliases in Elasticsearch to swap indexes atomically with zero downtime.
- Index at write time, not query time. The work of tokenizing, stemming, and building the inverted index happens during ingestion.

### Relevance Tuning
- **TF-IDF** (Term Frequency-Inverse Document Frequency) is the foundation: terms that appear frequently in a document but rarely across all documents score highest.
- **BM25** is the modern default scoring algorithm in Elasticsearch and most search engines. It improves on TF-IDF with better term saturation handling.
- Boost important fields: a match in the title should score higher than a match in the body. Use field-level boosts (e.g., `title^3, body^1`).
- Use **function scoring** to blend text relevance with business signals: popularity, recency, rating, sales volume.
- Test relevance with a set of representative queries and expected top results. Automate these as relevance regression tests.
- Avoid over-tuning. Small boost adjustments compound in unpredictable ways. Make changes incrementally and measure impact.

### Fuzzy Matching and Typo Tolerance
- Use **edit distance** (Levenshtein distance) to match terms that are 1-2 characters off: "recieved" matches "received."
- Configure maximum edit distance based on word length: 0 edits for words under 3 characters, 1 edit for 3-5 characters, 2 edits for longer words.
- Use **n-gram tokenization** for partial matching: "eleph" matches "elephant."
- Combine fuzzy matching with exact match boosting so that exact matches always rank above fuzzy matches.
- Phonetic matching (Soundex, Metaphone) helps when users know how a word sounds but not how it is spelled.

### Faceted Search
- Facets let users filter results by categories (brand, price range, color, size). They are essential for e-commerce and catalog search.
- Compute facet counts from the current result set, not the entire index. Facets should update as users apply filters.
- Display only facets with results. Do not show a "Red" filter if no red items match the current query.
- Use hierarchical facets for nested categories: Electronics > Phones > Smartphones.
- Implement facets using aggregations in Elasticsearch or built-in faceting in Algolia/Meilisearch.

### Autocomplete Implementation
- Use **prefix matching** for search-as-you-type: "app" matches "apple," "application," "appetizer."
- Implement with edge n-grams at index time (not query time) for best performance.
- Return suggestions in under 100ms. Autocomplete latency tolerance is much lower than regular search.
- Show 5-8 suggestions maximum. More creates decision fatigue.
- Highlight the matching portion of each suggestion so users can see why it matched.
- Consider separate suggestion indexes optimized for prefix queries rather than using the main search index.

### Search Result Ranking
- Combine text relevance score with business relevance signals using a weighted formula.
- Business signals to consider: click-through rate, conversion rate, recency, popularity, editorial curation.
- Use personalization cautiously: boost results based on user history or preferences, but do not create filter bubbles.
- Pin promoted or sponsored results at specific positions, clearly labeled, without distorting organic results.

## Best Practices

- Measure search quality with metrics: click-through rate on top results, zero-result rate, search refinement rate.
- Log search queries and results for analysis. Identify common queries with poor results and improve them.
- Handle zero-result searches gracefully: suggest alternative queries, show popular items, or broaden the search automatically.
- Implement query understanding: detect categories ("red shoes"), filter extraction ("under $50"), and intent classification.
- Use synonyms to match domain-specific vocabulary: "laptop" should match "notebook computer."
- Paginate with `search_after` (Elasticsearch) or cursor-based pagination, not deep offset pagination.
- Test search performance under load. Search queries can be expensive — monitor and optimize slow queries.

## Anti-Patterns

- **Using SQL LIKE for search.** `LIKE '%term%'` cannot use indexes, does not rank by relevance, and does not handle typos. Use a proper search engine.
- **Indexing everything.** Indexing irrelevant fields bloats the index, slows queries, and returns noisy results. Be selective.
- **Ignoring zero-result queries.** Every zero-result search is a user who failed to find what they wanted. Track and address these.
- **No relevance testing.** Changing boost values without measuring the impact on search quality leads to degraded results over time.
- **Synchronous indexing on write.** Updating the search index synchronously during user writes adds latency. Index asynchronously unless real-time search is required.
- **One-size-fits-all ranking.** The same ranking formula does not work for product search, document search, and user search. Tune per content type.
- **Ignoring search analytics.** Without understanding what users search for and whether they find it, you cannot improve the search experience.