Loupe Semantic Search — Design

Status: agreed design for v0.1.0. Implementation follows this document.

Purpose

Add semantic (vector) retrieval to Loupe Search:

Hybrid site search. Merge Loupe’s keyword hits with vector-similarity hits for the front-end WordPress search and the REST /search endpoint (used by Loupe Search Block).
RAG retrieval. Expose abilities that return relevant passages with source URLs, so AI agents (for example through the MCP Adapter) can answer from site content and cite it.

Identity

Item	Value
Plugin name	Loupe Semantic Search
Slug / folder / text domain	`loupe-semantic-search`
Main file	`loupe-semantic-search.php`
Namespace	`Soderlind\Plugin\LoupeSemanticSearch`
Abilities	`loupe-semantic/retrieve`, `loupe-semantic/search`
Hook / option prefix	`loupe_semantic_`
Version	0.1.0
Requires	WordPress 7.0.3, PHP 8.3
Requires Plugins	`loupe-search, ai`
License	GPL-2.0-or-later

Architecture

                    ┌──────────── ai plugin (WordPress/ai) ─────────────┐
 post saved ──────▶ │ Change_Listener → Sync_Queue → Sync_Worker         │
                    │   Post_Source::get_text → Text_Chunker → embed     │
                    │   via Connector provider → Embedding_Repository    │
                    └────────────────────────────────────────────────────┘
                                         ▲ register_embedding_consumer()
                                         │
 search query ──▶ Loupe keyword hits ──┐ │
                                       ├─┴─ Loupe Semantic Search: embed query, rank
 search query ──▶ query vector ────────┘    vectors, fuse (RRF), filter, rebuild passages

Dependency on the `ai` plugin

WordPress 7 core ships Connectors and the AI Client, but no embedding API. The ai plugin (1.5.0+) provides generate_embeddings() on top of the connectors and a complete sync pipeline. Loupe Semantic Search does not implement its own queue, chunker, or vector storage.

All calls into the ai plugin live in one adapter class (Ai_Adapter). It checks the ai plugin version and the presence of the classes and functions it uses; if anything is missing the add-on deactivates its features and shows an admin notice. Used surface:

WordPress\AI\supports_embedding_generation(), WordPress\AI\generate_embeddings()
WordPress\AI\register_embedding_consumer() (before init priority 20)
Embedding_Sync::start_backfill(), cancel_backfill(), reset_backfill(), get_status(), prune_target(), instance()->get_repository(), instance()->get_registry()
Embedding_Repository::iterate(), get_content_hash()
Text_Chunker, Object_Processor::hash_text(), Vector_Math / Vector_Ranker
Filters wpai_embedding_sync_is_indexable, wpai_embedding_sync_max_chunks

Provider and model

The admin connects a provider under Settings → Connectors (WordPress 7). Without a connected provider that offers embedding models, the add-on is inactive and shows a notice linking to Connectors.
The settings page lists only connected providers’ models with the embedding_generation capability. The admin must choose one; nothing is selected automatically.
Optional dimensions setting, passed through when the model supports it.
Changing the model: register the consumer with the new target, start a backfill, and prune the old target’s vectors once no other consumer uses it.

Indexing

Consumer ID loupe-semantic-search, objects [ 'post' => <Loupe post types> ], where the post types come from loupe_search_post_types (via Loupe’s utilities).
Indexability follows Loupe. Loupe Search gets a public helper loupe_search_is_post_indexable( WP_Post $post ): bool wrapping its existing checks (post type, indexable statuses, password protection, loupe_search_is_indexable). We hook wpai_embedding_sync_is_indexable and veto a post that Loupe would not index.
- The filter is shared by every ai consumer, so the veto only applies when no other consumer covers that post type. Otherwise the other consumer would lose content it needs.
Text, chunking and hashing are the ai plugin’s: title plus raw content as plain text (the_content is not applied), Text_Chunker defaults (750 characters, 125 overlap), and one content hash per post. Unchanged posts are not re-embedded.
Backfill: started from the settings page, or with wp ai embeddings sync backfill loupe-semantic-search. Our WP-CLI command adds status and query (see below).

Retrieval

Vector search

Vector_Store interface with a default Ai_Repository_Vector_Store. Selectable through the loupe_semantic_vector_store filter so a native vector backend can replace it later.
Default store: iterate the model’s vectors in batches of 200 with Embedding_Repository::iterate(), computing cosine similarity and keeping a running top-K. The ai table has no vector index, so this is a full scan; about 20,000 chunks is the documented practical limit.
Keep the best chunk per post. Drop chunks below the minimum similarity (default 0.35).
Query vectors are cached in a transient for 30 minutes, keyed by provider, model, dimensions and a hash of the normalised query.

Passage text

Stored records contain vectors, not text. For the top hits only:

Rebuild the text with the ai plugin’s post text source (same filter chain).
Compare Object_Processor::hash_text() with the stored hash.
If they match, chunk with Text_Chunker, apply the same max-chunks limit, and take the chunk at the stored chunk_index.
If they differ (content changed and not yet re-embedded), fall back to the post excerpt and mark the passage stale: true.

Hybrid ranking

Reciprocal rank fusion: score = w_k / (60 + rank_keyword) + w_s / (60 + rank_semantic). Default weights 1.0 and 1.0, set in settings.
Fallback to keyword-only when embedding the query fails, takes longer than 1.5 s, the rate limit or daily cap is hit, or no model is configured. Fallbacks never throw to the visitor.

Where hybrid applies

Surface	Hook	Default
Front-end WordPress search	`loupe_search_order_results` (existing)	on
REST `/search` and Loupe Search Block	`loupe_search_advanced_hits` (new in Loupe Search)	on
Abilities	own registration	on once a model is chosen

A REST request can opt out with semantic: false.
Hybrid is skipped when the request sorts explicitly by a field other than _score, or by geo distance: the caller asked for that order, not relevance. The front-end filter gets the applied field sort as its fourth argument for the same check.
REST filters: semantic-only hits are kept only if the context’s matches_filter callback returns them. Loupe runs the request’s own filter string as an empty-query search, so every filter Loupe supports (taxonomies, custom fields, dates, geo) applies. Loupe caps that lookup at 1000 matches; with a broader filter some semantic-only hits may be dropped, never wrongly added.
Facet counts stay as Loupe computes them from keyword hits (known v1 limitation).
Every semantic hit is checked with loupe_search_is_post_indexable() at query time, because bulk exclusion may change only post meta and the vector may still exist.
Query operators: quotes, -term and OR are stripped before embedding. If the query contains an exclusion (-term), semantic ranking only reorders keyword hits and adds none, so excluded terms never come back through the semantic side.
loupe_search_order_results runs before Loupe caches front-end results, so hybrid results are cached together with the keyword results. When the semantic side falls back, keyword-only results may be cached for the cache lifetime (accepted v1 limitation). REST keeps caching only Loupe’s keyword results; fusion runs on every request.

Abilities

Category loupe-semantic. Both default to current_user_can( 'read' ); the loupe_semantic_ability_permission filter can open them up.

`loupe-semantic/retrieve`

Input: query (required), limit (default 5, max 20), post_type (optional string or list).

Output:

{
  "passages": [
    { "post_id": 12, "title": "…", "url": "…", "post_type": "post", "post_date": "…",
      "chunk_index": 2, "text": "…", "score": 0.71, "stale": false }
  ],
  "model": { "provider": "openai", "model": "text-embedding-3-small" }
}