> ## Documentation Index
> Fetch the complete documentation index at: https://docs.enconvo.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Knowledge Base Deep Dive
> Advanced configuration and optimization for your knowledge bases
## Overview
This guide covers the technical architecture and advanced configuration options for EnConvo's knowledge base system. If you are new to knowledge bases, start with the [Knowledge Base introduction](/ai/knowledge-base) first.
EnConvo's knowledge base is built on a dual-database architecture: **LanceDB** for vector storage and similarity search, and **SQLite** (via Drizzle ORM) for metadata management. This combination provides fast semantic search with rich metadata capabilities.
## Architecture
### Dual Database Design
```
Document → Chunking → Embedding → Storage
│
┌──────────┼──────────┐
│ │
LanceDB SQLite
(Vector Store) (Metadata)
│ │
Embedding vectors Knowledge base records
Text chunks Attachment details
Similarity search Content chunk metadata
```
**LanceDB** stores document embeddings for semantic similarity search. Each knowledge base gets its own LanceDB table plus an attachments table. Data is stored locally at `~/.config/enconvo/cache/knowledge_base/{dbName}`.
**SQLite** stores structured metadata: knowledge base records, attachment details, and content chunk information. Schema migrations are managed through Drizzle ORM.
### Knowledge Base Identifiers
Each knowledge base is identified by a composite key in the format `{dbName}|{tableName}`. This key is used throughout the system when referencing a specific knowledge base.
## Document Processing Pipeline
When you add content to a knowledge base, it flows through a multi-stage pipeline.
### Stage 1: Content Loading
The `AttachmentLoader` routes content to the appropriate loader based on type:
| Content Type | Loader | Description |
| - | - | - |
| **PDF** | pdf-parse | Extracts text from PDF documents |
| **DOCX** | mammoth | Converts Word documents to text |
| **PPTX** | officeparser | Extracts text from PowerPoint presentations |
| **EPUB** | epub2 | Parses e-book content |
| **XLSX / CSV** | xlsx, d3-dsv | Reads spreadsheet data |
| **TXT / MD** | Direct read | Plain text and Markdown files |
| **HTML** | cheerio, html-to-text | Parses and extracts text from HTML |
| **XML / JSON** | fast-xml-parser | Structured data parsing |
| **Images** | OCR Provider | Optical character recognition |
| **Audio / Video** | Transcription Provider | Speech-to-text conversion |
| **Webpages** | Link Reader Provider | Scrapes page content (supports sitemap.xml for entire websites) |
| **YouTube** | Dedicated handler | Extracts transcript and metadata |
### Stage 2: Text Chunking
After content is extracted, it is split into manageable chunks using `RecursiveCharacterTextSplitter` from LangChain:
| Parameter | Value | Description |
| - | - | - |
| **Chunk size** | 6,000 characters | Maximum size of each text chunk |
| **Chunk overlap** | 100 characters | Overlap between consecutive chunks for context continuity |
The recursive splitter tries to split on natural boundaries in this order:
1. Double newlines (paragraphs)
2. Single newlines
3. Sentences (periods, exclamation marks, question marks)
4. Words (spaces)
5. Characters (last resort)
The 6,000-character chunk size balances retrieval precision with context completeness. Smaller chunks improve precision but may lose context. Larger chunks preserve context but may include irrelevant information.
### Stage 3: Embedding
Each text chunk is converted to a vector embedding using your configured embedding model. EnConvo uses a custom `EnConvoEmbeddingFunction` that wraps the platform's embedding provider system.
Embeddings are generated in batches with concurrent processing for performance.
### Stage 4: Storage
Embedded chunks are stored in LanceDB with the following schema:
| Field | Type | Description |
| - | - | - |
| `text` | Source text | The original text chunk |
| `vector` | Float32 array | The embedding vector |
| `id` | UTF-8 string | Unique chunk identifier |
| Additional metadata | Various | Source file, position, timestamps, etc. |
## Embedding Model Selection
Choose an embedding provider available in the [Provider Catalog](/providers/catalog#embeddings) and the knowledge base's settings. Keep the same model and dimensions for indexing and query embeddings. Changing the model can require re-embedding the stored content; a new model selection alone does not convert existing vectors.
Test a representative document and query before importing a large collection. Consider language support, local resource use, cloud processing, and the provider's current limits.
## Search and Retrieval
EnConvo uses a hybrid search approach combining vector similarity with full-text search.
### Vector Similarity Search
The primary search method. When you query the knowledge base, your query is embedded using the same model, then compared against all stored vectors using cosine similarity.
```
Query "How does authentication work?"
→ Embed query → [0.12, -0.45, 0.78, ...]
→ Compare against all chunk vectors
→ Return top-K most similar chunks
```
### Full-Text Search (FTS)
LanceDB also supports full-text search with fuzzy matching for keyword-based retrieval. This catches cases where exact terms matter more than semantic meaning.
```
Query "OAuth2 PKCE flow"
→ Full-text index lookup with fuzziness
→ Return matching chunks
```
### Hybrid Search Strategy
For best results, EnConvo combines both approaches:
1. Run vector similarity search for semantic matches
2. Run full-text search for keyword matches
3. Merge and deduplicate results
4. Apply reranking (if configured) to produce the final ranking
### Reranking
Reranking models re-score search results for improved relevance. After the initial retrieval, a reranker evaluates each result against the query and reorders them.
Use the rerankers offered by your installed build; see [Provider Catalog](/providers/catalog#reranking). Reranking may involve a separate local model or cloud request.
Reranking is especially useful when you have large knowledge bases. The initial search retrieves a broad set of candidates, and the reranker narrows them down to the most relevant results.
## Performance Optimization
### Batch Embedding
When adding multiple documents, EnConvo processes embeddings in batches with concurrent processing. This significantly reduces the time to index large document collections compared to sequential processing.
### Index Management
LanceDB supports creating indexes for faster search:
* **Vector index**: Speeds up similarity search for large tables
* **FTS index**: Enables full-text search with fuzzy matching
Indexes are created automatically when needed. For very large knowledge bases, you may want to trigger re-indexing after bulk imports.
### Storage Considerations
Storage depends on the original assets, extracted text, chunk count, and embedding dimensions. Monitor actual disk use for your collection instead of estimating it only from the number of documents. Keep source files and backups independently of the search index.
## Multi-Knowledge Base Management
You can create multiple knowledge bases for different purposes. Each operates independently with its own:
* Embedding model
* Document collection
* Search index
* Storage location
### Organization Strategies
| Strategy | Description | Example |
| - | - | - |
| **By project** | One KB per project | `project-alpha`, `project-beta` |
| **By type** | Separate KBs by content type | `code-docs`, `meeting-notes`, `research` |
| **By team** | Team-specific knowledge | `engineering`, `marketing`, `support` |
| **By scope** | Personal vs shared | `personal-notes`, `company-wiki` |
### Querying Multiple Knowledge Bases
Attach the intended knowledge bases with the context picker, or configure the Agent's attached knowledge bases. Available choices depend on the installed build and Agent settings.
## Knowledge Base API
The knowledge base extension exposes API endpoints for programmatic access. These are used internally by EnConvo and can be accessed by other extensions and workflows.
### Key Endpoints
| Endpoint | Purpose |
| - | - |
| **Create Knowledge Base** | Create a new knowledge base with specified embedding model |
| **Delete Knowledge Base** | Remove a knowledge base and all its data |
| **Add Content** | Ingest files, text, URLs, or media into a knowledge base |
| **Search / Retrieve** | Perform semantic search against a knowledge base |
| **List Knowledge Bases** | List all available knowledge bases |
| **List Attachments** | List all documents in a knowledge base |
| **Delete Attachment** | Remove a specific document from a knowledge base |
| **Summarize** | Generate an AI summary of knowledge base content |
### Using in Workflows
Knowledge base operations can be used as steps in workflows:
```
Trigger: New file in ~/Documents/research/
→ Add to Knowledge Base "research"
→ Search for related content
→ Generate summary
→ Save summary to notes
```
## Advanced Configuration
### Chunking
The current text loader uses 6,000-character chunks with a 100-character overlap. These are implementation defaults, not a universal user-facing settings control. Parsing and indexing changes should be tested on a small collection before a rebuild.
### Monitoring and Maintenance
* **Re-index**: If search quality degrades after many incremental updates, re-index the knowledge base to rebuild indexes
* **Embedding model upgrades**: When better models become available, create a new knowledge base with the new model and re-ingest your content
* **Storage cleanup**: Delete knowledge bases you no longer need to reclaim disk space
## Troubleshooting
Try these steps in order:
1. Check that the embedding model is appropriate for your content language
2. Re-index the knowledge base to rebuild search indexes
3. Consider enabling reranking for better result ordering
4. If content is very domain-specific, try a more capable embedding model
Large documents take longer to process. To improve speed:
* Use a cloud embedding provider for batch processing
* Ensure your Mac has sufficient RAM (embedding models are memory-intensive)
* For local models, check that your Mac supports Metal acceleration
If a file type is not directly supported, try converting it to a supported format (PDF, TXT, or MD) before adding it to the knowledge base.
Knowledge bases are identified by composite keys (`dbName|tableName`). Ensure you are referencing the correct name. Check **Settings** -> **Knowledge Base** for the list of available knowledge bases.
## Related Features
Getting started with knowledge bases
Chat with your knowledge base
Automate knowledge base tasks
Configure embedding and reranking providers
## Linked-folder and indexing checks
For a missing item, inspect the source folder's import and synchronization status first. Then inspect the item's content-extraction status and embedding status separately: readable extracted text does not guarantee that vector indexing has finished. Retry the failed stage after fixing its provider or file-access error.
The source folder remains the authority for linked files. Keep offline-drive recovery separate from file deletion, and review skipped-file counts before treating an import as complete.
For client access, use [Knowledge Base MCP](/advanced/knowledge-base-mcp). It exposes selected knowledge bases through a local, read-only service; it does not export Agent Memory or recordings.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.