Memvid: Store Millions of Text Chunks in a Single MP4 File

Memvid reimagines the video file as a database. By encoding raw text into MP4 frames, it enables users to search millions of text chunks in under a second—all without a database server.

While traditional vector databases often consume significant RAM and disk space, Memvid compresses knowledge into a single, portable video file. This approach improves storage efficiency by an order of magnitude. Once created, these files function entirely offline and run on standard CPUs. The core library is highly efficient, consisting of approximately 1,000 lines of Python code.

What It Does

• Video-Based Storage: A single MP4 can house millions of text segments.
• Semantic Search: Use natural language queries to locate information instantly.
• Integrated Chat: Conduct context-aware conversations with your video-stored memory.
• Direct PDF Import: Index documents immediately with no extra pre-processing.
• Rapid Retrieval: Access results in less than a second, even when handling massive datasets.
• Minimalist Architecture: Designed for CPUs; no GPU is required.
• Flexible Model Support: Compatible with OpenAI, Anthropic, and local embedding models.

Use Cases

• Digital Libraries: Consolidate thousands of volumes into one searchable file.
• Education: Archive course materials as searchable, portable video files.
• News Archives: Compress years of reporting into a manageable, offline format.
• Corporate Knowledge Bases: Search across internal documentation instantly.
• Scientific Research: Execute semantic searches over vast collections of academic papers.
• Personal Notes: Convert years of writing into a searchable, interactive assistant.

Installation

# Base installation
pip install memvid

# Installation with PDF support
pip install memvid PyPDF2

# Recommended: Use a virtual environment
mkdir my-memvid-project && cd $_
python -m venv venv
source venv/bin/activate      # macOS/Linux
venv\Scripts\activate         # Windows
pip install memvid PyPDF2

Basic Usage

from memvid import MemvidEncoder, MemvidChat

chunks = ["Important fact 1", "Important fact 2", "Historical event details", ...]
encoder = MemvidEncoder()
encoder.add_chunks(chunks)
encoder.build_video("memory.mp4", "memory_index.json")

# Interact with your stored data
chat = MemvidChat("memory.mp4", "memory_index.json")
chat.start_session()
response = chat.chat("What do you know about the historical event?")
print(response)

Building from Documents

from memvid import MemvidEncoder
import os

encoder = MemvidEncoder(chunk_size=512, overlap=50)

for file in os.listdir("documents"):
    with open(f"documents/{file}", "r") as f:
        encoder.add_text(f.read(), metadata={"source": file})

encoder.build_video(
    "knowledge_base.mp4",
    "knowledge_index.json",
    fps=30,           # Higher fps increases chunks processed per second
    frame_size=512    # Larger frames store more data per frame
)

Advanced Features

Semantic Search & Context

from memvid import MemvidRetriever

retriever = MemvidRetriever("knowledge_base.mp4", "knowledge_index.json")

results = retriever.search("machine learning algorithms", top_k=5)
for chunk, score in results:
    print(f"Score: {score:.3f} | Content: {chunk[:100]}...")

context = retriever.get_context("Explain neural networks", max_tokens=2000)
print(context)

Interactive Chat Interface

from memvid import MemvidInteractive

interactive = MemvidInteractive("knowledge_base.mp4", "knowledge_index.json")
interactive.run()  # Launches local interface at http://localhost:7860

Custom Embedding Models

from sentence_transformers import SentenceTransformer

custom_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
encoder = MemvidEncoder(embedding_model=custom_model)

Video Compression Tuning

encoder.build_video(
    "compressed.mp4",
    "index.json",
    fps=60,
    frame_size=256,
    video_codec='h265',
    crf=28
)

Troubleshooting

• Module not found: Ensure your virtual environment is active.
• Missing PDF support: Verify you have installed PyPDF2 via pip.
• OpenAI API key: Set your key using export OPENAI_API_KEY="your-key" (macOS/Linux) or set OPENAI_API_KEY=your-key (Windows).
• Handling Large PDFs: If memory usage is high, try reducing the chunk size: encoder = MemvidEncoder(chunk_size=400).

Memvid vs. Traditional Approaches