Deep Search Lighting: Lightweight Web Search for LLMs

Most LLM web search tools are over-engineered. They often require a high-end frontier model, a premium search API, and a complex parsing routine for a dozen JSON fields before an agent can even decide its next move. For developers running smaller models or working within a tight budget, this complexity usually leads to failure.

Deep Search Lighting offers a faster, leaner alternative. It serves as a thin, efficient layer between your LLM and the web, removing the need for heavy frameworks or vendor lock-in. It simply provides search results that a model can actually process and use.

The Limitations of Standard Approaches

• Inconsistent Quality: Search result quality varies wildly, which makes it difficult for a model to check its own work (reflection).
• High Entry Barrier: Existing pipelines often assume you have access to GPT-4 and a paid search subscription.
• Complexity Issues: Smaller models frequently struggle with the intricate tool-calling schemas required by heavier tools.
• Context Fluctuations: Comprehension and stability often waver when the same search tool is applied across different model sizes.

How Deep Search Lighting Differs

• Framework-Agnostic: It integrates easily with any tech stack.
• Prioritizes Free APIs: Support for Baidu and DuckDuckGo is included out of the box, with Bocha and Tavily available as options.
• Adjustable Search Depth: You can increase thoroughness for complex tasks or reduce it to minimize latency.
• Integrated Reflection: Built-in logic allows the model to evaluate the relevance and accuracy of its own results.
• Optimized for Small Models: The system remains stable even when used with lightweight LLMs.

Included Search Engines

• Baidu (Free)
• DuckDuckGo (Free)
• Bocha (Requires API key)
• Tavily (Requires registration)

Technical Features

• Customizable reflection strategies and evaluation gates.
• Flexible LLM pipeline configurations.
• Full compatibility with OpenAI-style APIs.
• Minimalist source code designed for easy embedding into larger projects.
• Native support for MCP (Model Context Protocol) servers.

Getting Started

Set up your environment:

conda create -n deepsearch_lightning python==3.11
conda activate deepsearch_lightning
pip install -r requirements.txt

# Optional: For LangChain integration
pip install -r requirements_langchain.txt

Configuration:

1. Rename .env.examples to .env.
2. Enter your model details (supports any OpenAI-style API).
3. Baidu search is enabled by default; you can toggle other engines in the configuration.

Running the tool:

# Run a test case
python test_demo.py

# Launch the Streamlit interface
streamlit run streamlit_app.py

# Start the MCP server
python mcp_server.py
python langgraph_mcp_client.py

The Purpose of Deep Search Lighting

This tool was created to solve a specific, practical problem: integrating web search into LLM workflows without the usual overhead. It is model-agnostic and does not demand a corporate-sized search budget. It simply delivers clean, relevant data so your AI agent can focus on processing information.