AI Peer Review Tool for Neuroscience: LLM-Driven Meta-Review Framework

The AI Peer Review tool is an open-source framework designed to assist in the evaluation of academic manuscripts, specifically within the field of neuroscience. Rather than depending on a single AI perspective, the system subjects a paper to a panel of diverse large language models—including GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro, DeepSeek R1, and Llama 4 Maverick. Each model conducts an independent assessment. The tool then synthesizes these critiques into a comprehensive meta-review, filtering out redundant information and producing a clear table that identifies which specific models raised particular concerns.

Core Functionality

• Submit a manuscript to receive independent evaluations from a suite of LLMs. • Extract unique viewpoints and critical insights from each model. • Generate a synthesized meta-review that highlights consensus themes and outlier observations. • Produce a structured concerns table to visualize the distribution of findings across models. • Export analytical data in Markdown, CSV, and JSON formats.

Supported Models

• GPT-4o & GPT-4o-mini (via OpenAI API) • Claude 3.7 Sonnet (via Anthropic API) • Google Gemini 2.5 Pro (via Google AI API) • DeepSeek R1 & Llama 4 Maverick (via Together API)

Installation and Setup

Create and Activate a Virtual Environment (Recommended)

python -m venv .venv
# On macOS/Linux:
source .venv/bin/activate  
# On Windows:
.venv\Scripts\activate

Install in Development Mode

pip install -e .

API Key Configuration

Via the CLI Configuration Command

ai-peer-review config openai "your-openai-key"
ai-peer-review config anthropic "your-anthropic-key"
ai-peer-review config google "your-google-key"
ai-peer-review config together "your-together-ai-key"

Configured keys are securely stored in ~/.ai-peer-review/config.json.

Via Environment Variables

Option 1: Export keys directly in the terminal session.

export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key"
export GOOGLE_API_KEY="your-google-key"
export TOGETHER_API_KEY="your-together-ai-key"

Option 2: Utilize a .env file.

Copy the provided template and enter your credentials:

cp .env.example .env
# Edit the .env file to include your API keys.

The system will automatically check for a .env file in the current directory or at ~/.ai-peer-review/.env.

Command Line Interface Usage

Analyze a Manuscript Using All Available Models

ai-peer-review review path/to/paper.pdf

Analyze Using Specific Models Only

ai-peer-review review path/to/paper.pdf --models "gpt4-o1,claude-3.7-sonnet"

Direct Output to a Specific Directory

ai-peer-review review path/to/paper.pdf --output-dir custom/output/path

Bypass Meta-Review Generation

ai-peer-review review path/to/paper.pdf --no-meta-review

Apply a Custom Configuration File

ai-peer-review --config-file /path/to/config.json review path/to/paper.pdf

Prompts and Configuration

Default Prompting Logic

Review Prompt By default, the following instructions are sent to each LLM:

You are an expert in neuroscience with a focus on brain imaging. Provide a thorough, critical peer review of this submitted research manuscript. Begin with a summary of the study and its findings, then proceed with a detailed, point-by-point analysis of the flaws and limitations.

Meta-Review Prompt To generate the synthesis, the tool uses the following logic:

The attached file contains individual peer reviews for a research manuscript. Synthesize these into a meta-review. Highlight both the consensus points shared by all reviewers and the specific concerns noted by a subset. Finally, rank the reviews based on their perceived utility and how well they identified critical issues.

Note: Model identifiers are removed from individual reviews prior to meta-analysis to minimize model-specific bias.

Custom Prompts

Users can modify the communication style by editing the configuration file:

1. Open or create the configuration file at ~/.ai-peer-review/config.json.
2. Add or modify the prompts object:

{
  "api_keys": {
    ...
  },
  "prompts": {
    "system": "Enter your custom system instructions here.",
    "review": "Enter your custom review instructions. You must include the {paper_text} placeholder.",
    "metareview": "Enter your custom meta-review instructions. You must include the {reviews_text} placeholder."
  }
}

If a configuration file is not present, the tool will generate one with the default prompts automatically.

Using an External Configuration Path Use the --config-file flag to apply specific settings for a single session. This will override existing API keys and prompt configurations for that specific run.

ai-peer-review --config-file /path/to/custom.json review paper.pdf

Output Structure

The tool generates several files within the output directory (defaulting to ./papers/[paper_name]):

• review_[model_name].md: The full independent critique from each specific LLM. • meta_review.md: A comprehensive synthesis and comparative analysis of all feedback. • concerns_table.csv: A structured table identifying specific issues and which models flagged them. • results.json: A machine-readable file containing all individual reviews, the meta-review, and the reviewer-model mapping.