ClipSketch AI: Frame-Accurate Video Tagging & AI Storyboard Generation

ClipSketch AI is a specialized productivity tool designed for video editors, social media managers, and remix creators. It extracts video content directly from Bilibili and Xiaohongshu links, providing frame-accurate tagging precision down to the millisecond. Once you have captured your desired moments, the platform utilizes Google Gemini’s multimodal model to transform those tagged clips into clean, hand-drawn storyboards with a single click. The AI also drafts social media captions in three distinct tones: emotional storytelling, instructional guides, and concise, punchy hooks. Users can upload a custom character or avatar, which the tool then integrates into the storyboard scenes. The interface supports high-quality cover generation, batch processing for storyboards, and granular shot refinement. Built with a responsive design, the platform functions across PC, iPad, and mobile devices without performance trade-offs.

ClipSketch AI manages the entire video preparation workflow by addressing several practical pain points:

• Multi-platform video import – Paste any Bilibili or Xiaohongshu link to begin. The system handles raw text containing short links effectively. The built-in player automatically adapts to vertical (9:16) or widescreen formats. It also features full keyboard support: use the Space bar to play or pause, and the left/right arrow keys for frame-by-frame scrubbing or smart stepping.

• Frame-level tagging – Press the "T" key to capture a specific shot instantly. You can export your timeline tags as a TXT file or download all tagged frames as a ZIP archive of images.

• AI-powered creative tools – Leveraging Google Gemini, the tool merges multiple tagged frames into a cohesive, hand-drawn storyboard. The AI assistant also generates captions tailored for platforms like Xiaohongshu—offering emotional, instructional, or short-form options. By uploading a personal character or face, the AI can weave that specific subject into the generated artwork. Additionally, you can create vertical covers based on your script and original frames or use batch-processing to enhance multiple storyboards at once.

• Universal accessibility – The interface is fully responsive. On a desktop, it utilizes the full widescreen layout; on iPads or smartphones, the elements stack vertically to ensure usability without the need for manual pinching or zooming.

How to Install and Use ClipSketch AI

Prerequisites

Ensure you have Node.js (v18 or higher) installed. You will also need a valid Google Gemini API Key to access the AI-driven features.

Installation & Setup

• Clone the repository: git clone https://github.com/RanFeng/clipsketch-ai.git cd clipsketch-ai

• Install dependencies: npm install

• Configure environment variables: Create a .env.local file in the root directory and add your key: GEMINI_API_KEY=your_api_key_here

• Launch the development server: npm run dev Open http://localhost:3000 in your web browser.

Using ClipSketch AI

• Import a video – Copy a share link from Bilibili or Xiaohongshu (the tool can parse the link even if it is embedded in descriptive text). Paste it into the input field on the home page and click “Import Video.”

• Tag your source clips (素材) – Use the Space bar to control playback and the arrow keys to navigate frames. Click the "Tag" button or press "T" to save specific moments.

• AI Generation – Once tagging is complete, select “Next: AI Drawing” at the bottom of the right panel. If you have not configured the API key in your environment file, you can paste it into the field in the top-right corner. The AI will analyze the video sequence to generate a hand-drawn storyboard. You can then blend in custom characters, redraw specific shots at higher resolutions, or use batch operations for efficiency.

• Exporting – Download your storyboard images, custom covers, or a ZIP file containing all assets. You can also copy your preferred AI-generated captions with a single click.

Technical Notes

• API Permissions – The AI drawing feature requires access to the gemini-1.5-pro or similar multimodal models. If you encounter a 403 error, verify your Google Cloud project permissions and API quotas.

• Cross-Origin Handling – To play external videos and capture screenshots, the tool employs a proxy and sets referrerPolicy="no-referrer".

• Tech Stack – The application is built on React 19 and TypeScript. It uses Tailwind CSS for styling and Lucide React for iconography. The AI functionality is powered by the Google GenAI SDK, while local state and file handling are managed via JSZip, the Canvas API, and IndexedDB.