MindSearch

What is MindSearch?

MindSearch is an open-source AI search framework developed by the Shanghai AI Lab. It combines large-scale information gathering and organization capabilities. Leveraging the InternLM2.5 7B dialogue model, MindSearch can gather effective information from over 300 web pages in just 3 minutes, completing tasks that would typically take humans 3 hours. Using a multi-agent framework to simulate human thinking, it plans before searching, enhancing the accuracy and completeness of information. The project is fully open-source, allowing users to experience and deploy it locally for free.

Key Features of MindSearch

• Complex Query Processing: Breaks down complex user queries into smaller, manageable sub-questions for more precise information retrieval.
• Dynamic Graph Construction: Simulates human problem-solving processes by constructing directed acyclic graphs (DAGs), gradually refining problems and exploring solutions.
• Parallel Information Retrieval: Utilizes a multi-agent architecture to enable parallel searches for multiple sub-questions, improving speed and efficiency.
• Hierarchical Retrieval Strategy: Employs a coarse-to-fine retrieval strategy, first broadly gathering information and then selecting the most valuable pages for in-depth reading and extraction.
• Context Management: Effectively manages context information in multi-agent systems, ensuring coherence and integrity during information retrieval and integration.
• Response Generation: Synthesizes retrieved information to generate accurate, comprehensive, and in-depth responses to complex queries.
• Performance Enhancement: Significantly improves the quality of answers, including depth and breadth, in both closed and open-set question-answering tasks.
• Human Preference: Generates responses that align more closely with human preferences, making MindSearch's solutions more favored by human evaluators compared to other AI search engines.

Technical Principles of MindSearch

• WebPlanner: Acts as a high-level planner, decomposing user queries into sub-questions and simulating multi-step information-seeking mental models through dynamic graph construction (DAG).
• WebSearcher: Executes hierarchical information retrieval, gathering and summarizing valuable information from the internet based on sub-questions assigned by WebPlanner.
• Multi-Agent Collaboration: WebPlanner and WebSearcher operate as independent agents, handling problem decomposition and information retrieval tasks respectively, enabling parallel processing and effective information integration.
• Dynamic Graph Construction: Dynamically constructs logical graphs for problem-solving through code generation and execution, allowing LLMs to gradually refine problems and retrieve relevant information.
• Context Management: Ensures effective context state transfer between multi-agents, preventing the loss of critical information during retrieval and integration.

Project Links

• Official Website: https://mindsearch.netlify.app/
• Online Demo: https://mindsearch.openxlab.org.cn/
• GitHub Repository: https://github.com/InternLM/mindsearch
• arXiv Technical Paper: https://arxiv.org/abs/2407.20183

Application Scenarios

• Academic Research: Researchers can use MindSearch to quickly gather and organize extensive literature to support their studies.
• Market Analysis: Businesses can use MindSearch to collect market data, analyze competitor information, and monitor industry trends.
• News Reporting: Journalists can use MindSearch to gather background information on news events and quickly draft reports.
• Legal Research: Legal professionals can use MindSearch to collect relevant legal provisions, cases, and precedents to aid in legal analysis and case preparation.
• Technical Support: Technical support teams can use MindSearch to quickly find solutions and steps to resolve technical issues.