MikeLuu99/searxng-rust 来源：https://github.com/MikeLuu99/searxng-rust # metadata-search-engine-rs 一个用 Rust 编写的 SearXNG 风格元搜索引擎。将查询并发分发到多个搜索引擎，抓取它们的 HTML 结果，按规范化 URL 去重，并使用 Reciprocal Rank Fusion（RRF）排序。 ## 工作原理 1. 搜索请求到达 GET /search?q= 2. 查询通过 reqwest 并发发送到 DuckDuckGo、Brave、Startpage 和 Yahoo 3. 每个引擎使用 scraper（基于 Mozilla 的 html5ever 的 CSS 选择器）解析 HTML 响应 4. 结果通过规范化 URL 去重（剥离跟踪参数、移除区域前缀、对查询参数排序） 5. 重复的 URL 被合并并使用 RRF 评分（score = Σ 1/(60 + rank) 跨引擎累加）——被多个引擎返回的页面排名更高 6. 最终结果以 JSON 形式返回 ## 环境要求 - Rust 1.75+ - Cargo ## 安装 ### 作为库使用 bash cargo add metadata-search-engine-rs 或手动添加到 Cargo.toml： toml [dependencies] metadata-search-engine-rs = "0.1" ### 作为服务器（从源码） bash git clone https://github.com/MikeLuu99/searxng-rust cd metadata-search-engine-rs cargo build --release ## 运行 bash cargo run bash PORT=8080 MAX_RESULTS=20 cargo run --release 启用调试日志： bash RUST_LOG=debug cargo run ## 示例 添加到你的 Cargo.toml： toml [dependencies] metadata-search-engine-rs = "0.1" tokio = { version = "1", features = ["full"] } ### 查询单个引擎 rust use std::sync::Arc; use metadata_search_engine_rs::engines::{DuckDuckGoEngine, SearchEngine, build_http_client}; #[tokio::main] async fn main() -> anyhow::Result<()> { let client = Arc::new(build_http_client()?); let engine = DuckDuckGoEngine { client }; let results = engine.search("rust programming", 5).await?; for r in results { println!("{}\n {}", r.title, r.url); } Ok(()) } ### 分发到所有引擎并获取 RRF 排序结果 rust use std::sync::Arc; use metadata_search_engine_rs::{ aggregator::{aggregate, query_all_engines}, engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, StartpageEngine, YahooEngine, build_http_client}, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let client = Arc::new(build_http_client()?); let engines: Vec> = vec![ Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }), Arc::new(BraveEngine { client: Arc::clone(&client) }), Arc::new(StartpageEngine { client: Arc::clone(&client) }), Arc::new(YahooEngine { client: Arc::clone(&client) }), ]; let (successes, failures) = query_all_engines(&engines, "rust programming", 10).await; for (name, err) in &failures { eprintln!("engine {name} failed: {err}"); } let results = aggregate(successes, 10); for r in &results { println!("[{:.3}] ({}) {}", r.score, r.engines.join(", "), r.title); println!(" {}", r.url); } Ok(()) } ### 只使用特定引擎 rust use std::sync::Arc; use metadata_search_engine_rs::{ aggregator::{aggregate, query_all_engines}, engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, build_http_client}, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let client = Arc::new(build_http_client()?); let engines: Vec> = vec![ Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }), Arc::new(BraveEngine { client: Arc::clone(&client) }), ]; let (successes, _) = query_all_engines(&engines, "tokio async rust", 5).await; for r in aggregate(successes, 5) { println!("{} — {}", r.title, r.url); if let Some(snippet) = r.snippet { println!(" {snippet}"); } } Ok(()) } ## API ### GET /health bash curl http://localhost:3000/health json { "status": "ok" } ### GET /search?q= bash curl "http://localhost:3000/search?q=rust" json { "query": "rust", "results": [ { "title": "Rust Programming Language", "url": "https://rust-lang.org/", "snippet": "A language empowering everyone to build reliable and efficient software.", "engines": ["duckduckgo", "brave", "startpage", "yahoo"], "score": 0.049 } ], "engines_queried": ["duckduckgo", "brave", "startpage", "yahoo"], "engines_failed": [] } 错误响应： | 情况 | 状态码 | 响应体 | | –––––––– | —— | ––––––––––––––––––––––––– | | 缺少 q | 400 | {"error": "query parameter 'q' is required"} | | q 为空 | 400 | {"error": "query parameter 'q' cannot be empty"} | | 所有引擎失败 | 503 | {"error": "all engines failed to respond"} | ## 测试 bash # 所有单元测试 cargo test # 指定模块 cargo test normalizer cargo test aggregator cargo test engines::duckduckgo cargo test engines::brave cargo test engines::startpage cargo test engines::yahoo cargo test server::handlers # 实时测试（访问真实搜索引擎——需要联网） cargo test -- --ignored test_live 实时测试标记为 #[ignore]，默认不会在 CI 中运行。请手动运行它们以验证 HTML 选择器仍然适用于真实网站。 ## 终端界面 一个基于 ratatui 的 TUI 可作为 crate 使用（https://crates.io/crates/search-tui），安装地址在此。通过 GitHub 访问代码（https://github.com/MikeLuu99/search-tui） stx TUI ## 添加新的搜索引擎 1. 创建 src/engines/<引擎名>.rs 2. 定义一个持有 Arc<reqwest::Client> 的结构体 3. 实现 SearchEngine trait： rust impl SearchEngine for MyEngine { fn name(&self) -> &'static str { "myengine" } fn search<'a>( &'a self, query: &'a str, max_results: usize, ) -> BoxFuture<'a, Result<Vec<SearchResult>, EngineError>> { Box::pin(async move { // 获取 HTML，用 scraper 解析，返回 Vec }) } } 将其添加到 engines/mod.rs 并在 main.rs 中接线。