增加Kimi多模态Api,这样可以灵活选择！

exp_multimodal/VLM_PROVIDER_README.md

@@ -0,0 +1,196 @@
+# VLM Provider 架构说明
+
+## 📖 概述
+
+本项目支持多种视觉语言模型（VLM）后端，通过统一的 Provider 接口实现：
+- **Ollama Provider**: 本地/自建 Ollama 服务
+- **Kimi Provider**: Moonshot AI 的 Kimi 1.5 厂商 API
+
+## 🏗️ 架构设计
+
+```
+exp_multimodal/
+├── vlm_providers/
+│   ├── __init__.py           # 导出接口
+│   ├── base.py               # VLMProvider 抽象基类
+│   ├── ollama_provider.py    # Ollama 实现
+│   └── kimi_provider.py      # Kimi 实现
+├── vlm_classifier.py         # 使用 provider 进行分类
+└── exp_multimodal_gui.py     # GUI 界面（支持 provider 切换）
+```
+
+### 抽象接口
+
+所有 Provider 都继承 `VLMProvider` 基类并实现 `chat_vision` 方法：
+
+```python
+from exp_multimodal.vlm_providers import VLMProvider
+
+class MyCustomProvider(VLMProvider):
+    def chat_vision(self, prompt, image_paths, temperature, timeout_sec):
+        # 调用自定义 API
+        ...
+        return response_text
+```
+
+## 🚀 使用方法
+
+### 1. GUI 界面使用
+
+1. 启动 GUI：
+   ```bash
+   python -m exp_multimodal.exp_multimodal_gui
+   ```
+
+2. 在"配置与运行"标签页选择 Provider：
+   - **Ollama**: 配置 `OLLAMA_URL` 和 `MODEL`
+   - **Kimi**: 配置 `API_KEY`、`BASE_URL`、`MODEL`
+
+3. 点击"保存当前配置"自动保存到 `vlm_config.json`，下次启动自动加载
+
+### 2. 代码调用
+
+```python
+from exp_multimodal.vlm_providers import OllamaProvider, KimiProvider
+from exp_multimodal.vlm_classifier import classify_image
+
+# 方式1: 使用 Ollama
+provider = OllamaProvider(
+    ollama_url="http://192.168.1.250:11434",
+    model="qwen2.5vl:32b"
+)
+
+# 方式2: 使用 Kimi
+provider = KimiProvider(
+    api_key="sk-xxx",
+    base_url="https://api.moonshot.cn/v1",
+    model="moonshot-v1-32k-vision-preview"
+)
+
+# 进行分类
+result = classify_image(
+    image_path="test.jpg",
+    labels=["宫保鸡丁", "麻婆豆腐", "红烧肉"],
+    provider=provider,
+    fewshot_hints=None,
+    ingredient_only=False
+)
+
+print(result)  # {'label': '宫保鸡丁', 'confidence': 0.95}
+```
+
+## 🔧 配置说明
+
+### Ollama Provider
+
+```python
+OllamaProvider(
+    ollama_url="http://192.168.1.250:11434",  # Ollama 服务地址
+    model="qwen2.5vl:32b"                      # 模型名称
+)
+```
+
+### Kimi Provider
+
+```python
+KimiProvider(
+    api_key="sk-JFxKmZnkkd1krxK4WsivcjdaAX36j0lu8tBdq3kL9OwHkdin",  # 必填
+    base_url="https://api.moonshot.cn/v1",                          # 可选
+    model="moonshot-v1-32k-vision-preview"                          # 可选
+)
+```
+
+**获取 Kimi API Key**:
+1. 访问 [Moonshot AI 开放平台](https://platform.moonshot.cn/)
+2. 注册/登录账号
+3. 在"API 密钥"页面创建新密钥
+
+## 📦 依赖安装
+
+```bash
+# 基础依赖（Ollama）
+pip install -r requirements.txt
+
+# Kimi Provider 额外依赖
+pip install openai>=1.0.0
+```
+
+## 🎯 扩展新 Provider
+
+假设要添加 OpenAI GPT-4V：
+
+1. 创建 `vlm_providers/openai_provider.py`:
+
+```python
+from .base import VLMProvider
+from openai import OpenAI
+
+class OpenAIProvider(VLMProvider):
+    def __init__(self, api_key, model="gpt-4-vision-preview"):
+        self.client = OpenAI(api_key=api_key)
+        self.model = model
+    
+    def chat_vision(self, prompt, image_paths, temperature, timeout_sec):
+        # 实现调用逻辑
+        ...
+        return response.choices[0].message.content
+```
+
+2. 在 `vlm_providers/__init__.py` 中导出：
+
+```python
+from .openai_provider import OpenAIProvider
+__all__ = ["VLMProvider", "OllamaProvider", "KimiProvider", "OpenAIProvider"]
+```
+
+3. 在 GUI 中添加新选项（`exp_multimodal_gui.py`）
+
+## 📊 性能对比
+
+| Provider | 优点 | 缺点 | 推荐场景 |
+|----------|------|------|----------|
+| **Ollama** | 本地部署，数据安全，无API费用 | 需要GPU资源，推理速度受硬件限制 | 对数据隐私要求高，有本地GPU |
+| **Kimi** | 云端推理快，无需本地资源，遵循指令能力强 | 需要付费，依赖网络 | 快速验证，大规模识别 |
+
+## ⚠️ 注意事项
+
+1. **Kimi API Key 保密**: 不要将 API Key 提交到版本控制
+2. **配置文件**: `vlm_config.json` 已加入 `.gitignore`（建议）
+3. **超时设置**: 菜品数量超过 1 万时，建议增大 `timeout_sec`
+4. **Prompt 长度**: Kimi 支持 32k token，Ollama 根据模型而定
+
+## 🐛 故障排查
+
+### Ollama 连接失败
+```
+[Ollama][Error] Connection refused
+```
+- 检查 `OLLAMA_URL` 是否正确
+- 确认 Ollama 服务是否启动
+- 检查防火墙设置
+
+### Kimi API 错误
+```
+[Kimi][Error] 401 Unauthorized
+```
+- 检查 API Key 是否正确
+- 确认账户余额是否充足
+- 查看 [Kimi API 文档](https://platform.moonshot.cn/docs)
+
+### 依赖缺失
+```
+ImportError: No module named 'openai'
+```
+- 安装缺失依赖：`pip install openai>=1.0.0`
+
+## 📝 更新日志
+
+- **2025-11-10**: 初始版本，支持 Ollama 和 Kimi Provider
+- 未来计划：支持 OpenAI、Claude、Gemini 等更多厂商
+
+## 📞 支持
+
+如有问题，请查看：
+- Ollama 文档: https://ollama.ai/docs
+- Kimi API 文档: https://platform.moonshot.cn/docs
+- 项目 Issue: (您的仓库链接)

exp_multimodal/exp_multimodal_gui.py

@@ -15,7 +15,7 @@ from tkinterdnd2 import DND_FILES, TkinterDnD
 from exp_multimodal.labels import build_labels, _normalize, _base_ingredient
 from exp_multimodal.vlm_classifier import classify_image
 from exp_multimodal.ollama_client import OLLAMA_URL as DEFAULT_OLLAMA_URL, DEFAULT_MODEL as DEFAULT_VLM_MODEL
-from exp_multimodal import ollama_client as _ollama_mod
+from exp_multimodal.vlm_providers import VLMProvider, OllamaProvider, KimiProvider
 
 
 ctk.set_appearance_mode("System")
@@ -38,8 +38,15 @@ class MultiModalFoodApp:
 
         # 配置
         self.mode_var = ctk.StringVar(value="dish")  # dish | whole | processed
+        
+        # VLM Provider 配置
+        self.provider_var = ctk.StringVar(value="ollama")  # ollama | kimi
         self.ollama_url_var = ctk.StringVar(value=DEFAULT_OLLAMA_URL)
         self.vlm_model_var = ctk.StringVar(value=DEFAULT_VLM_MODEL)
+        self.kimi_api_key_var = ctk.StringVar(value="")
+        self.kimi_base_url_var = ctk.StringVar(value="https://api.moonshot.cn/v1")
+        self.kimi_model_var = ctk.StringVar(value="moonshot-v1-32k-vision-preview")
+        
         self.alias_map_path: Optional[str] = None
 
         # Fewshot 可视化编辑（C 方案）: dict[label] = hint
@@ -59,6 +66,9 @@ class MultiModalFoodApp:
         # UI
         self.create_widgets()
         
+        # 加载保存的配置
+        self.load_config()
+
     # -------------------- 图像加载工具 --------------------
     def load_image_with_chinese_path(self, file_path: str):
         try:
@@ -323,14 +333,58 @@ class MultiModalFoodApp:
             rb = ctk.CTkRadioButton(mode_frame, text=text, variable=self.mode_var, value=val)
             rb.pack(side="left", padx=8, pady=8)
 
-        # 服务配置
+        # VLM 服务提供商选择
-        ctk.CTkLabel(panel, text="VLM 服务配置", font=("Arial", 14, "bold")).pack(anchor="w", pady=(10, 6))
+        ctk.CTkLabel(panel, text="VLM 服务提供商", font=("Arial", 14, "bold")).pack(anchor="w", pady=(10, 6))
-        svc = ctk.CTkFrame(panel)
+        provider_frame = ctk.CTkFrame(panel)
-        svc.pack(fill="x", pady=(0, 10))
+        provider_frame.pack(fill="x", pady=(0, 10))
-        ctk.CTkLabel(svc, text="OLLAMA_URL:").pack(side="left", padx=6, pady=8)
+        
-        ctk.CTkEntry(svc, textvariable=self.ollama_url_var, width=320).pack(side="left", padx=4, pady=8)
+        for val, text in [("ollama", "Ollama (本地/自建)"), ("kimi", "Kimi 1.5 (厂商API)")]:
-        ctk.CTkLabel(svc, text="VLM_MODEL:").pack(side="left", padx=12, pady=8)
+            rb = ctk.CTkRadioButton(
-        ctk.CTkEntry(svc, textvariable=self.vlm_model_var, width=220).pack(side="left", padx=4, pady=8)
+                provider_frame, 
+                text=text, 
+                variable=self.provider_var, 
+                value=val,
+                command=self.on_provider_change
+            )
+            rb.pack(side="left", padx=8, pady=8)
+
+        # Ollama 配置区域
+        self.ollama_config_frame = ctk.CTkFrame(panel)
+        self.ollama_config_frame.pack(fill="x", pady=(0, 10))
+        ctk.CTkLabel(self.ollama_config_frame, text="Ollama 配置", font=("Arial", 12, "bold")).pack(anchor="w", pady=(4, 4))
+        
+        ollama_row1 = ctk.CTkFrame(self.ollama_config_frame)
+        ollama_row1.pack(fill="x", pady=(0, 4))
+        ctk.CTkLabel(ollama_row1, text="OLLAMA_URL:").pack(side="left", padx=6, pady=4)
+        ctk.CTkEntry(ollama_row1, textvariable=self.ollama_url_var, width=420).pack(side="left", padx=4, pady=4)
+        
+        ollama_row2 = ctk.CTkFrame(self.ollama_config_frame)
+        ollama_row2.pack(fill="x", pady=(0, 4))
+        ctk.CTkLabel(ollama_row2, text="MODEL:").pack(side="left", padx=6, pady=4)
+        ctk.CTkEntry(ollama_row2, textvariable=self.vlm_model_var, width=420).pack(side="left", padx=4, pady=4)
+
+        # Kimi 配置区域
+        self.kimi_config_frame = ctk.CTkFrame(panel)
+        self.kimi_config_frame.pack(fill="x", pady=(0, 10))
+        ctk.CTkLabel(self.kimi_config_frame, text="Kimi 配置", font=("Arial", 12, "bold")).pack(anchor="w", pady=(4, 4))
+        
+        kimi_row1 = ctk.CTkFrame(self.kimi_config_frame)
+        kimi_row1.pack(fill="x", pady=(0, 4))
+        ctk.CTkLabel(kimi_row1, text="API_KEY:").pack(side="left", padx=6, pady=4)
+        ctk.CTkEntry(kimi_row1, textvariable=self.kimi_api_key_var, width=420, show="*").pack(side="left", padx=4, pady=4)
+        
+        kimi_row2 = ctk.CTkFrame(self.kimi_config_frame)
+        kimi_row2.pack(fill="x", pady=(0, 4))
+        ctk.CTkLabel(kimi_row2, text="BASE_URL:").pack(side="left", padx=6, pady=4)
+        ctk.CTkEntry(kimi_row2, textvariable=self.kimi_base_url_var, width=420).pack(side="left", padx=4, pady=4)
+        
+        kimi_row3 = ctk.CTkFrame(self.kimi_config_frame)
+        kimi_row3.pack(fill="x", pady=(0, 4))
+        ctk.CTkLabel(kimi_row3, text="MODEL:").pack(side="left", padx=6, pady=4)
+        ctk.CTkEntry(kimi_row3, textvariable=self.kimi_model_var, width=420).pack(side="left", padx=4, pady=4)
+
+        # 初始化显示状态
+        self.on_provider_change()
 
         # Alias Map
         alias = ctk.CTkFrame(panel)
@@ -385,6 +439,16 @@ class MultiModalFoodApp:
         self.refresh_extra_labels_view()
         self.refresh_fewshot_view()
 
+        # 配置保存/加载
+        cfg_manage = ctk.CTkFrame(panel)
+        cfg_manage.pack(fill="x", pady=(10, 10))
+        ctk.CTkLabel(cfg_manage, text="配置管理", font=("Arial", 14, "bold")).pack(anchor="w", pady=(0, 6))
+        cfg_row = ctk.CTkFrame(cfg_manage)
+        cfg_row.pack(fill="x")
+        ctk.CTkButton(cfg_row, text="保存当前配置", command=self.save_config, width=140, fg_color="blue", hover_color="darkblue").pack(side="left", padx=6)
+        ctk.CTkButton(cfg_row, text="加载配置", command=self.load_config_from_file, width=140).pack(side="left", padx=6)
+        ctk.CTkLabel(cfg_row, text="(自动保存到 config.json)", font=("Arial", 10), text_color="gray").pack(side="left", padx=12)
+
     def pick_alias_file(self):
         path = filedialog.askopenfilename(title="选择 alias_map.json", filetypes=[("JSON 文件", "*.json"), ("所有文件", "*.*")])
         if path:
@@ -395,6 +459,16 @@ class MultiModalFoodApp:
         self.alias_map_path = None
         self.alias_label_var.set("未选择")
 
+    def on_provider_change(self):
+        """切换 Provider 时显示/隐藏对应的配置区域"""
+        provider = self.provider_var.get()
+        if provider == "ollama":
+            self.ollama_config_frame.pack(fill="x", pady=(0, 10), before=self.kimi_config_frame)
+            self.kimi_config_frame.pack_forget()
+        elif provider == "kimi":
+            self.kimi_config_frame.pack(fill="x", pady=(0, 10), before=self.ollama_config_frame)
+            self.ollama_config_frame.pack_forget()
+
     # 额外标签编辑
     def refresh_extra_labels_view(self):
         self.extra_labels_listbox.configure(state="normal")
@@ -562,12 +636,13 @@ class MultiModalFoodApp:
             labels = self._build_final_labels(mode)
             ingredient_only = (mode in {"whole", "processed"})
             use_hints = self.fewshot_hints if self.fewshot_enabled_var.get() else None
-            # 环境覆盖（仅对运行时生效，不修改模块常量）
+
-            os.environ["OLLAMA_URL"] = self.ollama_url_var.get().strip() or DEFAULT_OLLAMA_URL
+            # 创建 VLM Provider
-            os.environ["VLM_MODEL"] = self.vlm_model_var.get().strip() or DEFAULT_VLM_MODEL
+            provider = self._create_vlm_provider()
-            # 同时覆盖已导入模块中的运行时常量，确保 classify 调用使用最新配置
+            if provider is None:
-            _ollama_mod.OLLAMA_URL = os.environ["OLLAMA_URL"]
+                self.root.after(0, lambda: messagebox.showerror("错误", "Provider 配置错误，请检查配置"))
-            _ollama_mod.DEFAULT_MODEL = os.environ["VLM_MODEL"]
+                self.root.after(0, self.recognition_completed)
+                return
 
             self.current_results.clear()
             for i, img in enumerate(self.uploaded_images):
@@ -575,12 +650,13 @@ class MultiModalFoodApp:
                     result = classify_image(
                         image_path=img['path'],
                         labels=labels,
+                        provider=provider,
                         fewshot_hints=use_hints,
                         ingredient_only=ingredient_only,
                     )
                     pred = result.get("label", "Unknown")
                     conf = result.get("confidence", 0.0)
-                    # 若非数值，后续展示为 N/A
+                    # 若非数值,后续展示为 N/A
                     if not isinstance(conf, (int, float)):
                         try:
                             conf = float(conf)
@@ -619,6 +695,33 @@ class MultiModalFoodApp:
             self.root.after(0, lambda: messagebox.showerror("错误", f"识别过程中出错: {e}"))
             self.root.after(0, self.recognition_completed)
 
+    def _create_vlm_provider(self) -> Optional[VLMProvider]:
+        """根据配置创建 VLM Provider"""
+        provider_type = self.provider_var.get()
+        
+        try:
+            if provider_type == "ollama":
+                url = self.ollama_url_var.get().strip() or DEFAULT_OLLAMA_URL
+                model = self.vlm_model_var.get().strip() or DEFAULT_VLM_MODEL
+                return OllamaProvider(ollama_url=url, model=model)
+            
+            elif provider_type == "kimi":
+                api_key = self.kimi_api_key_var.get().strip()
+                if not api_key:
+                    messagebox.showerror("错误", "Kimi API Key 不能为空")
+                    return None
+                base_url = self.kimi_base_url_var.get().strip() or "https://api.moonshot.cn/v1"
+                model = self.kimi_model_var.get().strip() or "moonshot-v1-32k-vision-preview"
+                return KimiProvider(api_key=api_key, base_url=base_url, model=model)
+            
+            else:
+                messagebox.showerror("错误", f"未知的 Provider 类型: {provider_type}")
+                return None
+                
+        except Exception as e:
+            messagebox.showerror("错误", f"创建 Provider 失败: {e}")
+            return None
+
     def update_progress(self, current: int, total: int):
         self.recognize_button.configure(text=f"识别中... ({current}/{total})")
         self.update_images_display()
@@ -631,6 +734,142 @@ class MultiModalFoodApp:
         self.update_stats()
         messagebox.showinfo("完成", f"所有图片识别完成！耗时: {self.recognition_duration:.2f}秒")
 
+    # -------------------- 配置管理 --------------------
+    def get_config_path(self) -> str:
+        """获取配置文件路径（项目根目录下的 config.json）"""
+        return os.path.join(os.path.dirname(os.path.dirname(__file__)), "vlm_config.json")
+
+    def save_config(self):
+        """保存当前配置到 JSON 文件"""
+        config = {
+            "mode": self.mode_var.get(),
+            "provider": self.provider_var.get(),
+            "ollama": {
+                "url": self.ollama_url_var.get(),
+                "model": self.vlm_model_var.get(),
+            },
+            "kimi": {
+                "api_key": self.kimi_api_key_var.get(),
+                "base_url": self.kimi_base_url_var.get(),
+                "model": self.kimi_model_var.get(),
+            },
+            "alias_map_path": self.alias_map_path,
+            "fewshot_enabled": self.fewshot_enabled_var.get(),
+            "fewshot_hints": self.fewshot_hints,
+            "fewshot_file_path": self.fewshot_file_path,
+            "extra_labels": self.extra_labels,
+            "extra_labels_file_path": self.extra_labels_file_path,
+        }
+        
+        try:
+            config_path = self.get_config_path()
+            with open(config_path, "w", encoding="utf-8") as f:
+                json.dump(config, f, ensure_ascii=False, indent=2)
+            messagebox.showinfo("成功", f"配置已保存到 {os.path.basename(config_path)}")
+        except Exception as e:
+            messagebox.showerror("错误", f"保存配置失败: {e}")
+
+    def load_config(self):
+        """自动加载配置（启动时调用）"""
+        config_path = self.get_config_path()
+        if not os.path.exists(config_path):
+            return
+        
+        try:
+            with open(config_path, "r", encoding="utf-8") as f:
+                config = json.load(f)
+            
+            # 恢复配置
+            if "mode" in config:
+                self.mode_var.set(config["mode"])
+            if "provider" in config:
+                self.provider_var.set(config["provider"])
+            
+            if "ollama" in config:
+                self.ollama_url_var.set(config["ollama"].get("url", DEFAULT_OLLAMA_URL))
+                self.vlm_model_var.set(config["ollama"].get("model", DEFAULT_VLM_MODEL))
+            
+            if "kimi" in config:
+                self.kimi_api_key_var.set(config["kimi"].get("api_key", ""))
+                self.kimi_base_url_var.set(config["kimi"].get("base_url", "https://api.moonshot.cn/v1"))
+                self.kimi_model_var.set(config["kimi"].get("model", "moonshot-v1-32k-vision-preview"))
+            
+            if "alias_map_path" in config and config["alias_map_path"]:
+                self.alias_map_path = config["alias_map_path"]
+                self.alias_label_var.set(os.path.basename(self.alias_map_path))
+            
+            if "fewshot_enabled" in config:
+                self.fewshot_enabled_var.set(config["fewshot_enabled"])
+            if "fewshot_hints" in config:
+                self.fewshot_hints = config["fewshot_hints"]
+            if "fewshot_file_path" in config:
+                self.fewshot_file_path = config["fewshot_file_path"]
+            
+            if "extra_labels" in config:
+                self.extra_labels = config["extra_labels"]
+            if "extra_labels_file_path" in config:
+                self.extra_labels_file_path = config["extra_labels_file_path"]
+            
+            self.refresh_extra_labels_view()
+            self.refresh_fewshot_view()
+            self.on_provider_change()
+            
+            print(f"[Config] 配置已从 {config_path} 加载")
+        except Exception as e:
+            print(f"[Config] 加载配置失败: {e}")
+
+    def load_config_from_file(self):
+        """从用户选择的文件加载配置"""
+        path = filedialog.askopenfilename(
+            title="选择配置文件",
+            filetypes=[("JSON 文件", "*.json"), ("所有文件", "*.*")]
+        )
+        if not path:
+            return
+        
+        try:
+            with open(path, "r", encoding="utf-8") as f:
+                config = json.load(f)
+            
+            # 恢复配置（同上）
+            if "mode" in config:
+                self.mode_var.set(config["mode"])
+            if "provider" in config:
+                self.provider_var.set(config["provider"])
+            
+            if "ollama" in config:
+                self.ollama_url_var.set(config["ollama"].get("url", DEFAULT_OLLAMA_URL))
+                self.vlm_model_var.set(config["ollama"].get("model", DEFAULT_VLM_MODEL))
+            
+            if "kimi" in config:
+                self.kimi_api_key_var.set(config["kimi"].get("api_key", ""))
+                self.kimi_base_url_var.set(config["kimi"].get("base_url", "https://api.moonshot.cn/v1"))
+                self.kimi_model_var.set(config["kimi"].get("model", "moonshot-v1-32k-vision-preview"))
+            
+            if "alias_map_path" in config and config["alias_map_path"]:
+                self.alias_map_path = config["alias_map_path"]
+                self.alias_label_var.set(os.path.basename(self.alias_map_path))
+            
+            if "fewshot_enabled" in config:
+                self.fewshot_enabled_var.set(config["fewshot_enabled"])
+            if "fewshot_hints" in config:
+                self.fewshot_hints = config["fewshot_hints"]
+            if "fewshot_file_path" in config:
+                self.fewshot_file_path = config["fewshot_file_path"]
+            
+            if "extra_labels" in config:
+                self.extra_labels = config["extra_labels"]
+            if "extra_labels_file_path" in config:
+                self.extra_labels_file_path = config["extra_labels_file_path"]
+            
+            self.refresh_extra_labels_view()
+            self.refresh_fewshot_view()
+            self.on_provider_change()
+            
+            messagebox.showinfo("成功", f"配置已从 {os.path.basename(path)} 加载")
+        except Exception as e:
+            messagebox.showerror("错误", f"加载配置失败: {e}")
+
 
 def main():
     root = TkinterDnD.Tk()  # 必须使用 TkinterDnD.Tk 以支持拖拽

exp_multimodal/vlm_classifier.py

@@ -2,7 +2,7 @@ import json
 import re
 from typing import Dict, List, Optional
 
-from .ollama_client import chat_vision
+from .vlm_providers.base import VLMProvider
 from .prompts import build_closedset_prompt
 
 
@@ -37,6 +37,7 @@ def _extract_json_obj(text: str):
 def classify_image(
     image_path: str,
     labels: List[str],
+    provider: VLMProvider,
     fewshot_hints: Optional[Dict[str, str]] = None,
     ingredient_only: bool = False,
 ) -> Dict:
@@ -46,7 +47,7 @@ def classify_image(
 
     print(
         f"[VLM] Start classify image={image_path} labels={len(labels)} "
-        f"ingredient_only={ingredient_only} fewshot={bool(fewshot_hints)}"
+        f"ingredient_only={ingredient_only} fewshot={bool(fewshot_hints)} provider={provider}"
     )
     prompt = build_closedset_prompt(labels, fewshot_hints, ingredient_only)
     # _preview_prompt = (prompt[:120]).replace("\n", " ")
@@ -54,8 +55,8 @@ def classify_image(
     # print(f"[VLM] Prompt length={len(prompt)} preview={_preview_prompt}...")
     print(f"[VLM] Prompt length={len(prompt)} preview={final_prompt}")
 
-    print("[VLM] Calling chat_vision...")
+    print(f"[VLM] Calling {provider}.chat_vision()...")
-    text = chat_vision(prompt, [image_path], temperature=0.1)
+    text = provider.chat_vision(prompt, [image_path], temperature=0.1)
     _preview_text = (str(text)[:200]).replace("\n", " ")
     print(f"[VLM] Received text length={len(str(text))} preview={_preview_text}...")
 

exp_multimodal/vlm_providers/__init__.py

@@ -0,0 +1,10 @@
+"""
+VLM Provider 抽象层
+支持多种视觉语言模型后端：Ollama、Kimi、OpenAI GPT-4V 等
+"""
+
+from .base import VLMProvider
+from .ollama_provider import OllamaProvider
+from .kimi_provider import KimiProvider
+
+__all__ = ["VLMProvider", "OllamaProvider", "KimiProvider"]

exp_multimodal/vlm_providers/base.py

@@ -0,0 +1,41 @@
+"""
+VLM Provider 抽象基类
+"""
+
+from abc import ABC, abstractmethod
+from typing import List
+
+
+class VLMProvider(ABC):
+    """
+    视觉语言模型 Provider 抽象接口
+    所有 VLM 后端实现都应该继承此类并实现 chat_vision 方法
+    """
+
+    @abstractmethod
+    def chat_vision(
+        self,
+        prompt: str,
+        image_paths: List[str],
+        temperature: float = 0.1,
+        timeout_sec: int = 120,
+    ) -> str:
+        """
+        调用 VLM 进行图像理解
+
+        Args:
+            prompt: 用户提示词
+            image_paths: 图片路径列表（支持单张或多张）
+            temperature: 采样温度（0.0-1.0）
+            timeout_sec: 超时时间（秒）
+
+        Returns:
+            模型生成的文本响应
+
+        Raises:
+            Exception: 调用失败时抛出异常
+        """
+        pass
+
+    def __repr__(self):
+        return f"<{self.__class__.__name__}>"

exp_multimodal/vlm_providers/kimi_provider.py

@@ -0,0 +1,123 @@
+"""
+Kimi VLM Provider
+使用 Moonshot AI 的 Kimi 1.5 视觉模型
+"""
+
+import base64
+import time
+from typing import List
+
+from .base import VLMProvider
+
+
+class KimiProvider(VLMProvider):
+    """
+    Kimi (Moonshot AI) 视觉模型提供商
+    使用 OpenAI SDK 兼容接口
+    """
+
+    def __init__(
+        self,
+        api_key: str,
+        base_url: str = "https://api.moonshot.cn/v1",
+        model: str = "moonshot-v1-32k-vision-preview",
+    ):
+        """
+        Args:
+            api_key: Moonshot AI API Key
+            base_url: API 端点地址
+            model: 模型名称
+        """
+        if not api_key:
+            raise ValueError("Kimi API Key 不能为空")
+
+        self.api_key = api_key
+        self.base_url = base_url
+        self.model = model
+
+    def chat_vision(
+        self,
+        prompt: str,
+        image_paths: List[str],
+        temperature: float = 0.1,
+        timeout_sec: int = 120,
+    ) -> str:
+        """调用 Kimi API 进行视觉推理"""
+        try:
+            from openai import OpenAI
+        except ImportError:
+            raise ImportError(
+                "Kimi Provider 需要 openai 库，请安装: pip install openai"
+            )
+
+        print(
+            f"[Kimi] Prepare chat model={self.model} temp={temperature} "
+            f"images={len(image_paths)} timeout={timeout_sec}s"
+        )
+
+        # 初始化 OpenAI 客户端
+        client = OpenAI(
+            api_key=self.api_key,
+            base_url=self.base_url,
+            timeout=timeout_sec,
+        )
+
+        # 构建消息内容（文本 + 图片）
+        # Kimi 支持 image_url 格式，使用 base64 编码
+        content_parts = [{"type": "text", "text": prompt}]
+        
+        for img_path in image_paths:
+            try:
+                with open(img_path, "rb") as f:
+                    img_data = f.read()
+                b64_str = base64.b64encode(img_data).decode("utf-8")
+                
+                # 根据文件扩展名判断 MIME 类型
+                if img_path.lower().endswith((".jpg", ".jpeg")):
+                    mime = "image/jpeg"
+                elif img_path.lower().endswith(".png"):
+                    mime = "image/png"
+                else:
+                    mime = "image/jpeg"  # 默认
+                
+                data_uri = f"data:{mime};base64,{b64_str}"
+                content_parts.append({
+                    "type": "image_url",
+                    "image_url": {"url": data_uri}
+                })
+            except Exception as e:
+                print(f"[Kimi] Warning: Failed to load image {img_path}: {e}")
+                continue
+
+        # 调用 API
+        t0 = time.time()
+        try:
+            completion = client.chat.completions.create(
+                model=self.model,
+                messages=[
+                    {
+                        "role": "user",
+                        "content": content_parts,
+                    }
+                ],
+                temperature=temperature,
+            )
+
+            dt = time.time() - t0
+            text = completion.choices[0].message.content or ""
+            
+            result = text.replace("\n", " ")
+            print(f"[Kimi] Response elapsed={dt:.2f}s length={len(text)} preview={result}")
+
+            if not text:
+                raise RuntimeError("Kimi 返回内容为空")
+
+            return text
+
+        except Exception as e:
+            dt = time.time() - t0
+            print(f"[Kimi][Error] during request elapsed={dt:.2f}s: {e}")
+            raise
+
+    def __repr__(self):
+        return f"<KimiProvider model={self.model} base_url={self.base_url}>"

exp_multimodal/vlm_providers/ollama_provider.py

@@ -0,0 +1,61 @@
+"""
+Ollama VLM Provider
+支持本地/自建 Ollama 服务
+"""
+
+import os
+from typing import List
+
+from .base import VLMProvider
+
+
+class OllamaProvider(VLMProvider):
+    """
+    Ollama 本地/自建服务提供商
+    使用 OpenAI-compatible API (/v1/chat/completions)
+    """
+
+    def __init__(
+        self,
+        ollama_url: str = None,
+        model: str = None,
+    ):
+        """
+        Args:
+            ollama_url: Ollama 服务地址，默认从环境变量 OLLAMA_URL 读取
+            model: 模型名称，默认从环境变量 VLM_MODEL 读取
+        """
+        self.ollama_url = ollama_url or os.getenv("OLLAMA_URL", "http://192.168.1.250:11434")
+        self.model = model or os.getenv("VLM_MODEL", "qwen2.5vl:32b")
+
+    def chat_vision(
+        self,
+        prompt: str,
+        image_paths: List[str],
+        temperature: float = 0.1,
+        timeout_sec: int = 120,
+    ) -> str:
+        """调用 Ollama 服务进行视觉推理"""
+        # 导入原有的 ollama_client 逻辑（复用代码）
+        from .. import ollama_client
+
+        # 临时覆盖模块常量（确保使用实例配置）
+        original_url = ollama_client.OLLAMA_URL
+        original_model = ollama_client.DEFAULT_MODEL
+        try:
+            ollama_client.OLLAMA_URL = self.ollama_url
+            ollama_client.DEFAULT_MODEL = self.model
+            return ollama_client.chat_vision(
+                prompt=prompt,
+                image_paths=image_paths,
+                temperature=temperature,
+                model=self.model,
+                timeout_sec=timeout_sec,
+            )
+        finally:
+            # 恢复原始值
+            ollama_client.OLLAMA_URL = original_url
+            ollama_client.DEFAULT_MODEL = original_model
+
+    def __repr__(self):
+        return f"<OllamaProvider url={self.ollama_url} model={self.model}>"

requirements.txt

@@ -26,3 +26,5 @@ torchvision==0.22.0
 tqdm==4.65.0
 typing_extensions==4.15.0
 faiss-cpu>=1.7.0
+requests>=2.28.0
+openai>=1.0.0