Compare commits

2 Commits
develop ... dev

Author SHA1 Message Date
iHeyTang
86a1bff09b fix 2025-08-25 14:44:22 +08:00
iHeyTang
f41fd24316 refactor 2025-08-25 14:42:35 +08:00
17 changed files with 4355 additions and 943 deletions

6
.gitignore vendored
View File

@@ -1,4 +1,8 @@
model/*.pth
model/*.pt
!model/.gitkeep
.yaml
.yaml
__pycache__/
config.yaml

529
Readme.md
View File

@@ -1,213 +1,350 @@
# ComfyUI-CustomNode 项目文档
# ComfyUI CustomNode - 对象存储节点
## 一、项目概述
ComfyUI-CustomNode 是一个为 ComfyUI 定制的自定义节点集合项目提供了丰富的功能节点涵盖文本处理、图像和视频处理、LLM 调用、对象存储操作等多个领域,方便用户在 ComfyUI 中进行多样化的任务处理。
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code Quality](https://img.shields.io/badge/code%20quality-A-green.svg)](https://github.com/psf/black)
**本地使用请下载[config.yaml](https://console.cloud.tencent.com/cos/bucket?type=object&tab=objectDetail&bucket=sucai-1324682537&path=%252Fconfig.yaml&region=ap-shanghai)到custom_nodes/ComfyUI-CustomNode目录**
## 二、项目结构
```
ComfyUI-CustomNode/
├── download/ # 下载文件目录
├── ext/ # 扩展文件目录
│ ├── pyproject.toml # 项目构建和依赖配置
│ ├── comfyui_modal_deploy.py # Comfy GPU服务器部署脚本
│ ├── comfyui_modal_deploy_4ui.py # Comfy批量CPU服务器部署脚本
│ └── nodes_bfl.py # 用于替换comfy内置api节点的代码
├── model/ # 模型目录
├── nodes/ # 节点代码目录
│ ├── image_face_nodes.py # 面部处理节点
│ ├── image_gesture_nodes.py # 姿态处理节点
│ ├── image_modal_nodes.py # Modal在线图像节点
│ ├── image_nodes.py # 图像节点
│ ├── llm_nodes.py # LLM 调用节点
│ ├── object_storage_nodes.py # 对象存储节点
│ ├── text_nodes.py # 文本处理节点
│ ├── util_nodes.py # 工具节点
│ ├── video_lipsync_nodes.py # 视频口型同步节点
│ └── video_nodes.py # 视频处理节点
├── utils/ # 工具函数目录
│ ├── model_module.py # 模型定义
│ ├── model_utils.py # 模型工具
│ ├── face_occu_detect.py # 面部遮挡识别工具
│ └── task_table.py # Task表定义--与DB持久化节点合用
├── __init__.py # 定义节点映射
├── .gitignore # Git 忽略文件配置
├── install.sh # Linux 安装脚本
├── install.bat # Windows 安装脚本
├── embedded_install.bat # 嵌入式 Windows 安装脚本
├── Readme.md # 项目文档
└── requirements.txt # 项目依赖文件
```
ComfyUI自定义节点集合提供强大的云存储功能支持AWS S3和腾讯云COS等多种对象存储服务。
## 三、用户手册
## ✨ 特性
### 1. 环境准备
确保你已经安装了 Python 3.10 或更高版本。
- 🏗️ **架构优雅**:采用抽象工厂模式和策略模式,遵循开闭原则
- 🔌 **易于扩展**:支持多种云存储服务,可轻松添加新的存储提供者
- 🛡️ **错误处理**:完善的错误处理机制和重试策略
- 📊 **类型安全**:完整的类型注解和参数验证
- 🧪 **测试覆盖**:全面的单元测试和集成测试
- 📝 **文档完善**详细的API文档和使用指南
## 🚀 快速开始
### 安装
1. 将项目克隆到ComfyUI的自定义节点目录
```bash
source venv/bin/activate # 对于 Windows 用户使用 `venv\Scripts\activate`
cd ComfyUI/custom_nodes/
git clone https://github.com/your-repo/ComfyUI-CustomNode.git
```
2. 安装依赖:
```bash
cd ComfyUI-CustomNode
pip install -r requirements.txt
```
### 2. 运行项目
将项目文件夹放置在 ComfyUI 的 `custom_nodes` 目录下,启动 ComfyUI 即可在节点列表中看到自定义节点。
### 配置
### 3. 节点使用
在 ComfyUI 界面中,根据需要选择相应的节点,配置节点的输入参数,然后连接节点并运行工作流。
创建 `config.yaml` 文件并配置你的存储服务:
## 四、节点介绍
```yaml
# AWS S3 配置
aws_key_id: "your_aws_access_key"
aws_access_key: "your_aws_secret_key"
### 常用节点
- **LLMChatMultiModalImageTensor**:多模态 LLM 调用,传入图片张量。
- **输入**LLM 提供商、提示词、图片张量、温度、最大令牌数、超时时间
- **输出**LLM 输出结果
- **用途**:结合图片张量与 LLM 进行交互
- **JMCustom**即梦自定义Prompt生视频
- **输入**参考图、Prompt、视频长度
- **输出**:生成视频路径
- **用途**自定义Prompt视频时长生成视频
- **RandomLineSelector**:从多行文本中随机选择一行。
- **输入**:多行文本、随机种子
- **输出**:随机选择的一行文本
- **用途**:随机选取文本中的一行
- **LoadImg**:从网络/本地加载图片(网络图片优先)。
- **输入**:图片 URL、选择本地图片
- **输出**:图像张量
- **用途**:加载网络图片
- **S3UploadURL**:上传文件到 S3 并返回 URL。
- **输入**待上传文件、S3 存储桶信息
- **输出**:上传后的文件 URL
- **用途**:上传文件到 S3 并获取文件 URL
# 腾讯云 COS 配置
cos_secret_id: "your_cos_secret_id"
cos_secret_key: "your_cos_secret_key"
cos_region: "ap-beijing"
cos_sucai_bucket_name: "your-cos-bucket"
```
### 1. 文本处理节点
- **LoadTextCustom**:从本地文件读取文本(文件路径优先)。
- **输入**:文件路径、选择本地文件、编码格式
- **输出**:文本字符串
- **用途**:读取本地文本文件内容
- **StringEmptyJudgement**:判断字符串是否为空。
- **输入**:字符串
- **输出**:布尔值(是否为空)
- **用途**:检查字符串是否为空
- **RandomLineSelector**:从多行文本中随机选择一行。
- **输入**:多行文本、随机种子
- **输出**:随机选择的一行文本
- **用途**:随机选取文本中的一行
或者使用环境变量:
### 2. LLM 调用节点
- **LLMChat**:调用 LLM 进行聊天。
- **输入**LLM 提供商、提示词、温度、最大令牌数、超时时间
- **输出**LLM 输出结果
- **用途**:与 LLM 进行文本交互
- **LLMChatMultiModalImageUpload**:多模态 LLM 调用,上传图片。
- **输入**LLM 提供商、提示词、图片文件、温度、最大令牌数、超时时间
- **输出**LLM 输出结果
- **用途**:结合图片与 LLM 进行交互
- **LLMChatMultiModalImageTensor**:多模态 LLM 调用,传入图片张量。
- **输入**LLM 提供商、提示词、图片张量、温度、最大令牌数、超时时间
- **输出**LLM 输出结果
- **用途**:结合图片张量与 LLM 进行交互
- **Jinja2RenderTemplate**:使用 Jinja2 渲染 prompt 模板。
- **输入**:模板、键值映射字典
- **输出**:渲染后的字符串
- **用途**:根据提供的模板和变量,使用 Jinja2 引擎渲染出最终的字符串,常用于生成动态的 prompt
```bash
export aws_key_id="your_aws_access_key"
export aws_access_key="your_aws_secret_key"
export cos_secret_id="your_cos_secret_id"
export cos_secret_key="your_cos_secret_key"
export cos_region="ap-beijing"
```
### 3. 图像和视频处理节点
- **JMCustom**即梦自定义Prompt生视频
- **输入**参考图、Prompt、视频长度
- **输出**:生成视频路径
- **用途**自定义Prompt视频时长生成视频
- **LoadImg**:从网络/本地加载图片(网络图片优先)。
- **输入**:图片 URL、选择本地图片
- **输出**:图像张量
- **用途**:加载网络图片
- **FaceDetect**:面部检测。
- **输入**:待检测图像
- **输出**:检测结果
- **用途**:检测图像中的面部
- **FaceExtract**:面部提取。
- **输入**:包含面部的图像
- **输出**:提取的面部图像
- **用途**:从图像中提取面部
- **VideoCut**:视频剪裁。
- **输入**:视频文件、剪裁起始时间、剪裁结束时间
- **输出**:剪裁后的视频文件
- **用途**:对视频进行剪裁
- **VideoCutByFramePoint**:视频按精确帧位剪裁。
- **输入**:视频文件、起始帧、结束帧
- **输出**:剪裁后的视频文件
- **用途**:按精确帧位对视频进行剪裁
- **VideoChangeFPS**:视频转换帧率。
- **输入**:视频文件、目标帧率
- **输出**:转换帧率后的视频文件
- **用途**:改变视频的帧率
- **HeyGemF2F**HeyGem 嘴型同步。
- **输入**视频张量、音频数据、HeyGem 服务 URL、临时文件路径、是否为 Windows 系统
- **输出**:视频存储路径
- **用途**:实现视频的嘴型同步
- **HeyGemF2FFromFile**HeyGem 嘴型同步,直接读取文件。
- **输入**视频文件路径、音频文件路径、HeyGem 服务 URL、临时文件路径、是否为 Windows 系统
- **输出**:视频存储路径
- **用途**:实现视频的嘴型同步
- **ModalEditCustom**: 生修图-Gemini
- **输入**Prompt, temperature, topP, 端点
- **输出**:生/修完的图片
- **用途**使用Gemini2.0模型进行图像生成/修正
- **ModalMidJourneyGenerateImage**: 生修图-Midjourney
- **输入**Prompt, 提供商, 端点, 超时时间
- **输出**:生/修完的图片
- **用途**使用Midjourney模型进行图像生成/修正
- **ModalMidJourneyDescribeImage**: 反推生图关键词-Midjourney
- **输入**:图片, 端点
- **输出**生图Prompt词
- **用途**使用Midjourney模型进行生图Prompt反推
## 📖 使用指南
### 4. 对象存储节点
- **COSUpload**:上传文件到 COS。
- **输入**待上传文件、COS 存储桶信息
- **输出**上传文件key
- **用途**:将文件上传到腾讯云 COS
- **COSDownload**:从 COS 下载文件。
- **输入**COS 存储桶文件信息、本地保存路径
- **输出**:下载的文件路径
- **用途**:从腾讯云 COS 下载文件
- **S3Upload**:上传文件到 S3。
- **输入**待上传文件、S3 存储桶信息
- **输出**上传文件key
- **用途**:将文件上传到 S3 存储桶
- **S3Download**:从 S3 下载文件。
- **输入**S3 存储桶文件信息、本地保存路径
- **输出**:下载的文件路径
- **用途**:从 S3 存储桶下载文件
- **S3UploadURL**:上传文件到 S3 并返回 URL。
- **输入**待上传文件、S3 存储桶信息
- **输出**:上传后的文件 URL
- **用途**:上传文件到 S3 并获取文件 URL
### 可用节点
### 5. 工具节点
- **LogToDB**:将日志记录到数据库(数据库需要根据utils/task_table.py结构建表)。
- **输入**:作业 ID、日志内容、状态、数据库连接 URL
- **输出**:记录结果
- **用途**:将任务日志保存到数据库
- **VodToLocalNode**:从腾讯云 VOD 下载视频到本地。
- **输入**:文件 ID、子应用 ID
- **输出**:本地文件路径
- **用途**:从腾讯云 VOD 下载视频文件
- **UnloadAllModels**:卸载所有已加载的模型。
- **输入**:任意输入
- **输出**:无
- **用途**:释放模型占用的内存
- **TraverseFolder**:遍历文件夹。
- **输入**:文件夹路径、文件后缀、是否递归、索引
- **输出**:文件路径
- **用途**:遍历文件夹并获取指定后缀的文件
- **PlugAndPlayWebhook**:即插即用 Webhook 节点。
- **输入**Webhook URL、图片 URL、提示 ID
- **输出**:无
- **用途**:将数据转发到指定的 Webhook URL
- **TaskIdGenerate**:生成任务 ID。
- **输入**:无
- **输出**:任务 ID
- **用途**:生成唯一的任务 ID
#### COS 节点
- **COSDownload**: 从腾讯云COS下载文件
- **COSUpload**: 上传文件到腾讯云COS
## 五、注意事项
- 部分节点需要配置相应的 API 密钥或数据库连接信息,**请下载[config.yaml](https://console.cloud.tencent.com/cos/bucket?type=object&tab=objectDetail&bucket=sucai-1324682537&path=%252Fconfig.yaml&region=ap-shanghai)到custom_nodes/ComfyUI-CustomNode目录**。
- 运行项目时,请确保网络连接正常,特别是涉及在线文件读取和 LLM 调用的节点。
#### S3 节点
- **S3Download**: 从AWS S3下载文件
- **S3Upload**: 上传文件到AWS S3
- **S3UploadURL**: 上传文件并返回访问URL
- **S3UploadIMAGEURL**: 上传图像张量并返回URL
### 基本用法示例
#### 1. 上传文件到S3
```python
# 在ComfyUI工作流中使用S3Upload节点
# 输入:
# - s3_bucket: "my-bucket"
# - path: "/path/to/file.jpg"
# - subfolder: "images"
# 输出文件在S3中的键名
```
#### 2. 从COS下载文件
```python
# 使用COSDownload节点
# 输入:
# - cos_bucket: "my-cos-bucket"
# - cos_key: "images/photo.jpg"
# 输出:本地文件路径
```
#### 3. 上传图像并获取URL
```python
# 使用S3UploadIMAGEURL节点
# 输入:
# - image: [图像张量]
# - subfolder: "generated"
# 输出图像的CDN访问URL
```
## 🏗️ 架构设计
### 设计模式
本项目采用了多种设计模式来确保代码的可维护性和可扩展性:
#### 抽象工厂模式
```python
# 存储提供者工厂
class StorageFactory(ABC):
@abstractmethod
def create_provider(self, config: Dict[str, Any]) -> StorageProvider:
pass
# 具体工厂实现
class S3StorageFactory(StorageFactory):
def create_provider(self, config):
return S3StorageProvider(config)
```
#### 策略模式
```python
# 存储管理器支持动态切换存储策略
storage_manager.register_factory("s3", S3StorageFactory())
storage_manager.register_factory("cos", COSStorageFactory())
# 运行时选择存储提供者
provider = storage_manager.create_provider("s3", config)
```
### 核心组件
#### 1. 抽象接口层 (`storage_interface.py`)
- `StorageProvider`: 存储提供者抽象基类
- `StorageFactory`: 存储工厂抽象基类
- `StorageManager`: 存储管理器
- `UploadResult`/`DownloadResult`: 结果封装类
#### 2. 具体实现层
- `S3StorageProvider`: AWS S3实现
- `COSStorageProvider`: 腾讯云COS实现
- 各自的工厂类
#### 3. 节点层 (`object_storage_nodes.py`)
- ComfyUI节点实现
- 统一的错误处理
- 类型安全的接口
#### 4. 配置管理 (`config_utils.py`)
- 统一配置加载
- 环境变量支持
- 配置验证
### 扩展新的存储服务
添加新的存储服务只需要三个步骤:
1. **实现存储提供者**
```python
class NewStorageProvider(StorageProvider):
def upload_file(self, local_path, remote_key, **kwargs):
# 实现上传逻辑
pass
# ... 实现其他抽象方法
```
2. **实现存储工厂**
```python
class NewStorageFactory(StorageFactory):
def create_provider(self, config):
return NewStorageProvider(config)
```
3. **注册到管理器**
```python
storage_manager.register_factory("new_storage", NewStorageFactory())
```
## 🧪 测试
### 运行测试
```bash
# 运行所有测试
pytest
# 运行特定测试文件
pytest tests/test_s3_provider.py
# 运行详细输出
pytest -v
# 运行代码覆盖率测试
pytest --cov=utils --cov-report=html
```
### 测试结构
```
tests/
├── __init__.py # 测试模块初始化
├── conftest.py # pytest配置和fixtures
├── test_storage_interface.py # 接口层测试
├── test_s3_provider.py # S3提供者测试
├── test_cos_provider.py # COS提供者测试
├── test_nodes.py # 节点测试
└── test_integration.py # 集成测试
```
### 测试标记
```bash
# 只运行单元测试
pytest -m unit
# 只运行集成测试
pytest -m integration
# 排除慢速测试
pytest -m "not slow"
```
## 📊 性能优化
### 1. 懒加载
- 存储客户端采用懒加载模式
- 减少初始化开销
### 2. 连接复用
- 复用HTTP连接
- 减少网络开销
### 3. 重试机制
- 指数退避重试
- 提高成功率
### 4. 并发处理
- 支持异步操作
- 提高并发性能
## 🔧 配置选项
### 全局配置
| 参数 | 类型 | 必需 | 描述 | 默认值 |
|------|------|------|------|-------|
| `aws_key_id` | string | 是 | AWS访问密钥ID | - |
| `aws_access_key` | string | 是 | AWS秘密访问密钥 | - |
| `cos_secret_id` | string | 是 | COS秘钥ID | - |
| `cos_secret_key` | string | 是 | COS秘钥 | - |
| `cos_region` | string | 是 | COS区域 | - |
### 节点配置
每个节点支持的参数请参考节点的 `INPUT_TYPES()` 方法定义。
## 🚨 错误处理
### 常见错误及解决方案
#### 1. 配置错误
```
ValueError: S3配置缺失必要参数: ['access_key_id']
```
**解决方案**: 检查 `config.yaml` 或环境变量中的AWS配置
#### 2. 网络错误
```
Exception: S3上传失败: 网络连接超时
```
**解决方案**: 检查网络连接,存储服务会自动重试
#### 3. 权限错误
```
Exception: COS下载失败: 权限不足
```
**解决方案**: 检查存储桶权限和访问密钥
## 📈 监控和日志
### 日志级别
使用 `loguru` 进行日志记录:
- `INFO`: 正常操作信息
- `WARNING`: 警告信息(重试等)
- `ERROR`: 错误信息
- `DEBUG`: 调试信息
### 监控指标
建议监控以下指标:
- 上传/下载成功率
- 响应时间
- 错误率
- 重试次数
## 🤝 贡献指南
1. Fork 本项目
2. 创建特性分支 (`git checkout -b feature/AmazingFeature`)
3. 提交改动 (`git commit -m 'Add some AmazingFeature'`)
4. 推送到分支 (`git push origin feature/AmazingFeature`)
5. 创建 Pull Request
### 开发规范
- 遵循 PEP 8 代码规范
- 添加类型注解
- 编写测试用例
- 更新文档
## 📄 License
本项目基于 MIT 许可证 - 查看 [LICENSE](LICENSE) 文件了解详情
## 🙏 致谢
- [ComfyUI](https://github.com/comfyanonymous/ComfyUI) - 强大的Stable Diffusion GUI
- [boto3](https://github.com/boto/boto3) - AWS SDK for Python
- [qcloud-cos-python-sdk](https://github.com/tencentyun/cos-python-sdk-v5) - 腾讯云COS SDK
## 📞 支持
如果遇到问题或有功能建议,请:
1. 查看 [FAQ](#常见问题)
2. 搜索现有的 [Issues](https://github.com/your-repo/issues)
3. 创建新的 Issue
## 🔄 版本历史
### v2.0.0 (当前版本)
- 🏗️ 完全重构架构,采用抽象工厂模式
- ✨ 支持多种存储服务
- 🧪 添加全面的测试覆盖
- 📝 完善文档和注释
- 🛡️ 增强错误处理和重试机制
### v1.0.0 (历史版本)
- 基础的S3和COS上传下载功能
- 简单的错误处理
---
**⭐ 如果这个项目对你有帮助,请给我们一个星标!**

View File

@@ -106,15 +106,16 @@ class FaceExtract:
device = model_management.get_torch_device()
image_np = 255.0 * image.cpu().numpy()
model_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"model",
"yolov8n-face-lindevs.pt",
)
if not os.path.exists(model_path):
download_file("https://github.com/lindevs/yolov8-face/releases/latest/download/yolov8n-face-lindevs.pt",model_path)
model = YOLO(
model=model_path
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"model",
"yolov8n-face-lindevs.pt",
)
if not os.path.exists(model_path):
download_file(
"https://github.com/lindevs/yolov8-face/releases/latest/download/yolov8n-face-lindevs.pt",
model_path,
)
model = YOLO(model=model_path)
total_images = image_np.shape[0]
out_images = np.ndarray(shape=(total_images, 512, 512, 3))
print("shape", image_np.shape)
@@ -139,10 +140,10 @@ class FaceExtract:
template.fill(20)
for a, a1 in zip(list(range(int(x1), int(x2))), list(range(face_size))):
for b, b1 in zip(
list(range(int(y1), int(y2))), list(range(face_size))
list(range(int(y1), int(y2))), list(range(face_size))
):
if (a >= 0 and a < r.orig_img.shape[0]) and (
b >= 0 and b < r.orig_img.shape[1]
b >= 0 and b < r.orig_img.shape[1]
):
template[a1][b1] = r.orig_img[a][b]
print(int(x1), int(x2), int(y1), int(y2))

View File

@@ -6,42 +6,90 @@ import tempfile
import time
import uuid
from time import sleep
from typing import Any, Dict
import cv2
import folder_paths
import numpy as np
import requests
import torch
import yaml
from PIL import Image
from loguru import logger
from qcloud_cos import CosConfig, CosS3Client
from PIL import Image
from torchvision.transforms import transforms
from tqdm import tqdm
from ..utils.config_utils import config
from ..utils.object_storage import UploadResult, get_provider
class JMUtils:
"""
即梦AI工具类
提供即梦AI视频生成服务的完整功能包括
- 图像上传到云存储
- 任务提交和状态查询
- 视频下载和处理
- 张量和图像格式转换
使用统一的存储抽象层,支持多种云存储服务。
"""
def __init__(self):
if "aws_key_id" in list(os.environ.keys()):
yaml_config = os.environ
else:
with open(
os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "config.yaml"
),
encoding="utf-8",
mode="r+",
) as f:
yaml_config = yaml.load(f, Loader=yaml.FullLoader)
"""
初始化即梦工具实例
self.api_key = yaml_config["jm_api_key"]
self.cos_region = yaml_config["cos_region"]
self.cos_secret_id = yaml_config["cos_secret_id"]
self.cos_secret_key = yaml_config["cos_secret_key"]
self.cos_bucket_name = yaml_config["cos_sucai_bucket_name"]
def submit_task(self, prompt: str, img_url: str, duration: str = "10", resolution:str="720p"):
从配置中读取API密钥和存储配置信息
"""
try:
# 获取即梦API配置
self.api_key = config.get_config("jm_api_key")
if not self.api_key:
raise ValueError("即梦API密钥未配置")
# 获取COS存储配置用于素材上传
cos_config = config.get_cos_config()
self.cos_bucket_name = cos_config.get("bucket_name") or config.get_config(
"cos_sucai_bucket_name"
)
if not self.cos_bucket_name:
raise ValueError("COS素材存储桶未配置")
# 获取存储提供者
self.storage_provider = get_provider("cos")
logger.info(f"即梦工具初始化成功,使用存储桶: {self.cos_bucket_name}")
except Exception as e:
logger.error(f"即梦工具初始化失败: {e}")
raise
def submit_task(
self, prompt: str, img_url: str, duration: str = "10", resolution: str = "720p"
) -> Dict[str, Any]:
"""
提交即梦AI视频生成任务
Args:
prompt: 生成提示词
img_url: 输入图像URL
duration: 视频时长(秒)
resolution: 视频分辨率
Returns:
Dict: 任务提交结果
- status: 是否成功
- data: 任务ID或原图URL
- msg: 消息
"""
try:
# 验证输入参数
if not prompt or not prompt.strip():
return {"status": False, "data": None, "msg": "提示词不能为空"}
if not img_url or not img_url.strip():
return {"status": False, "data": None, "msg": "图像URL不能为空"}
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {self.api_key}",
@@ -52,80 +100,244 @@ class JMUtils:
"content": [
{
"type": "text",
"text": f"{prompt} --resolution {resolution} --dur {duration} --camerafixed false",
"text": f"{prompt.strip()} --resolution {resolution} --dur {duration} --camerafixed false",
},
{
"type": "image_url",
"image_url": {
"url": img_url,
"url": img_url.strip(),
},
},
],
}
response = requests.post("https://ark.cn-beijing.volces.com/api/v3/contents/generations/tasks",
headers=headers, json=json_data)
logger.info(f"submit task: {json.dumps(response.json())}")
logger.info(
f"即梦任务提交中: prompt='{prompt[:50]}...', resolution={resolution}, duration={duration}"
)
response = requests.post(
"https://ark.cn-beijing.volces.com/api/v3/contents/generations/tasks",
headers=headers,
json=json_data,
timeout=30,
)
response.raise_for_status()
resp_json = response.json()
logger.info(
f"即梦任务提交响应: {json.dumps(resp_json, ensure_ascii=False)}"
)
if "id" not in resp_json:
return {"status": False, "data": img_url, "msg": resp_json["error"]["message"]}
error_msg = "未知错误"
if "error" in resp_json and "message" in resp_json["error"]:
error_msg = resp_json["error"]["message"]
return {
"status": False,
"data": img_url,
"msg": f"任务提交失败: {error_msg}",
}
else:
job_id = resp_json["id"]
logger.info(f"即梦任务提交成功任务ID: {job_id}")
return {"data": job_id, "status": True, "msg": "任务提交成功"}
except requests.RequestException as e:
logger.error(f"即梦API请求失败: {e}")
return {"data": None, "status": False, "msg": f"网络请求失败: {str(e)}"}
except Exception as e:
logger.error(e)
logger.error(f"即梦任务提交异常: {e}")
return {"data": None, "status": False, "msg": str(e)}
def query_status(self, job_id: str):
def query_status(self, job_id: str) -> Dict[str, Any]:
"""
查询即梦AI任务状态
Args:
job_id: 任务ID
Returns:
Dict: 任务状态查询结果
- status: 任务是否完成成功
- data: 视频URL如果完成
- msg: 状态消息
"""
resp_dict = {"status": False, "data": None, "msg": ""}
try:
if not job_id or not job_id.strip():
resp_dict["msg"] = "任务ID不能为空"
return resp_dict
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {self.api_key}",
}
response = requests.get(f"https://ark.cn-beijing.volces.com/api/v3/contents/generations/tasks/{job_id}",
headers=headers)
resp_json = response.json()
resp_dict["status"] = resp_json["status"] == "succeeded"
resp_dict["msg"] = resp_json["status"]
resp_dict["data"] = resp_json["content"]["video_url"] if "content" in resp_json else None
except Exception as e:
logger.error(f"error:{str(e)}")
resp_dict["msg"] = str(e)
finally:
return resp_dict
def upload_io_to_cos(self, file: io.IOBase, mime_type: str = "image/png"):
resp_data = {'status': True, 'data': '', 'msg': ''}
category = mime_type.split('/')[0]
suffix = mime_type.split('/')[1]
try:
object_key = f'tk/{category}/{uuid.uuid4()}.{suffix}'
config = CosConfig(Region=self.cos_region, SecretId=self.cos_secret_id, SecretKey=self.cos_secret_key)
client = CosS3Client(config)
_ = client.upload_file_from_buffer(
Bucket=self.cos_bucket_name,
Key=object_key,
Body=file
response = requests.get(
f"https://ark.cn-beijing.volces.com/api/v3/contents/generations/tasks/{job_id.strip()}",
headers=headers,
timeout=15,
)
url = f'https://{self.cos_bucket_name}.cos.{self.cos_region}.myqcloud.com/{object_key}'
resp_data['data'] = url
resp_data['msg'] = '上传成功'
response.raise_for_status()
resp_json = response.json()
task_status = resp_json.get("status", "unknown")
# 任务完成成功
if task_status == "succeeded":
resp_dict["status"] = True
resp_dict["msg"] = "任务完成"
if "content" in resp_json and "video_url" in resp_json["content"]:
resp_dict["data"] = resp_json["content"]["video_url"]
else:
resp_dict["status"] = False
resp_dict["msg"] = "任务完成但未找到视频URL"
# 任务失败
elif task_status in ["failed", "error"]:
resp_dict["status"] = False
error_msg = "任务失败"
if "error" in resp_json:
error_msg += f": {resp_json['error'].get('message', '未知错误')}"
resp_dict["msg"] = error_msg
# 任务进行中
elif task_status in ["pending", "running", "processing"]:
resp_dict["status"] = False
resp_dict["msg"] = f"任务进行中: {task_status}"
# 其他状态
else:
resp_dict["status"] = False
resp_dict["msg"] = f"未知任务状态: {task_status}"
except requests.RequestException as e:
logger.error(f"即梦状态查询网络错误: {e}")
resp_dict["msg"] = f"网络请求失败: {str(e)}"
except Exception as e:
logger.error(f"即梦状态查询异常: {e}")
resp_dict["msg"] = str(e)
return resp_dict
def upload_io_to_cos(
self, file: io.IOBase, mime_type: str = "image/png"
) -> Dict[str, Any]:
"""
上传IO对象到COS存储
Args:
file: 文件IO对象
mime_type: MIME类型
Returns:
Dict: 包含上传结果的字典
- status: 是否成功
- data: 上传后的URL
- msg: 消息
"""
resp_data = {"status": True, "data": "", "msg": ""}
try:
# 解析MIME类型
parts = mime_type.split("/")
category = parts[0] if len(parts) > 0 else "file"
suffix = parts[1] if len(parts) > 1 else "bin"
# 生成存储键名
object_key = f"tk/{category}/{uuid.uuid4()}.{suffix}"
logger.info(f"开始上传文件到COS: {object_key}")
# 读取文件内容
file_content = file.read()
file.seek(0) # 重置文件指针
# 使用统一存储接口上传
result: UploadResult = self.storage_provider.upload_bytes(
file_content, object_key, bucket_name=self.cos_bucket_name
)
if result.success:
# 构造COS URL如果result中没有提供
if result.url:
resp_data["data"] = result.url
else:
# 构造默认的COS URL
cos_config = config.get_cos_config()
region = cos_config.get("region", "ap-beijing")
resp_data["data"] = (
f"https://{self.cos_bucket_name}.cos.{region}.myqcloud.com/{object_key}"
)
resp_data["msg"] = "上传成功"
logger.info(f"文件上传成功: {resp_data['data']}")
else:
resp_data["status"] = False
resp_data["msg"] = result.message or "上传失败"
logger.error(f"文件上传失败: {resp_data['msg']}")
except Exception as e:
logger.error(e)
resp_data['status'] = False
resp_data['msg'] = str(e)
logger.error(f"上传文件时发生异常: {e}")
resp_data["status"] = False
resp_data["msg"] = str(e)
return resp_data
def tensor_to_io(srlf, tensor: torch.Tensor):
# 转换为PIL图像
img = Image.fromarray(np.clip(255. * tensor.cpu().squeeze().numpy(), 0, 255).astype(np.uint8))
image_data = io.BytesIO()
img.save(image_data, format='PNG')
image_data.seek(0)
return image_data
def tensor_to_io(self, tensor: torch.Tensor) -> io.BytesIO:
"""
将PyTorch张量转换为PNG格式的IO对象
Args:
tensor: PyTorch图像张量支持多种格式
- (H, W) 灰度图
- (H, W, C) RGB图像
- (1, H, W, C) 批次图像
Returns:
io.BytesIO: PNG格式的字节流对象
Raises:
ValueError: 当张量格式不支持时
"""
try:
# 处理张量维度
if tensor.dim() == 4: # (1, H, W, C)
tensor = tensor.squeeze(0)
elif tensor.dim() == 2: # (H, W) 灰度图
pass # 保持原样
elif tensor.dim() == 3: # (H, W, C)
pass # 保持原样
else:
raise ValueError(f"不支持的张量维度: {tensor.dim()}D")
# 转换为numpy数组
numpy_array = tensor.cpu().numpy()
# 确保数值在有效范围内
numpy_array = np.clip(numpy_array, 0.0, 1.0)
# 转换为0-255范围的uint8
image_array = (numpy_array * 255).astype(np.uint8)
# 处理灰度图
if len(image_array.shape) == 2:
img = Image.fromarray(image_array, mode="L")
else:
img = Image.fromarray(image_array, mode="RGB")
# 保存为PNG格式的BytesIO
image_data = io.BytesIO()
img.save(image_data, format="PNG", optimize=True)
image_data.seek(0)
logger.debug(f"张量转换为PNG成功大小: {len(image_data.getvalue())} bytes")
return image_data
except Exception as e:
logger.error(f"张量转换失败: {e}")
raise ValueError(f"张量转换为图像失败: {str(e)}")
def read_video_last_frame_to_tensor(self, video_path: str) -> torch.Tensor:
"""
@@ -170,11 +382,13 @@ class JMUtils:
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# 转换为PyTorch张量并调整维度为BCHW
transform = transforms.Compose([
transforms.ToTensor() # 转换为[C, H, W]格式的张量值范围从0到1
])
transform = transforms.Compose(
[transforms.ToTensor()] # 转换为[C, H, W]格式的张量值范围从0到1
)
tensor = transform(frame_rgb).unsqueeze(0).permute(0, 2, 3, 1) # 添加批次维度,变为[1, H, W, C]
tensor = (
transform(frame_rgb).unsqueeze(0).permute(0, 2, 3, 1)
) # 添加批次维度,变为[1, H, W, C]
return tensor
@@ -186,7 +400,7 @@ class JMUtils:
if path:
temp_path = path
else:
temp_file = tempfile.NamedTemporaryFile(suffix='.mp4', delete=False)
temp_file = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
temp_path = temp_file.name
temp_file.close()
@@ -196,16 +410,16 @@ class JMUtils:
response.raise_for_status()
# 获取文件大小
total_size = int(response.headers.get('content-length', 0))
total_size = int(response.headers.get("content-length", 0))
block_size = 1024 # 1 KB
# 使用tqdm显示下载进度
with open(temp_path, 'wb') as f, tqdm(
desc=url.split('/')[-1],
total=total_size,
unit='B',
unit_scale=True,
unit_divisor=1024
with open(temp_path, "wb") as f, tqdm(
desc=url.split("/")[-1],
total=total_size,
unit="B",
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in response.iter_content(block_size):
size = f.write(data)
@@ -221,21 +435,19 @@ class JMUtils:
else:
raise
def jpg_to_tensor(self, image_path, channel_first=False):
def jpg_to_tensor(self, image_path):
"""
将JPG图像转换为PyTorch张量
参数:
- image_path: JPG图像文件路径
- normalize: 是否将像素值归一化到[0.0, 1.0]
- channel_first: 是否将通道维度放在前面 (C, H, W)
返回:
- tensor: PyTorch张量
"""
try:
# 打开图像文件
image = Image.open(image_path).convert('RGB')
image = Image.open(image_path).convert("RGB")
# 转换为张量
tensor = torch.from_numpy(np.array(image).astype(np.float32) / 255.0)[None,]
@@ -246,7 +458,7 @@ class JMUtils:
print(f"转换失败: {str(e)}")
raise
def get_last_15th_frame_tensor(self, video_url, cleanup=True):
def get_last_15th_frame_tensor(self, video_url):
"""
从视频URL截取倒数第15帧并转换为Tensor
先下载视频到本地临时文件再处理
@@ -257,18 +469,20 @@ class JMUtils:
# 获取视频总帧数
cmd_frames = [
'ffprobe', '-v', 'error',
'-select_streams', 'v:0',
'-show_entries', 'stream=nb_frames',
'-of', 'default=nokey=1:noprint_wrappers=1',
video_path
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=nb_frames",
"-of",
"default=nokey=1:noprint_wrappers=1",
video_path,
]
result = subprocess.run(
cmd_frames,
capture_output=True,
text=True,
check=True
cmd_frames, capture_output=True, text=True, check=True
)
# 处理可能的非数字输出
@@ -276,19 +490,14 @@ class JMUtils:
if not frame_count.isdigit():
# 备选方案:通过解码获取帧数
print("无法获取准确帧数,尝试直接解码...")
cmd_decode = [
'ffmpeg', '-i', video_path,
'-f', 'null', '-'
]
cmd_decode = ["ffmpeg", "-i", video_path, "-f", "null", "-"]
decode_result = subprocess.run(
cmd_decode,
capture_output=True,
text=True
cmd_decode, capture_output=True, text=True
)
for line in decode_result.stderr.split('\n'):
if 'frame=' in line:
parts = line.split('frame=')[-1].split()[0]
for line in decode_result.stderr.split("\n"):
if "frame=" in line:
parts = line.split("frame=")[-1].split()[0]
if parts.isdigit():
frame_count = int(parts)
break
@@ -302,25 +511,29 @@ class JMUtils:
print(f"视频总帧数: {frame_count}, 目标帧: {target_frame}")
# 截取指定帧
with tempfile.NamedTemporaryFile(suffix='%03d.jpg', delete=True) as frame_file:
with tempfile.NamedTemporaryFile(
suffix="%03d.jpg", delete=True
) as frame_file:
frame_path = frame_file.name
cmd_extract = [
'ffmpeg',
'-ss', f'00:00:00',
'-i', video_path,
'-vframes', '1',
'-vf', f'select=eq(n\,{target_frame})',
'-vsync', '0',
'-an', '-y',
frame_path
"ffmpeg",
"-ss",
f"00:00:00",
"-i",
video_path,
"-vframes",
"1",
"-vf",
f"select=eq(n\,{target_frame})",
"-vsync",
"0",
"-an",
"-y",
frame_path,
]
subprocess.run(
cmd_extract,
capture_output=True,
check=True
)
subprocess.run(cmd_extract, capture_output=True, check=True)
# 转换为Tensor
tensor = self.jpg_to_tensor(frame_path.replace("%03d", "001"))
@@ -332,12 +545,7 @@ class JMUtils:
class JMGestureCorrect:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"image": ("IMAGE",),
"resolution":(["720p","1080p"])
}
}
return {"required": {"image": ("IMAGE",), "resolution": (["720p", "1080p"])}}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("正面图",)
@@ -355,7 +563,9 @@ class JMGestureCorrect:
else:
raise Exception("上传失败")
prompt = "Stand straight ahead, facing the camera, showing your full body, maintaining a proper posture, keeping the camera still, and ensuring that your head and feet are all within the frame"
submit_data = client.submit_task(prompt, image_url, duration="5", resolution=resolution)
submit_data = client.submit_task(
prompt, image_url, duration="5", resolution=resolution
)
if submit_data["status"]:
job_id = submit_data["data"]
else:
@@ -368,7 +578,11 @@ class JMGestureCorrect:
job_data = query["data"]
break
else:
if "error" in query["msg"] or "失败" in query["msg"] or "fail" in query["msg"]:
if (
"error" in query["msg"]
or "失败" in query["msg"]
or "fail" in query["msg"]
):
raise Exception("即梦任务失败 {}".format(query["msg"]))
sleep(interval)
if not job_data:
@@ -382,21 +596,35 @@ class JMCustom:
return {
"required": {
"image": ("IMAGE",),
"prompt": ("STRING", {
"default": "Stand straight ahead, facing the camera, showing your full body, maintaining a proper posture, keeping the camera still, and ensuring that your head and feet are all within the frame",
"multiline": True}),
"prompt": (
"STRING",
{
"default": "Stand straight ahead, facing the camera, showing your full body, maintaining a proper posture, keeping the camera still, and ensuring that your head and feet are all within the frame",
"multiline": True,
},
),
"duration": ("INT", {"default": 5, "min": 2, "max": 10}),
"resolution": (["720p", "1080p"]),
"wait_time": ("INT", {"default": 180, "min": 60, "max": 600}),
}
}
RETURN_TYPES = ("STRING", "IMAGE",)
RETURN_TYPES = (
"STRING",
"IMAGE",
)
RETURN_NAMES = ("视频存储路径", "视频最后一帧")
FUNCTION = "gen"
CATEGORY = "不忘科技-自定义节点🚩/视频/即梦"
def gen(self, image: torch.Tensor, prompt: str, duration: int, resolution: str, wait_time: int):
def gen(
self,
image: torch.Tensor,
prompt: str,
duration: int,
resolution: str,
wait_time: int,
):
interval = 2
client = JMUtils()
image_io = client.tensor_to_io(image)
@@ -405,7 +633,9 @@ class JMCustom:
image_url = upload_data["data"]
else:
raise Exception("上传失败")
submit_data = client.submit_task(prompt, image_url, str(duration), resolution=resolution)
submit_data = client.submit_task(
prompt, image_url, str(duration), resolution=resolution
)
if submit_data["status"]:
job_id = submit_data["data"]
else:
@@ -418,11 +648,21 @@ class JMCustom:
job_data = query["data"]
break
else:
if "error" in query["msg"] or "失败" in query["msg"] or "fail" in query["msg"]:
if (
"error" in query["msg"]
or "失败" in query["msg"]
or "fail" in query["msg"]
):
raise Exception("即梦任务失败 {}".format(query["msg"]))
sleep(interval)
if not job_data:
raise Exception("即梦任务等待超时")
video_path, last_scene = client.download_video(job_data, path=os.path.join(folder_paths.get_output_directory(),
f"{uuid.uuid4()}.mp4"))
return (video_path, last_scene,)
output_dir = folder_paths.get_output_directory()
video_path, last_scene = client.download_video(
job_data, path=os.path.join(output_dir, f"{uuid.uuid4()}.mp4")
)
return (
video_path,
last_scene,
)

View File

@@ -2,15 +2,14 @@ import io
import json
from time import sleep, time
import folder_paths
import requests
import torch
from PIL import Image
from loguru import logger
from PIL import Image
from torchvision import transforms
from ..utils.http_utils import send_request
from ..utils.image_utils import tensor_to_image_bytes, base64_to_tensor
from ..utils.image_utils import base64_to_tensor, tensor_to_image_bytes
def url_to_tensor(image_url: str, max_retries: int = 3):

View File

@@ -2,12 +2,12 @@ import os
import uuid
from io import BytesIO
import folder_paths
import loguru
import numpy as np
import requests
import torch
from PIL import Image
import folder_paths
# 定义节点类

View File

@@ -11,8 +11,8 @@ import folder_paths
import httpx
import numpy as np
import torch
from jinja2 import StrictUndefined, Template
from PIL import Image
from jinja2 import Template, StrictUndefined
from retry import retry
@@ -33,18 +33,22 @@ def find_value_recursive(key: str, data: Union[dict, list]) -> str | None | Any:
def image_tensor_to_base64(image):
pil_image = Image.fromarray(np.clip(255. * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
pil_image = Image.fromarray(
np.clip(255.0 * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8)
)
# 创建一个BytesIO对象用于临时存储图像数据
image_data = io.BytesIO()
# 将图像保存到BytesIO对象中格式为PNG
pil_image.save(image_data, format='PNG')
pil_image.save(image_data, format="PNG")
# 将BytesIO对象的内容转换为字节串
image_data_bytes = image_data.getvalue()
# 将图像数据编码为Base64字符串
encoded_image = "data:image/png;base64," + base64.b64encode(image_data_bytes).decode('utf-8')
encoded_image = "data:image/png;base64," + base64.b64encode(
image_data_bytes
).decode("utf-8")
return encoded_image
@@ -56,14 +60,18 @@ class LLMChat:
def INPUT_TYPES(s):
return {
"required": {
"llm_provider": (["claude-3-5-sonnet-20241022-v2",
"claude-3-5-sonnet-20241022-v3",
"claude-3-7-sonnet-20250219-v1",
"claude-4-sonnet-20250514-v1",
"gpt-4o-1120",
"gpt-4.1",
"deepseek-v3",
"deepseek-r1"],),
"llm_provider": (
[
"claude-3-5-sonnet-20241022-v2",
"claude-3-5-sonnet-20241022-v3",
"claude-3-7-sonnet-20250219-v1",
"claude-4-sonnet-20250514-v1",
"gpt-4o-1120",
"gpt-4.1",
"deepseek-v3",
"deepseek-r1",
],
),
"prompt": ("STRING", {"multiline": True}),
"temperature": ("FLOAT", {"default": 0.7, "min": 0.0, "max": 1.0}),
"max_tokens": ("INT", {"default": 4096, "min": 1, "max": 65535}),
@@ -76,32 +84,38 @@ class LLMChat:
FUNCTION = "chat"
CATEGORY = "不忘科技-自定义节点🚩/LLM"
def chat(self, llm_provider: str, prompt: str, temperature: float, max_tokens: int, timeout: int):
def chat(
self,
llm_provider: str,
prompt: str,
temperature: float,
max_tokens: int,
timeout: int,
):
@retry(Exception, tries=3, delay=1)
def _chat():
try:
with httpx.Client(timeout=httpx.Timeout(timeout, connect=15)) as session:
resp = session.post("https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777"
},
json={
"model": llm_provider,
"messages": [
{
"role": "user",
"content": prompt
}
],
"temperature": temperature,
"max_tokens": max_tokens
})
with httpx.Client(
timeout=httpx.Timeout(timeout, connect=15)
) as session:
resp = session.post(
"https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777",
},
json={
"model": llm_provider,
"messages": [{"role": "user", "content": prompt}],
"temperature": temperature,
"max_tokens": max_tokens,
},
)
resp.raise_for_status()
resp = resp.json()
content = find_value_recursive("content", resp)
content = re.sub(r'\n{2,}', '\n', content)
content = re.sub(r"\n{2,}", "\n", content)
except Exception as e:
raise Exception("llm调用失败 {}".format(e))
return (content,)
@@ -115,12 +129,15 @@ class LLMChatMultiModalImageUpload:
@classmethod
def INPUT_TYPES(s):
input_dir = folder_paths.get_input_directory()
files = [f for f in os.listdir(input_dir) if os.path.isfile(os.path.join(input_dir, f))]
files = [
f
for f in os.listdir(input_dir)
if os.path.isfile(os.path.join(input_dir, f))
]
files = folder_paths.filter_files_content_types(files, ["image"])
return {
"required": {
"llm_provider": (["gpt-4o-1120",
"gpt-4.1"],),
"llm_provider": (["gpt-4o-1120", "gpt-4.1"],),
"prompt": ("STRING", {"multiline": True}),
"image": (sorted(files), {"image_upload": True}),
"temperature": ("FLOAT", {"default": 0.7, "min": 0.0, "max": 1.0}),
@@ -134,43 +151,58 @@ class LLMChatMultiModalImageUpload:
FUNCTION = "chat"
CATEGORY = "不忘科技-自定义节点🚩/LLM"
def chat(self, llm_provider: str, prompt: str, image, temperature: float, max_tokens: int, timeout: int):
def chat(
self,
llm_provider: str,
prompt: str,
image,
temperature: float,
max_tokens: int,
timeout: int,
):
@retry(Exception, tries=3, delay=1)
def _chat():
try:
image_path = folder_paths.get_annotated_filepath(image)
mime_type, _ = guess_type(image_path)
with open(image_path, "rb") as image_file:
base64_encoded_data = base64.b64encode(image_file.read()).decode('utf-8')
with httpx.Client(timeout=httpx.Timeout(timeout, connect=15)) as session:
resp = session.post("https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777"
base64_encoded_data = base64.b64encode(image_file.read()).decode(
"utf-8"
)
with httpx.Client(
timeout=httpx.Timeout(timeout, connect=15)
) as session:
resp = session.post(
"https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777",
},
json={
"model": llm_provider,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {
"url": f"data:{mime_type};base64,{base64_encoded_data}"
},
},
json={
"model": llm_provider,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {
"url": f"data:{mime_type};base64,{base64_encoded_data}"},
},
]
}
],
"temperature": temperature,
"max_tokens": max_tokens
})
],
}
],
"temperature": temperature,
"max_tokens": max_tokens,
},
)
resp.raise_for_status()
resp = resp.json()
content = find_value_recursive("content", resp)
content = re.sub(r'\n{2,}', '\n', content)
content = re.sub(r"\n{2,}", "\n", content)
except Exception as e:
# logger.exception("llm调用失败 {}".format(e))
raise Exception("llm调用失败 {}".format(e))
@@ -186,8 +218,7 @@ class LLMChatMultiModalImageTensor:
def INPUT_TYPES(s):
return {
"required": {
"llm_provider": (["gpt-4o-1120",
"gpt-4.1"],),
"llm_provider": (["gpt-4o-1120", "gpt-4.1"],),
"prompt": ("STRING", {"multiline": True}),
"image": ("IMAGE",),
"temperature": ("FLOAT", {"default": 0.7, "min": 0.0, "max": 1.0}),
@@ -201,39 +232,52 @@ class LLMChatMultiModalImageTensor:
FUNCTION = "chat"
CATEGORY = "不忘科技-自定义节点🚩/LLM"
def chat(self, llm_provider: str, prompt: str, image: torch.Tensor, temperature: float, max_tokens: int,
timeout: int):
def chat(
self,
llm_provider: str,
prompt: str,
image: torch.Tensor,
temperature: float,
max_tokens: int,
timeout: int,
):
@retry(Exception, tries=3, delay=1)
def _chat():
try:
with httpx.Client(timeout=httpx.Timeout(timeout, connect=15)) as session:
resp = session.post("https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777"
with httpx.Client(
timeout=httpx.Timeout(timeout, connect=15)
) as session:
resp = session.post(
"https://gateway.bowong.cc/chat/completions",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer auth-bowong7777",
},
json={
"model": llm_provider,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {
"url": image_tensor_to_base64(image)
},
},
json={
"model": llm_provider,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {"url": image_tensor_to_base64(image)},
},
]
}
],
"temperature": temperature,
"max_tokens": max_tokens
})
],
}
],
"temperature": temperature,
"max_tokens": max_tokens,
},
)
resp.raise_for_status()
resp = resp.json()
content = find_value_recursive("content", resp)
content = re.sub(r'\n{2,}', '\n', content)
content = re.sub(r"\n{2,}", "\n", content)
except Exception as e:
# logger.exception("llm调用失败 {}".format(e))
raise Exception("llm调用失败 {}".format(e))

File diff suppressed because it is too large Load Diff

View File

@@ -1,25 +1,43 @@
"""
工具节点模块 - 重构版本
提供各种实用功能的ComfyUI节点包括
- 数据库日志记录
- VOD文件下载使用统一存储抽象层
- 模型管理
- 文件遍历
- Webhook通知
- 任务ID生成
本模块已经过重构使用统一的存储抽象层替代直接SDK调用。
"""
import glob
import json
import os
import time
import uuid
from pathlib import Path
from typing import Any, Dict, Tuple
import comfy.model_management
import requests
import server
import yaml
from loguru import logger
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from tencentcloud.common import credential
from tencentcloud.common.profile.client_profile import ClientProfile
from tencentcloud.common.profile.http_profile import HttpProfile
from tencentcloud.vod.v20180717 import vod_client, models
from ..utils.object_storage import DownloadResult, get_provider
from ..utils.task_table import Task
class LogToDB:
"""
数据库日志记录节点
将ComfyUI工作流的执行结果记录到数据库中支持任务状态跟踪。
"""
@classmethod
def INPUT_TYPES(s):
return {
@@ -27,87 +45,112 @@ class LogToDB:
"job_id": ("STRING", {"forceInput": True}),
"log": ("STRING", {"forceInput": True}),
"status": ("INT", {"default": 1, "max": 1}),
"sql_url": ("STRING", {
"default": "mysql+pymysql://root:root@example.com:3306/test"}),
"sql_url": (
"STRING",
{"default": "mysql+pymysql://root:root@example.com:3306/test"},
),
},
"hidden": {
"unique_id": "UNIQUE_ID",
}
},
}
RETURN_TYPES = ("STRING",)
FUNCTION = "log2db"
OUTPUT_NODE = True
OUTPUT_IS_LIST = (True,)
# OUTPUT_NODE = False
CATEGORY = "不忘科技-自定义节点🚩/工具"
def log2db(self, log, status, sql_url, unique_id):
# 获取comfy服务器队列信息
(_, prompt_id, prompt, extra_data, outputs_to_execute) = next(
iter(server.PromptServer.instance.prompt_queue.currently_running.values()))
job_id = extra_data["client_id"]
engine = create_engine(
sql_url,
echo=True
)
# Base.metadata.create_all(engine)
session = sessionmaker(bind=engine)()
# 查询
tasks = session.query(Task).filter(Task.prompt_id == prompt_id).all()
print(prompt)
result = {
"curr_node_id": str(unique_id),
"last_node_id": list(prompt.keys())[-1],
"node_output": str(log)
}
if len(tasks) == 0:
# 不存在插入
task = Task(prompt_id=prompt_id, job_id=job_id, result=json.dumps(result), status=status)
session.add(task)
elif len(tasks) == 1:
# 存在更新
session.query(Task).filter(Task.prompt_id == prompt_id).update({"result": json.dumps(result),
"status": status})
else:
# 异常报错
raise RuntimeError("状态数据库prompt_id不唯一, 无法记录状态!")
session.commit()
return {"ui": {"text": json.dumps(result)}, "result": (json.dumps(result),)}
def log2db(
self, log: str, status: int, sql_url: str, unique_id: str, **kwargs
) -> Dict[str, Any]:
"""
记录日志到数据库
Args:
log: 日志内容
status: 状态代码
sql_url: 数据库连接URL
unique_id: 唯一ID
Returns:
Dict: 包含结果信息的字典
"""
try:
# 获取comfy服务器队列信息
(_, prompt_id, prompt, extra_data, outputs_to_execute) = next(
iter(
server.PromptServer.instance.prompt_queue.currently_running.values()
)
)
job_id = extra_data["client_id"]
engine = create_engine(sql_url, echo=True)
session = sessionmaker(bind=engine)()
# 查询现有任务
tasks = session.query(Task).filter(Task.prompt_id == prompt_id).all()
result = {
"curr_node_id": str(unique_id),
"last_node_id": list(prompt.keys())[-1],
"node_output": str(log),
}
if len(tasks) == 0:
# 不存在则插入
task = Task(
prompt_id=prompt_id,
job_id=job_id,
result=json.dumps(result),
status=status,
)
session.add(task)
logger.info(f"新增任务记录: {prompt_id}")
elif len(tasks) == 1:
# 存在则更新
session.query(Task).filter(Task.prompt_id == prompt_id).update(
{"result": json.dumps(result), "status": status}
)
logger.info(f"更新任务记录: {prompt_id}")
else:
# 异常情况
session.rollback()
raise RuntimeError("状态数据库prompt_id不唯一, 无法记录状态!")
session.commit()
session.close()
return {"ui": {"text": json.dumps(result)}, "result": (json.dumps(result),)}
except Exception as e:
logger.error(f"数据库日志记录失败: {e}")
return {"ui": {"text": str(e)}, "result": (str(e),)}
class VodToLocalNode:
"""
腾讯云VOD文件下载节点
使用统一的存储抽象层从腾讯云视频点播(VOD)服务下载媒体文件到本地。
支持通过文件ID和子应用ID定位和下载视频文件。
"""
def __init__(self):
if "aws_key_id" in list(os.environ.keys()):
yaml_config = os.environ
else:
with open(
os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "config.yaml"
),
encoding="utf-8",
mode="r+",
) as f:
yaml_config = yaml.load(f, Loader=yaml.FullLoader)
self.secret_id = yaml_config["cos_secret_id"]
self.secret_key = yaml_config["cos_secret_key"]
self.region = yaml_config["cos_region"]
self.vod_client = self.init_vod_client()
"""
初始化VOD下载节点
def init_vod_client(self):
"""初始化VOD客户端"""
使用统一的存储管理器获取VOD存储提供者
替代直接使用腾讯云VOD SDK的方式。
"""
try:
http_profile = HttpProfile(endpoint="vod.tencentcloudapi.com")
client_profile = ClientProfile(httpProfile=http_profile)
cred = credential.Credential(self.secret_id, self.secret_key)
return vod_client.VodClient(cred, self.region, client_profile)
# 使用统一存储管理器获取VOD提供者
self.vod_provider = get_provider("vod")
logger.info("VOD下载节点初始化成功")
except Exception as e:
raise RuntimeError(f"VOD client initialization failed: {e}")
logger.error(f"VOD下载节点初始化失败: {e}")
raise RuntimeError(f"VOD节点初始化失败: {e}")
@classmethod
def INPUT_TYPES(cls):
@@ -123,81 +166,98 @@ class VodToLocalNode:
FUNCTION = "execute"
CATEGORY = "不忘科技-自定义节点🚩/工具"
def execute(self, file_id, sub_app_id):
# 调用下载逻辑
local_path = self.download_vod(file_id, sub_app_id)
print(f"下载成功: {local_path}")
return (local_path,)
def execute(self, file_id: str, sub_app_id: str) -> Tuple[str]:
"""
执行VOD文件下载
def _get_download_url(self, file_id, sub_app_id):
"""获取媒体文件下载地址"""
Args:
file_id: VOD文件ID
sub_app_id: 子应用ID
Returns:
tuple: 包含本地文件路径的元组
Raises:
Exception: 下载失败时抛出异常
"""
try:
req = models.DescribeMediaInfosRequest()
req.FileIds = [file_id]
req.SubAppId = int(sub_app_id)
# 参数验证
if not file_id or not file_id.strip():
raise ValueError("文件ID不能为空")
if not sub_app_id or not sub_app_id.strip():
raise ValueError("子应用ID不能为空")
resp = self.vod_client.DescribeMediaInfos(req)
if not resp.MediaInfoSet:
raise ValueError("File not found")
# 调用下载逻辑
local_path = self.download_vod(file_id.strip(), sub_app_id.strip())
logger.info(f"VOD文件下载成功: {local_path}")
return (local_path,)
media_info = resp.MediaInfoSet[0]
if not media_info.BasicInfo.MediaUrl:
raise ValueError("No download URL available")
return media_info.BasicInfo.MediaUrl
except Exception as e:
raise RuntimeError(f"Tencent API error: {e}")
logger.error(f"VOD文件下载执行失败: {e}")
raise Exception(f"VOD文件下载失败: {str(e)}")
def create_directory(self, path):
def create_directory(self, path: str) -> None:
"""
创建目录(如果不存在)
Args:
path: 目录路径
"""
p = Path(path)
if not p.exists():
p.mkdir(
parents=True, exist_ok=True
) # parents=True会自动创建所有必需的父目录exist_ok=True表示如果目录已存在则不会引发异常
print(f"目录已创建: {path}")
p.mkdir(parents=True, exist_ok=True)
logger.info(f"目录已创建: {path}")
else:
print(f"目录已存在: {path}")
logger.debug(f"目录已存在: {path}")
def download_vod(self, file_id, sub_app_id):
def download_vod(self, file_id: str, sub_app_id: str) -> str:
"""
需要补充腾讯云VOD SDK调用逻辑
返回本地文件路径
下载腾讯云VOD文件到本地
Args:
file_id: VOD文件ID
sub_app_id: 子应用ID
Returns:
str: 本地文件路径
Raises:
Exception: 下载失败时抛出异常
"""
media_url = self._get_download_url(file_id=file_id, sub_app_id=sub_app_id)
print(f"download from url: {media_url}")
# 生成一个临时目录路径名并创建该目录
self.create_directory(
os.path.join(
try:
# 生成本地存储路径
download_dir = os.path.join(
os.path.dirname(os.path.abspath(__file__)), "download", f"{sub_app_id}"
)
)
output_dir = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"download",
f"{sub_app_id}",
f"{file_id}.mp4",
)
# 判断文件是否存在
if os.path.exists(output_dir):
return output_dir
return self._download_file(url=media_url, save_path=output_dir, timeout=60 * 10)
self.create_directory(download_dir)
output_path = os.path.join(download_dir, f"{file_id}.mp4")
# 如果文件已存在,直接返回
if os.path.exists(output_path):
logger.info(f"VOD文件已存在直接返回: {output_path}")
return output_path
logger.info(f"开始下载VOD文件: {file_id} -> {output_path}")
# 使用VOD提供者下载文件
result: DownloadResult = self.vod_provider.download_file(
file_id, output_path, sub_app_id, timeout=600
)
if result.success:
logger.info(f"VOD文件下载成功: {result.local_path}")
return result.local_path
else:
raise Exception(result.message or "VOD下载失败")
def _download_file(self, url: str, save_path: str, timeout: int = 30):
"""下载文件到本地"""
try:
with requests.get(url, stream=True, timeout=timeout) as response:
response.raise_for_status()
with open(save_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
if chunk:
f.write(chunk)
return save_path
except Exception as e:
raise RuntimeError(f"Download error: {e}")
logger.error(f"VOD文件下载异常: {e}")
raise Exception(f"VOD下载失败: {str(e)}")
class AnyType(str):
"""A special class that is always equal in not equal comparisons. Credit to pythongosssss"""
"""特殊类型,用于在不等比较中始终相等。Credit to pythongosssss"""
def __ne__(self, __value: object) -> bool:
return False
@@ -207,12 +267,16 @@ any = AnyType("*")
class UnloadAllModels:
"""
卸载所有模型节点
释放GPU内存卸载所有已加载的模型。
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"any": (any, {"forceInput": True})
},
"required": {"any": (any, {"forceInput": True})},
"optional": {},
}
@@ -221,20 +285,48 @@ class UnloadAllModels:
CATEGORY = "不忘科技-自定义节点🚩/工具"
OUTPUT_NODE = True
def unload_models(self, any=None):
# 卸载所有已加载的模型
comfy.model_management.soft_empty_cache()
comfy.model_management.unload_all_models()
comfy.model_management.soft_empty_cache()
return ()
def unload_models(self, any=None) -> Tuple:
"""
卸载所有已加载的模型
Args:
any: 输入参数(任意类型)
Returns:
tuple: 空元组
"""
try:
logger.info("开始卸载所有模型...")
# 卸载所有已加载的模型
comfy.model_management.soft_empty_cache()
comfy.model_management.unload_all_models()
comfy.model_management.soft_empty_cache()
logger.info("所有模型已成功卸载")
return ()
except Exception as e:
logger.error(f"模型卸载失败: {e}")
# 即使失败也返回空元组,避免中断工作流
return ()
class TraverseFolder:
"""
文件夹遍历节点
遍历指定文件夹,查找符合条件的文件。
"""
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"folder": ("STRING", {"default": r"E:\comfy\ComfyUI\input\s3", "required": True}),
"folder": (
"STRING",
{"default": r"E:\comfy\ComfyUI\input\s3", "required": True},
),
"subfix": ("STRING", {"default": ".mp4", "required": True}),
"recursive": ("BOOLEAN", {"default": True, "required": True}),
"idx": (
@@ -246,34 +338,94 @@ class TraverseFolder:
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("文件路径",)
FUNCTION = "compute"
CATEGORY = "不忘科技-自定义节点🚩/工具"
def compute(self, folder, subfix, recursive, idx):
files = glob.glob(os.path.join(folder, r"**\*%s" % subfix), recursive=recursive)
if len(files) == 0:
raise RuntimeError("No Files Found")
return (str(files[idx % len(files)]),)
def compute(
self, folder: str, subfix: str, recursive: bool, idx: int
) -> Tuple[str]:
"""
遍历文件夹查找文件
Args:
folder: 文件夹路径
subfix: 文件后缀
recursive: 是否递归查找
idx: 文件索引
Returns:
tuple: 包含文件路径的元组
Raises:
RuntimeError: 未找到文件时抛出异常
"""
try:
# 参数验证
if not folder or not folder.strip():
raise ValueError("文件夹路径不能为空")
if not subfix or not subfix.strip():
raise ValueError("文件后缀不能为空")
folder = folder.strip()
subfix = subfix.strip()
# 构建搜索模式
if recursive:
pattern = os.path.join(folder, f"**/*{subfix}")
else:
pattern = os.path.join(folder, f"*{subfix}")
# 查找文件
files = glob.glob(pattern, recursive=recursive)
if len(files) == 0:
logger.warning(f"在文件夹 {folder} 中未找到后缀为 {subfix} 的文件")
raise RuntimeError("No Files Found")
# 选择文件
selected_file = files[idx % len(files)]
logger.info(
f"找到 {len(files)} 个文件,选择第 {idx % len(files)} 个: {selected_file}"
)
return (str(selected_file),)
except Exception as e:
logger.error(f"文件夹遍历失败: {e}")
if isinstance(e, RuntimeError):
raise
else:
raise RuntimeError(f"文件夹遍历错误: {str(e)}")
class PlugAndPlayWebhook:
"""即插即用Webhook节点连上线就能转发数据"""
"""
即插即用Webhook节点
连上线就能转发数据到指定的Webhook地址。
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"webhook_url": ("STRING", {"default": "http://127.0.0.1:8010/handler/webhook",
"placeholder": "https://your-api.com/webhook"}),
"image_url": ("STRING", {"default": "",
"placeholder": "图片的url"}),
"webhook_url": (
"STRING",
{
"default": "http://127.0.0.1:8010/handler/webhook",
"placeholder": "https://your-api.com/webhook",
},
),
"image_url": ("STRING", {"default": "", "placeholder": "图片的url"}),
},
"optional": {
"prompt_id": ("STRING", {"default": ""}),
},
"hidden": {"prompt": "PROMPT", "extra_pnginfo": "EXTRA_PNGINFO", "unique_id": "UNIQUE_ID"},
"hidden": {
"prompt": "PROMPT",
"extra_pnginfo": "EXTRA_PNGINFO",
"unique_id": "UNIQUE_ID",
},
}
RETURN_TYPES = ()
@@ -281,43 +433,85 @@ class PlugAndPlayWebhook:
OUTPUT_NODE = True
CATEGORY = "不忘科技-自定义节点🚩/工具"
def send(self, webhook_url, image_url, prompt_id="", prompt=None, extra_pnginfo=None, unique_id=None):
if not webhook_url:
raise ValueError("❌ 请填写Webhook URL")
def send(
self,
webhook_url: str,
image_url: str,
prompt_id: str = "",
prompt=None,
extra_pnginfo=None,
unique_id=None,
) -> Tuple:
"""
发送数据到Webhook
# 使用传入的prompt_id如果没有则用unique_id
final_prompt_id = prompt_id or unique_id or "unknown"
Args:
webhook_url: Webhook URL
image_url: 图片URL
prompt_id: 提示ID
prompt: 提示信息
extra_pnginfo: 额外PNG信息
unique_id: 唯一ID
# 准备发送的数据
data = {
"img_base64": image_url,
"format": "png",
"image_url": image_url,
"prompt_id": final_prompt_id,
"timestamp": time.time()
}
Returns:
tuple: 空元组
# 发送Webhook
Raises:
ValueError: URL为空时抛出异常
"""
try:
response = requests.post(webhook_url, json=data)
# 参数验证
if not webhook_url or not webhook_url.strip():
raise ValueError("❌ 请填写Webhook URL")
# 使用传入的prompt_id如果没有则用unique_id
final_prompt_id = prompt_id or unique_id or "unknown"
# 准备发送的数据
data = {
"img_base64": image_url,
"format": "png",
"image_url": image_url,
"prompt_id": final_prompt_id,
"timestamp": time.time(),
}
logger.info(f"准备发送Webhook数据到: {webhook_url}")
# 发送Webhook
response = requests.post(webhook_url.strip(), json=data, timeout=30)
response.raise_for_status()
print(f'发送的数据:{data}')
logger.info(f"Webhook发送成功响应状态: {response.status_code}")
logger.debug(f"发送的数据: {data}")
except requests.RequestException as e:
logger.error(f"❌ Webhook发送失败: {str(e)}")
# 不抛出异常,避免中断工作流
except Exception as e:
print(f"发送失败: {str(e)}")
logger.error(f"Webhook发送异常: {str(e)}")
# 不抛出异常,避免中断工作流
# 终端节点,无需返回
return ()
class TaskIdGenerate:
"""TaskID生成器用户可传入或自动生成TaskID"""
"""
TaskID生成器
用户可传入自定义TaskID或自动生成UUID。
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {},
"optional": {
"custom_task_id": ("STRING", {"default": "", "placeholder": "留空则自动生成"}),
"custom_task_id": (
"STRING",
{"default": "", "placeholder": "留空则自动生成"},
),
},
}
@@ -327,14 +521,52 @@ class TaskIdGenerate:
OUTPUT_NODE = False
CATEGORY = "不忘科技-自定义节点🚩/工具"
def generate_task_id(self, custom_task_id=""):
if custom_task_id and custom_task_id.strip():
# 用户输入了自定义ID
task_id = custom_task_id.strip()
print(f"📝 使用自定义TaskID: {task_id}")
else:
# 自动生成UUID
task_id = str(uuid.uuid4())
print(f"🎲 自动生成TaskID: {task_id}")
def generate_task_id(self, custom_task_id: str = "") -> Tuple[str]:
"""
生成或使用自定义TaskID
return (task_id,)
Args:
custom_task_id: 自定义TaskID
Returns:
tuple: 包含TaskID的元组
"""
try:
if custom_task_id and custom_task_id.strip():
# 用户输入了自定义ID
task_id = custom_task_id.strip()
logger.info(f"📝 使用自定义TaskID: {task_id}")
else:
# 自动生成UUID
task_id = str(uuid.uuid4())
logger.info(f"🎲 自动生成TaskID: {task_id}")
return (task_id,)
except Exception as e:
logger.error(f"TaskID生成失败: {e}")
# 发生异常时生成一个基础UUID
fallback_id = str(uuid.uuid4())
logger.warning(f"使用备用TaskID: {fallback_id}")
return (fallback_id,)
# 节点映射字典
NODE_CLASS_MAPPINGS = {
"LogToDB": LogToDB,
"VodToLocalNode": VodToLocalNode,
"UnloadAllModels": UnloadAllModels,
"TraverseFolder": TraverseFolder,
"PlugAndPlayWebhook": PlugAndPlayWebhook,
"TaskIdGenerate": TaskIdGenerate,
}
# 节点显示名称映射
NODE_DISPLAY_NAME_MAPPINGS = {
"LogToDB": "数据库日志记录",
"VodToLocalNode": "VOD文件下载",
"UnloadAllModels": "卸载所有模型",
"TraverseFolder": "文件夹遍历",
"PlugAndPlayWebhook": "Webhook通知",
"TaskIdGenerate": "TaskID生成器",
}

View File

@@ -13,10 +13,8 @@ from torch import Tensor
def task_submit(uid, video, audio, heygem_url):
"""Submit a task to the API"""
task_submit_api = f'{heygem_url}/easy/submit'
result_json = {
'status': False, 'data': {}, 'msg': ''
}
task_submit_api = f"{heygem_url}/easy/submit"
result_json = {"status": False, "data": {}, "msg": ""}
try:
data = {
"code": uid,
@@ -24,104 +22,112 @@ def task_submit(uid, video, audio, heygem_url):
"audio_url": audio,
"chaofen": 1,
"watermark_switch": 0,
"pn": 1
"pn": 1,
}
loguru.logger.info(f'data={data}')
loguru.logger.info(f"data={data}")
with httpx.Client() as client:
resp = client.post(task_submit_api, json=data)
resp_dict = resp.json()
loguru.logger.info(f'submit data: {resp_dict}')
if resp_dict['code'] != 10000:
result_json['status'] = False
result_json['msg'] = result_json['msg']
loguru.logger.info(f"submit data: {resp_dict}")
if resp_dict["code"] != 10000:
result_json["status"] = False
result_json["msg"] = result_json["msg"]
else:
result_json['status'] = True
result_json['data'] = uid
result_json['msg'] = '任务提交成功'
result_json["status"] = True
result_json["data"] = uid
result_json["msg"] = "任务提交成功"
except Exception as e:
loguru.logger.info(f'submit task fail case by:{str(e)}')
loguru.logger.info(f"submit task fail case by:{str(e)}")
raise RuntimeError(str(e))
return result_json
def query_task_progress(heygem_url, heygem_temp_path, task_id: str, interval: int = 10, timeout: int = 60 * 15):
def query_task_progress(
heygem_url,
heygem_temp_path,
task_id: str,
interval: int = 10,
timeout: int = 60 * 15,
):
"""Query task progress and wait for completion"""
result_json = {'status': False, 'data': {}, 'msg': ''}
result_json = {"status": False, "data": {}, "msg": ""}
def query_result(t_id: str):
tmp_dict = {'status': True, 'data': dict(), 'msg': ''}
tmp_dict = {"status": True, "data": dict(), "msg": ""}
try:
query_task_url = f'{heygem_url}/easy/query'
params = {
'code': t_id
}
query_task_url = f"{heygem_url}/easy/query"
params = {"code": t_id}
with httpx.Client() as client:
resp = client.get(query_task_url, params=params)
resp_dict = resp.json()
status_code = resp_dict['code']
status_code = resp_dict["code"]
if status_code in (9999, 10002, 10003, 10001):
tmp_dict['status'] = False
tmp_dict['msg'] = resp_dict['msg']
tmp_dict["status"] = False
tmp_dict["msg"] = resp_dict["msg"]
elif status_code == 10000:
loguru.logger.info(f'query task data: {json.dumps(resp_dict)}')
status_code = resp_dict['data'].get('status', 1)
loguru.logger.info(f"query task data: {json.dumps(resp_dict)}")
status_code = resp_dict["data"].get("status", 1)
if status_code == 3:
tmp_dict['status'] = False
tmp_dict['msg'] = resp_dict['data']['msg']
tmp_dict["status"] = False
tmp_dict["msg"] = resp_dict["data"]["msg"]
else:
process = resp_dict['data'].get('progress', 20)
process = resp_dict["data"].get("progress", 20)
if status_code == 2:
process = 100
else:
process = process
result = resp_dict['data'].get('result', '')
tmp_dict['data'] = {'progress': process,
'path': result,
}
result = resp_dict["data"].get("result", "")
tmp_dict["data"] = {
"progress": process,
"path": result,
}
else:
pass
except Exception as e:
loguru.logger.info(f'query task fail case by:{str(e)}')
loguru.logger.info(f"query task fail case by:{str(e)}")
raise RuntimeError(str(e))
return tmp_dict
end = time.time() + timeout
while time.time() < end:
tmp_result = query_result(task_id)
if not tmp_result['status'] or tmp_result['data'].__eq__({}):
result_json['status'] = False
result_json['msg'] = tmp_result['msg']
if not tmp_result["status"] or tmp_result["data"].__eq__({}):
result_json["status"] = False
result_json["msg"] = tmp_result["msg"]
break
else:
process = tmp_result['data']['progress']
loguru.logger.info(f'query task progress :{process}')
if tmp_result['data']['progress'] < 100:
process = tmp_result["data"]["progress"]
loguru.logger.info(f"query task progress :{process}")
if tmp_result["data"]["progress"] < 100:
time.sleep(interval)
loguru.logger.info(f'wait next interval:{interval}')
loguru.logger.info(f"wait next interval:{interval}")
else:
p = tmp_result['data']['path']
p = p.replace('/', '').replace('\\', '')
result_json['data'] = "%s/%s"%(heygem_temp_path, p)
result_json['status'] = True
p = tmp_result["data"]["path"]
p = p.replace("/", "").replace("\\", "")
result_json["data"] = "%s/%s" % (heygem_temp_path, p)
result_json["status"] = True
return result_json
return result_json
def path_convert(path):
if ":" in path:
path = path.replace(os.sep,"/").split(":")
path = path.replace(os.sep, "/").split(":")
path[0] = path[0].lower()
path[1] = path[1][1:]
path = "/".join(["/mnt",*path])
path = "/".join(["/mnt", *path])
return path
def result_path_convert(result_path:str):
def result_path_convert(result_path: str):
if result_path.startswith("/"):
result_path = result_path.replace("/","\\")
result_path = result_path.replace("/", "\\")
result_path = r"\\wsl.localhost\Debian" + result_path
return result_path
class HeyGemF2F:
"""HeyGem 嘴型同步"""
@@ -132,8 +138,8 @@ class HeyGemF2F:
"video": ("IMAGE", {"forceInput": True}),
"audio": ("AUDIO", {"forceInput": True}),
"heygem_url": ("STRING", {"default": "http://127.0.0.1:8383"}),
"heygem_temp_path": ("STRING", {"default":"/code/data/temp"}),
"is_Windows": ("BOOLEAN", {"default": False})
"heygem_temp_path": ("STRING", {"default": "/code/data/temp"}),
"is_Windows": ("BOOLEAN", {"default": False}),
}
}
@@ -142,36 +148,49 @@ class HeyGemF2F:
FUNCTION = "f2f"
CATEGORY = "不忘科技-自定义节点🚩/视频/口型"
def f2f(self, video:Tensor, audio:dict, heygem_url:str, heygem_temp_path:str, is_Windows:bool):
def f2f(
self,
video: Tensor,
audio: dict,
heygem_url: str,
heygem_temp_path: str,
is_Windows: bool,
):
uid = str(uuid.uuid4())
video_path = os.path.join(os.path.dirname(__file__),"%s.mp4" % uid)
audio_path = os.path.join(os.path.dirname(__file__),"%s.wav" % uid)
video_path = os.path.join(os.path.dirname(__file__), "%s.mp4" % uid)
audio_path = os.path.join(os.path.dirname(__file__), "%s.wav" % uid)
try:
try:
torchvision.io.write_video(video_path, video.mul_(255).int(),25)
torchaudio.save(audio_path, audio["waveform"].squeeze(0), audio["sample_rate"], True)
torchvision.io.write_video(video_path, video.mul_(255).int(), 25)
torchaudio.save(
audio_path, audio["waveform"].squeeze(0), audio["sample_rate"], True
)
except:
traceback.print_exc()
raise RuntimeError("Save Temp File Error! ")
submit_result = task_submit(uid, path_convert(video_path), path_convert(audio_path), heygem_url)
if not submit_result['status']:
submit_result = task_submit(
uid, path_convert(video_path), path_convert(audio_path), heygem_url
)
if not submit_result["status"]:
return {
'status': False,
'data': {},
'msg': f"Task submission failed: {submit_result['msg']}"
"status": False,
"data": {},
"msg": f"Task submission failed: {submit_result['msg']}",
}
task_id = submit_result['data']
loguru.logger.info(f'Submitted task: {task_id}')
task_id = submit_result["data"]
loguru.logger.info(f"Submitted task: {task_id}")
# Query task progress
progress_result = query_task_progress(heygem_url, heygem_temp_path, task_id, interval=5)
progress_result = query_task_progress(
heygem_url, heygem_temp_path, task_id, interval=5
)
if not progress_result['status']:
if not progress_result["status"]:
raise RuntimeError(f"Task processing failed: {progress_result['msg']}")
# Return the file for download
file_path = progress_result['data']
file_path = progress_result["data"]
if is_Windows:
file_path = result_path_convert(file_path)
if os.path.exists(file_path):
@@ -183,11 +202,12 @@ class HeyGemF2F:
raise Exception(str(e))
finally:
try:
os.remove(os.path.join(os.path.dirname(__file__),"%s.mp4" % uid))
os.remove(os.path.join(os.path.dirname(__file__),"%s.wav" % uid))
os.remove(os.path.join(os.path.dirname(__file__), "%s.mp4" % uid))
os.remove(os.path.join(os.path.dirname(__file__), "%s.wav" % uid))
except:
pass
class HeyGemF2FFromFile:
"""HeyGem 嘴型同步 直接读取文件"""
@@ -198,8 +218,8 @@ class HeyGemF2FFromFile:
"video": ("STRING", {"forceInput": True}),
"audio": ("STRING", {"forceInput": True}),
"heygem_url": ("STRING", {"default": "http://127.0.0.1:8383"}),
"heygem_temp_path": ("STRING", {"default":"/code/data/temp"}),
"is_Windows": ("BOOLEAN", {"default": False})
"heygem_temp_path": ("STRING", {"default": "/code/data/temp"}),
"is_Windows": ("BOOLEAN", {"default": False}),
}
}
@@ -208,28 +228,35 @@ class HeyGemF2FFromFile:
FUNCTION = "f2f"
CATEGORY = "不忘科技-自定义节点🚩/视频/口型"
def f2f(self, video:str, audio:str, heygem_url:str, heygem_temp_path:str, is_Windows:bool):
def f2f(
self,
video: str,
audio: str,
heygem_url: str,
heygem_temp_path: str,
is_Windows: bool,
):
uid = str(uuid.uuid4())
try:
submit_result = task_submit(uid, video, audio, heygem_url)
if not submit_result['status']:
if not submit_result["status"]:
return {
'status': False,
'data': {},
'msg': f"Task submission failed: {submit_result['msg']}"
"status": False,
"data": {},
"msg": f"Task submission failed: {submit_result['msg']}",
}
task_id = submit_result['data']
loguru.logger.info(f'Submitted task: {task_id}')
task_id = submit_result["data"]
loguru.logger.info(f"Submitted task: {task_id}")
# Query task progress
progress_result = query_task_progress(heygem_url,heygem_temp_path,task_id)
progress_result = query_task_progress(heygem_url, heygem_temp_path, task_id)
if not progress_result['status']:
if not progress_result["status"]:
raise RuntimeError(f"Task processing failed: {progress_result['msg']}")
# Return the file for download
file_path = progress_result['data']
file_path = progress_result["data"]
if is_Windows:
file_path = result_path_convert(file_path)
if os.path.exists(file_path):
@@ -238,4 +265,4 @@ class HeyGemF2FFromFile:
raise FileNotFoundError(f"Output file not found at {file_path}")
except Exception as e:
loguru.logger.error(f"Error processing request: {str(e)}")
raise Exception(str(e))
raise Exception(str(e))

View File

@@ -15,7 +15,7 @@ import folder_paths
import loguru
import torchvision.io
video_extensions = ['webm', 'mp4', 'mkv', 'gif', 'mov']
video_extensions = ["webm", "mp4", "mkv", "gif", "mov"]
class VideoCut:
@@ -26,7 +26,12 @@ class VideoCut:
return {
"required": {
"video_path": (
"STRING", {"placeholder": "X://insert/path/here.mp4", "vhs_path_extensions": video_extensions}),
"STRING",
{
"placeholder": "X://insert/path/here.mp4",
"vhs_path_extensions": video_extensions,
},
),
"start": ("STRING", {"default": "00:00:00.000"}),
"end": ("STRING", {"default": "00:00:10.000"}),
},
@@ -70,7 +75,8 @@ class VideoCut:
output = (
os.sep.join([*video_path.split(os.sep)[:-1], output_name])
.replace(
os.sep.join(["ComfyUI", "input"]), os.sep.join(["ComfyUI", "output"])
os.sep.join(["ComfyUI", "input"]),
os.sep.join(["ComfyUI", "output"]),
)
.replace(" ", "")
)
@@ -96,7 +102,7 @@ class VideoCut:
"-force_key_frames",
"expr:gte(t, n_forced * 1)",
"-v",
"-8"
"-8",
]
},
)
@@ -124,7 +130,13 @@ class VideoCut:
except:
pass
return (
video / 255.0, {"waveform": audio, "sample_rate": info["audio_fps"]} if "audio_fps" in info else None,)
video / 255.0,
(
{"waveform": audio, "sample_rate": info["audio_fps"]}
if "audio_fps" in info
else None
),
)
except:
traceback.print_exc()
raise Exception("Cut Failed")
@@ -138,7 +150,12 @@ class VideoCutByFramePoint:
return {
"required": {
"video_path": (
"STRING", {"placeholder": "X://insert/path/here.mp4", "vhs_path_extensions": video_extensions}),
"STRING",
{
"placeholder": "X://insert/path/here.mp4",
"vhs_path_extensions": video_extensions,
},
),
"start_point": ("FLOAT", {"default": "0.0"}),
"duration": ("FLOAT", {"default": "10.0"}),
"fps": ("INT", {"default": "25"}),
@@ -184,7 +201,8 @@ class VideoCutByFramePoint:
output = (
os.sep.join([os.path.dirname(__file__), output_name])
.replace(
os.sep.join(["ComfyUI", "input"]), os.sep.join(["ComfyUI", "output"])
os.sep.join(["ComfyUI", "input"]),
os.sep.join(["ComfyUI", "output"]),
)
.replace(" ", "")
)
@@ -212,7 +230,7 @@ class VideoCutByFramePoint:
"-r" if force_match_fps else "",
"%d" % fps if force_match_fps else "",
"-v",
"-8"
"-8",
]
},
)
@@ -239,7 +257,10 @@ class VideoCutByFramePoint:
os.remove(output)
except:
pass
return (video / 255.0, {"waveform": audio, "sample_rate": info["audio_fps"]},)
return (
video / 255.0,
{"waveform": audio, "sample_rate": info["audio_fps"]},
)
except:
traceback.print_exc()
raise Exception("Cut Failed")
@@ -253,7 +274,12 @@ class VideoChangeFPS:
return {
"required": {
"video_path": (
"STRING", {"placeholder": "X://insert/path/here.mp4", "vhs_path_extensions": video_extensions}),
"STRING",
{
"placeholder": "X://insert/path/here.mp4",
"vhs_path_extensions": video_extensions,
},
),
"fps": ("INT", {"default": 30}),
},
}
@@ -270,11 +296,16 @@ class VideoChangeFPS:
def get_media_duration(self, input_file, stream_type="v"):
"""获取视频或音频的精确时长(秒)"""
cmd = [
"ffprobe", "-v", "error",
"-select_streams", f"{stream_type}:0",
"-show_entries", "stream=duration",
"-of", "csv=p=0",
input_file
"ffprobe",
"-v",
"error",
"-select_streams",
f"{stream_type}:0",
"-show_entries",
"stream=duration",
"-of",
"csv=p=0",
input_file,
]
return float(subprocess.check_output(cmd).decode().strip())
@@ -284,7 +315,7 @@ class VideoChangeFPS:
inputs={input_video: None},
outputs={
output_temp: f"-y -vf fps={target_fps} -c:v libx264 -c:a aac -preset slow -crf 16"
}
},
)
print("调整FPS命令:", ff.cmd)
with open("user/ffmpeg.txt", "a") as log:
@@ -316,7 +347,7 @@ class VideoChangeFPS:
-map "[v]" -map "[a]"
-c:v libx264 -c:a aac -preset slow -crf 16
"""
}
},
)
print("对齐音频命令:", ff.cmd)
with open("user/ffmpeg.txt", "a") as log:
@@ -326,11 +357,22 @@ class VideoChangeFPS:
def changeFps(self, video_path, fps):
try:
if not (video_path.startswith("/") or video_path.startswith("output/") or video_path[1] == ":"):
if not (
video_path.startswith("/")
or video_path.startswith("output/")
or video_path[1] == ":"
):
video_path = "output/" + video_path
loguru.logger.info("Processing video: %s" % video_path)
output_temp = ".".join([video_path.split(".")[-2] + "-%dfps-temp" % fps, video_path.split(".")[-1]])
output = ".".join([video_path.split(".")[-2] + "-%dfps" % fps, video_path.split(".")[-1]])
output_temp = ".".join(
[
video_path.split(".")[-2] + "-%dfps-temp" % fps,
video_path.split(".")[-1],
]
)
output = ".".join(
[video_path.split(".")[-2] + "-%dfps" % fps, video_path.split(".")[-1]]
)
# 分步执行
self.adjust_video_fps(video_path, output_temp, fps) # 第一步调整FPS
self.align_audio_to_video(output_temp, output) # 第二步:对齐音频
@@ -338,12 +380,17 @@ class VideoChangeFPS:
# 校验结果
final_video_dur = self.get_media_duration(output, "v")
final_audio_dur = self.get_media_duration(output, "a")
print("video_duration:", final_video_dur, "audio_duration:", final_audio_dur)
print(
"video_duration:", final_video_dur, "audio_duration:", final_audio_dur
)
if abs(final_video_dur - final_audio_dur) > 0.01:
loguru.logger.warning(
f"音视频长度未对齐!视频长度: {final_video_dur:.3f}s, 音频长度: {final_audio_dur:.3f}s")
f"音视频长度未对齐!视频长度: {final_video_dur:.3f}s, 音频长度: {final_audio_dur:.3f}s"
)
else:
loguru.logger.success(f"处理成功!视频长度: {final_video_dur:.3f}s, 音频长度: {final_audio_dur:.3f}s")
loguru.logger.success(
f"处理成功!视频长度: {final_video_dur:.3f}s, 音频长度: {final_audio_dur:.3f}s"
)
try:
os.remove(output_temp)
except:
@@ -355,7 +402,7 @@ class VideoChangeFPS:
def validate_time_format(time_str):
pattern = r'^([0-1][0-9]|2[0-3]):([0-5][0-9]):([0-5][0-9]|\d{1,2}).(\d{3})$'
pattern = r"^([0-1][0-9]|2[0-3]):([0-5][0-9]):([0-5][0-9]|\d{1,2}).(\d{3})$"
return bool(re.match(pattern, time_str))
@@ -377,7 +424,10 @@ class VideoStartPointDurationCompute:
},
}
RETURN_TYPES = ("FLOAT", "FLOAT",)
RETURN_TYPES = (
"FLOAT",
"FLOAT",
)
RETURN_NAMES = ("起始帧位", "帧数")
FUNCTION = "compute"
@@ -386,7 +436,9 @@ class VideoStartPointDurationCompute:
def compute(self, start_time, audio, fps, end_padding):
if not validate_time_format(start_time):
raise ValueError("start_time或者end_time时间格式不对start_time or end_time is not in time format")
raise ValueError(
"start_time或者end_time时间格式不对start_time or end_time is not in time format"
)
time_format = "%H:%M:%S.%f"
start_dt = datetime.strptime(start_time, time_format)
@@ -396,7 +448,10 @@ class VideoStartPointDurationCompute:
loguru.logger.info("audio duration %.3f s" % duration)
duration = duration + end_padding
loguru.logger.info("audio duration with padding %.3f s" % duration)
return (start_point, duration * fps,)
return (
start_point,
duration * fps,
)
def merge_videos(input_paths: List[str], output_path: str) -> str:
@@ -413,7 +468,7 @@ def merge_videos(input_paths: List[str], output_path: str) -> str:
raise FileNotFoundError(f"输入文件不存在: {path}")
# 创建临时文件列表
temp_filelist = os.path.join(os.path.dirname(__file__),"filelist.txt")
temp_filelist = os.path.join(os.path.dirname(__file__), "filelist.txt")
with open(temp_filelist, "w", encoding="utf-8") as f:
for path in input_paths:
# 处理路径中的引号和特殊字符
@@ -424,19 +479,18 @@ def merge_videos(input_paths: List[str], output_path: str) -> str:
# 使用ffmpeg执行拼接操作
cmd = [
"ffmpeg",
"-f", "concat",
"-safe", "0",
"-i", str(temp_filelist),
"-c", "copy",
output_path
"-f",
"concat",
"-safe",
"0",
"-i",
str(temp_filelist),
"-c",
"copy",
output_path,
]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True
)
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
print(f"视频拼接成功,输出文件: {output_path}")
print("ffmpeg 输出:", result.stderr)
@@ -454,9 +508,7 @@ class VideoMerge:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"video_list": ("STRING", {"default": "[]"})
},
"required": {"video_list": ("STRING", {"default": "[]"})},
}
RETURN_TYPES = ("STRING",)
@@ -469,4 +521,11 @@ class VideoMerge:
def process(self, video_list):
if isinstance(video_list, str):
video_list = json.loads(video_list)
return (merge_videos(video_list, os.path.join(folder_paths.get_output_directory(), f"merged_{uuid.uuid4()}.mp4")),)
return (
merge_videos(
video_list,
os.path.join(
folder_paths.get_output_directory(), f"merged_{uuid.uuid4()}.mp4"
),
),
)

150
utils/config_utils.py Normal file
View File

@@ -0,0 +1,150 @@
import os
import yaml
from typing import Dict, Any, Optional
from pathlib import Path
class Config:
"""统一的配置管理类"""
def __init__(self):
self._config = self._load_config()
self._init_config_attributes()
def _load_config(self) -> Dict[str, Any]:
"""加载配置,优先从环境变量读取,否则从配置文件读取"""
# 优先从环境变量读取
if "aws_key_id" in os.environ:
return os.environ
# 从配置文件读取
config_path = self._get_config_path()
if config_path.exists():
try:
with open(config_path, encoding="utf-8", mode="r") as f:
return yaml.load(f, Loader=yaml.FullLoader)
except Exception as e:
print(f"读取配置文件失败: {e}")
return {}
else:
print(f"配置文件不存在: {config_path}")
return {}
def _get_config_path(self) -> Path:
"""获取配置文件路径"""
# 获取当前文件所在目录的上级目录
current_dir = Path(__file__).parent.parent
return current_dir / "config.yaml"
def _init_config_attributes(self):
"""初始化配置属性"""
# 腾讯云COS配置
self.cos_secret_id = self._config.get("cos_secret_id")
self.cos_secret_key = self._config.get("cos_secret_key")
self.cos_region = self._config.get("cos_region")
self.cos_sucai_bucket_name = self._config.get("cos_sucai_bucket_name")
# AWS S3配置
self.aws_key_id = self._config.get("aws_key_id")
self.aws_access_key = self._config.get("aws_access_key")
# JM API配置
self.jm_api_key = self._config.get("jm_api_key")
# VOD配置使用COS相同的密钥
self.vod_secret_id = self._config.get("vod_secret_id") or self.cos_secret_id
self.vod_secret_key = self._config.get("vod_secret_key") or self.cos_secret_key
self.vod_region = self._config.get("vod_region") or self.cos_region
# 其他配置项
self._other_configs = {
k: v
for k, v in self._config.items()
if k
not in [
"cos_secret_id",
"cos_secret_key",
"cos_region",
"cos_sucai_bucket_name",
"aws_key_id",
"aws_access_key",
"jm_api_key",
"vod_secret_id",
"vod_secret_key",
"vod_region",
]
}
def get(self, key: str, default: Any = None) -> Any:
"""获取配置值"""
return self._config.get(key, default)
def get_config(self, key: str, default: Any = None) -> Any:
"""获取配置值(别名方法)"""
return self.get(key, default)
def get_cos_config(self) -> Dict[str, str]:
"""获取腾讯云COS配置"""
return {
"secret_id": self.cos_secret_id,
"secret_key": self.cos_secret_key,
"region": self.cos_region,
"bucket_name": self.cos_sucai_bucket_name,
}
def get_aws_config(self) -> Dict[str, str]:
"""获取AWS S3配置"""
return {
"access_key_id": self.aws_key_id,
"secret_access_key": self.aws_access_key,
}
def get_jm_config(self) -> Dict[str, str]:
"""获取JM API配置"""
return {"api_key": self.jm_api_key}
def get_vod_config(self) -> Dict[str, str]:
"""获取腾讯云VOD配置"""
return {
"secret_id": self.vod_secret_id,
"secret_key": self.vod_secret_key,
"region": self.vod_region,
}
def has_cos_config(self) -> bool:
"""检查是否有完整的腾讯云COS配置"""
return all([self.cos_secret_id, self.cos_secret_key, self.cos_region])
def has_aws_config(self) -> bool:
"""检查是否有完整的AWS S3配置"""
return all([self.aws_key_id, self.aws_access_key])
def has_jm_config(self) -> bool:
"""检查是否有JM API配置"""
return bool(self.jm_api_key)
def has_vod_config(self) -> bool:
"""检查是否有完整的腾讯云VOD配置"""
return all([self.vod_secret_id, self.vod_secret_key, self.vod_region])
def reload(self):
"""重新加载配置"""
self._config = self._load_config()
self._init_config_attributes()
def to_dict(self) -> Dict[str, Any]:
"""将配置转换为字典"""
return self._config.copy()
def __str__(self) -> str:
"""字符串表示"""
config_info = {
"cos_config": self.get_cos_config() if self.has_cos_config() else "不完整",
"aws_config": self.get_aws_config() if self.has_aws_config() else "不完整",
"jm_config": self.get_jm_config() if self.has_jm_config() else "不完整",
}
return f"Config({config_info})"
# 创建全局配置实例
config = Config()

View File

@@ -0,0 +1,279 @@
"""
存储管理器初始化模块
本模块负责初始化和配置存储管理器,注册所有可用的存储提供者工厂。
提供统一的存储服务访问入口,简化使用复杂度。
使用方式:
from utils.storage_manager_init import get_storage_manager, get_provider
# 获取存储管理器
manager = get_storage_manager()
# 直接获取默认存储提供者
provider = get_provider()
# 获取指定类型的存储提供者
s3_provider = get_provider("s3", {"access_key_id": "...", "secret_access_key": "..."})
"""
from typing import Any, Dict, Optional
import loguru
from ..config_utils import config
from .providers.cos_provider import COSStorageFactory
from .providers.s3_provider import S3StorageFactory
from .providers.vod_provider import VODStorageFactory
from .storage_interface import (
DownloadResult,
StorageProvider,
UploadResult,
storage_manager,
)
def initialize_storage_manager():
"""
初始化存储管理器,注册所有可用的存储工厂
注册的存储类型:
- S3: AWS S3存储服务
- COS: 腾讯云COS存储服务
- VOD: 腾讯云VOD视频点播服务
"""
try:
# 注册S3存储工厂
s3_factory = S3StorageFactory()
for storage_type in s3_factory.get_supported_types():
storage_manager.register_factory(storage_type, s3_factory)
# 注册COS存储工厂
cos_factory = COSStorageFactory()
for storage_type in cos_factory.get_supported_types():
storage_manager.register_factory(storage_type, cos_factory)
# 注册VOD存储工厂
vod_factory = VODStorageFactory()
for storage_type in vod_factory.get_supported_types():
storage_manager.register_factory(storage_type, vod_factory)
loguru.logger.info("存储管理器初始化完成")
loguru.logger.info(f"支持的存储类型: {storage_manager.get_supported_types()}")
except Exception as e:
loguru.logger.error(f"存储管理器初始化失败: {e}")
raise
def create_default_providers():
"""
创建默认的存储提供者
根据配置文件中的可用配置自动创建存储提供者实例。
优先级: S3 > COS
"""
providers_created = []
# 尝试创建S3提供者
if config.has_aws_config():
try:
aws_config = config.get_aws_config()
s3_provider = storage_manager.create_provider(
"s3", aws_config, "default_s3"
)
providers_created.append("S3")
loguru.logger.info("默认S3存储提供者创建成功")
except Exception as e:
loguru.logger.warning(f"S3存储提供者创建失败: {e}")
# 尝试创建COS提供者
if config.has_cos_config():
try:
cos_config = config.get_cos_config()
cos_provider = storage_manager.create_provider(
"cos", cos_config, "default_cos"
)
providers_created.append("COS")
loguru.logger.info("默认COS存储提供者创建成功")
except Exception as e:
loguru.logger.warning(f"COS存储提供者创建失败: {e}")
# 尝试创建VOD提供者
if config.has_vod_config():
try:
vod_config = config.get_vod_config()
vod_provider = storage_manager.create_provider(
"vod", vod_config, "default_vod"
)
providers_created.append("VOD")
loguru.logger.info("默认VOD存储提供者创建成功")
except Exception as e:
loguru.logger.warning(f"VOD存储提供者创建失败: {e}")
if providers_created:
loguru.logger.info(f"默认存储提供者创建完成: {', '.join(providers_created)}")
else:
loguru.logger.warning("未能创建任何默认存储提供者,请检查配置文件")
def get_storage_manager():
"""
获取已初始化的存储管理器实例
Returns:
StorageManager: 存储管理器实例
"""
if not storage_manager.get_supported_types():
initialize_storage_manager()
create_default_providers()
return storage_manager
def get_provider(
provider_type: Optional[str] = None, config_dict: Optional[Dict[str, Any]] = None
) -> StorageProvider:
"""
获取存储提供者实例
Args:
provider_type: 存储类型(如:"s3", "cos"为None时返回默认提供者
config_dict: 自定义配置字典为None时使用全局配置
Returns:
StorageProvider: 存储提供者实例
Raises:
ValueError: 当找不到合适的存储提供者时抛出异常
"""
manager = get_storage_manager()
# 如果没有指定类型,返回默认提供者
if provider_type is None:
try:
return manager.get_provider()
except ValueError:
# 如果没有默认提供者,尝试创建一个
if config.has_aws_config():
aws_config = config.get_aws_config()
return manager.create_provider("s3", aws_config)
elif config.has_cos_config():
cos_config = config.get_cos_config()
return manager.create_provider("cos", cos_config)
elif config.has_vod_config():
vod_config = config.get_vod_config()
return manager.create_provider("vod", vod_config)
else:
raise ValueError("无法找到有效的存储配置,请检查配置文件")
# 创建指定类型的提供者
if config_dict:
provider_id = f"{provider_type}_{hash(str(config_dict))}"
return manager.create_provider(provider_type, config_dict, provider_id)
else:
# 尝试从已有提供者中获取
try:
return manager.get_provider(f"default_{provider_type}")
except ValueError:
# 如果不存在,使用全局配置创建
if provider_type in ["s3", "aws", "amazon"] and config.has_aws_config():
aws_config = config.get_aws_config()
return manager.create_provider(
"s3", aws_config, f"default_{provider_type}"
)
elif (
provider_type in ["cos", "qcloud", "tencent"]
and config.has_cos_config()
):
cos_config = config.get_cos_config()
return manager.create_provider(
"cos", cos_config, f"default_{provider_type}"
)
elif (
provider_type in ["vod", "tencent_vod", "qcloud_vod"]
and config.has_vod_config()
):
vod_config = config.get_vod_config()
return manager.create_provider(
"vod", vod_config, f"default_{provider_type}"
)
else:
raise ValueError(
f"无法为类型 {provider_type} 创建存储提供者,缺少配置信息"
)
def get_available_storage_types() -> Dict[str, bool]:
"""
获取所有可用的存储类型及其可用状态
Returns:
Dict[str, bool]: 存储类型及其可用状态的字典
"""
manager = get_storage_manager()
supported_types = manager.get_supported_types()
availability = {}
for storage_type in supported_types:
try:
# 尝试创建提供者来检查可用性
if storage_type in ["s3", "aws", "amazon"]:
availability[storage_type] = config.has_aws_config()
elif storage_type in ["cos", "qcloud", "tencent"]:
availability[storage_type] = config.has_cos_config()
elif storage_type in ["vod", "tencent_vod", "qcloud_vod"]:
availability[storage_type] = config.has_vod_config()
else:
availability[storage_type] = False
except Exception:
availability[storage_type] = False
return availability
def validate_storage_config(storage_type: str, config_dict: Dict[str, Any]) -> bool:
"""
验证存储配置的有效性
Args:
storage_type: 存储类型
config_dict: 配置字典
Returns:
bool: 配置是否有效
"""
try:
manager = get_storage_manager()
# 尝试创建提供者以验证配置
test_provider = manager.create_provider(
storage_type, config_dict, "test_provider"
)
# 清理测试提供者
# 注意这里我们不从manager中移除因为create_provider会缓存它
return True
except Exception as e:
loguru.logger.error(f"存储配置验证失败: {e}")
return False
# 在模块加载时自动初始化
try:
initialize_storage_manager()
create_default_providers()
except Exception as e:
loguru.logger.error(f"存储管理器自动初始化失败: {e}")
# 导出主要接口
__all__ = [
"get_storage_manager",
"get_provider",
"get_available_storage_types",
"validate_storage_config",
"UploadResult",
"DownloadResult",
]

View File

@@ -0,0 +1,544 @@
"""
腾讯云COS存储提供者实现
本模块实现了腾讯云COS的具体存储操作继承自抽象存储接口。
提供完整的COS文件上传、下载、删除等功能。
特性:
- 支持文件、字节数据、PyTorch张量的上传
- 完整的错误处理和重试机制
- 统一的日志记录
- 与S3接口保持一致的API设计
"""
import os
from typing import Optional, Dict, Any, List
import torch
import loguru
from qcloud_cos import CosConfig, CosS3Client, CosClientError, CosServiceError
from ..storage_interface import (
StorageProvider,
StorageFactory,
UploadResult,
DownloadResult,
)
from ...config_utils import config
class COSStorageProvider(StorageProvider):
"""
腾讯云COS存储提供者实现
实现了StorageProvider接口的所有方法提供完整的COS存储功能。
采用懒加载模式初始化COS客户端提高性能。
"""
def __init__(self, config: Dict[str, Any]):
"""
初始化COS存储提供者
Args:
config: COS配置字典必须包含secret_id、secret_key和region
"""
super().__init__(config)
self.bucket_name = config.get("bucket_name", "bwkj-cos-1324682537")
self.region = config["region"]
self.secret_id = config["secret_id"]
self.secret_key = config["secret_key"]
self._client = None
def _validate_config(self) -> None:
"""
验证COS配置的完整性
Raises:
ValueError: 配置信息缺失时抛出异常
"""
required_keys = ["secret_id", "secret_key", "region"]
missing_keys = [key for key in required_keys if not self.config.get(key)]
if missing_keys:
raise ValueError(
f"COS配置缺失必要参数: {missing_keys}. " f"请检查配置文件或环境变量"
)
@property
def client(self):
"""
获取COS客户端实例懒加载模式
Returns:
CosS3Client: COS客户端实例
"""
if self._client is None:
try:
cos_config = CosConfig(
Region=self.region,
SecretId=self.secret_id,
SecretKey=self.secret_key,
)
self._client = CosS3Client(cos_config)
loguru.logger.info(f"COS客户端初始化成功区域: {self.region}")
except Exception as e:
loguru.logger.error(f"COS客户端初始化失败: {e}")
raise
return self._client
def upload_file(
self,
local_path: str,
remote_key: str,
content_type: Optional[str] = None,
**kwargs,
) -> UploadResult:
"""
上传本地文件到COS
Args:
local_path: 本地文件路径
remote_key: COS中的键名
content_type: 文件内容类型
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
if not os.path.exists(local_path):
error_msg = f"本地文件不存在: {local_path}"
loguru.logger.error(error_msg)
return UploadResult(
success=False,
key=remote_key,
message=error_msg,
error=FileNotFoundError(error_msg),
)
file_size = os.path.getsize(local_path)
loguru.logger.info(
f"开始上传文件到COS: {local_path} -> cos://{self.bucket_name}/{remote_key} "
f"({file_size} bytes)"
)
# 执行上传操作,包含重试机制
for attempt in range(3): # 最多重试3次
try:
response = self.client.upload_file(
Bucket=self.bucket_name,
Key=remote_key,
LocalFilePath=local_path,
**kwargs,
)
break
except (CosClientError, CosServiceError) as e:
if attempt == 2: # 最后一次尝试失败
raise e
loguru.logger.warning(
f"COS上传尝试 {attempt + 1} 失败,重试中: {e}"
)
success_msg = f"文件上传成功: cos://{self.bucket_name}/{remote_key}"
loguru.logger.info(success_msg)
return UploadResult(
success=True, key=remote_key, size=file_size, message=success_msg
)
except Exception as e:
error_msg = f"COS文件上传失败: {str(e)}"
loguru.logger.error(f"{error_msg} (文件: {local_path}, 键: {remote_key})")
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def upload_bytes(
self, data: bytes, remote_key: str, content_type: Optional[str] = None, **kwargs
) -> UploadResult:
"""
上传字节数据到COS
Args:
data: 字节数据
remote_key: COS中的键名
content_type: 文件内容类型
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
loguru.logger.info(
f"开始上传字节数据到COS: cos://{self.bucket_name}/{remote_key} "
f"({len(data)} bytes)"
)
# 执行上传操作,包含重试机制
for attempt in range(3):
try:
response = self.client.put_object(
Bucket=self.bucket_name,
Key=remote_key,
Body=data,
ContentType=content_type,
**kwargs,
)
break
except (CosClientError, CosServiceError) as e:
if attempt == 2:
raise e
loguru.logger.warning(
f"COS字节数据上传尝试 {attempt + 1} 失败,重试中: {e}"
)
success_msg = f"字节数据上传成功: cos://{self.bucket_name}/{remote_key}"
loguru.logger.info(success_msg)
return UploadResult(
success=True, key=remote_key, size=len(data), message=success_msg
)
except Exception as e:
error_msg = f"COS字节数据上传失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 大小: {len(data)} bytes)"
)
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def upload_tensor(
self, tensor: torch.Tensor, remote_key: str, format: str = "PNG", **kwargs
) -> UploadResult:
"""
上传PyTorch张量作为图像到COS
Args:
tensor: PyTorch张量
remote_key: COS中的键名
format: 图像格式PNG, JPEG等
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
from ...image_utils import tensor_to_tempfile
loguru.logger.info(
f"开始上传张量到COS: cos://{self.bucket_name}/{remote_key} "
f"(形状: {tensor.shape}, 格式: {format})"
)
# 将张量转换为临时文件
temp_file = tensor_to_tempfile(tensor, format=format)
temp_path = temp_file.name
try:
# 设置内容类型
content_type = f"image/{format.lower()}"
if format.upper() == "JPEG":
content_type = "image/jpeg"
# 上传临时文件
result = self.upload_file(
temp_path, remote_key, content_type=content_type, **kwargs
)
if result.success:
loguru.logger.info(
f"张量上传成功: cos://{self.bucket_name}/{remote_key}"
)
return result
finally:
# 清理临时文件
if os.path.exists(temp_path):
os.unlink(temp_path)
loguru.logger.debug(f"临时文件已清理: {temp_path}")
except Exception as e:
error_msg = f"COS张量上传失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 张量形状: {tensor.shape})"
)
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def download_file(
self, remote_key: str, local_path: str, **kwargs
) -> DownloadResult:
"""
从COS下载文件到本地
Args:
remote_key: COS中的键名
local_path: 本地保存路径
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果
"""
try:
loguru.logger.info(
f"开始从COS下载文件: cos://{self.bucket_name}/{remote_key} -> {local_path}"
)
# 确保本地目录存在
os.makedirs(os.path.dirname(local_path), exist_ok=True)
# 检查文件是否存在
if not self.file_exists(remote_key):
error_msg = f"COS文件不存在: cos://{self.bucket_name}/{remote_key}"
loguru.logger.error(error_msg)
return DownloadResult(
success=False, message=error_msg, error=FileNotFoundError(error_msg)
)
# 执行下载操作,包含重试机制
for attempt in range(3):
try:
response = self.client.download_file(
Bucket=self.bucket_name,
Key=remote_key,
DestFilePath=local_path,
**kwargs,
)
break
except (CosClientError, CosServiceError) as e:
if attempt == 2:
raise e
loguru.logger.warning(
f"COS下载尝试 {attempt + 1} 失败,重试中: {e}"
)
success_msg = f"文件下载成功: {local_path}"
loguru.logger.info(success_msg)
return DownloadResult(
success=True, local_path=local_path, message=success_msg
)
except Exception as e:
error_msg = f"COS文件下载失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 本地路径: {local_path})"
)
return DownloadResult(success=False, message=error_msg, error=e)
def download_bytes(self, remote_key: str, **kwargs) -> DownloadResult:
"""
从COS下载文件为字节数据
Args:
remote_key: COS中的键名
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果数据包含在data字段中
"""
try:
loguru.logger.info(
f"开始从COS下载字节数据: cos://{self.bucket_name}/{remote_key}"
)
# 检查文件是否存在
if not self.file_exists(remote_key):
error_msg = f"COS文件不存在: cos://{self.bucket_name}/{remote_key}"
loguru.logger.error(error_msg)
return DownloadResult(
success=False, message=error_msg, error=FileNotFoundError(error_msg)
)
# 执行下载操作,包含重试机制
for attempt in range(3):
try:
response = self.client.get_object(
Bucket=self.bucket_name, Key=remote_key, **kwargs
)
data = response["Body"].read()
break
except (CosClientError, CosServiceError) as e:
if attempt == 2:
raise e
loguru.logger.warning(
f"COS字节数据下载尝试 {attempt + 1} 失败,重试中: {e}"
)
success_msg = f"字节数据下载成功: {len(data)} bytes"
loguru.logger.info(success_msg)
return DownloadResult(success=True, data=data, message=success_msg)
except Exception as e:
error_msg = f"COS字节数据下载失败: {str(e)}"
loguru.logger.error(f"{error_msg} (键: {remote_key})")
return DownloadResult(success=False, message=error_msg, error=e)
def delete_file(self, remote_key: str, **kwargs) -> bool:
"""
删除COS中的文件
Args:
remote_key: COS中的键名
**kwargs: 额外的删除参数
Returns:
bool: 删除是否成功
"""
try:
loguru.logger.info(f"删除COS文件: cos://{self.bucket_name}/{remote_key}")
# 执行删除操作,包含重试机制
for attempt in range(3):
try:
self.client.delete_object(
Bucket=self.bucket_name, Key=remote_key, **kwargs
)
break
except (CosClientError, CosServiceError) as e:
if attempt == 2:
raise e
loguru.logger.warning(
f"COS删除尝试 {attempt + 1} 失败,重试中: {e}"
)
loguru.logger.info(f"文件删除成功: cos://{self.bucket_name}/{remote_key}")
return True
except Exception as e:
loguru.logger.error(f"COS文件删除失败: {e} (键: {remote_key})")
return False
def file_exists(self, remote_key: str, **kwargs) -> bool:
"""
检查COS中文件是否存在
Args:
remote_key: COS中的键名
**kwargs: 额外的检查参数
Returns:
bool: 文件是否存在
"""
try:
self.client.head_object(Bucket=self.bucket_name, Key=remote_key, **kwargs)
return True
except Exception:
return False
def list_files(
self, prefix: str = "", max_keys: int = 1000, **kwargs
) -> List[Dict[str, Any]]:
"""
列出COS中的文件
Args:
prefix: 文件前缀过滤
max_keys: 最大返回数量
**kwargs: 额外的列表参数
Returns:
List[Dict[str, Any]]: 文件信息列表
"""
try:
loguru.logger.info(
f"列出COS文件: cos://{self.bucket_name}/{prefix} (最大: {max_keys})"
)
response = self.client.list_objects(
Bucket=self.bucket_name, Prefix=prefix, MaxKeys=max_keys, **kwargs
)
files = []
if "Contents" in response:
for obj in response["Contents"]:
files.append(
{
"key": obj["Key"],
"size": obj["Size"],
"last_modified": obj["LastModified"],
"etag": obj.get("ETag", "").strip('"'),
"storage_class": obj.get("StorageClass", "STANDARD"),
}
)
loguru.logger.info(f"找到 {len(files)} 个文件")
return files
except Exception as e:
loguru.logger.error(f"COS文件列表获取失败: {e} (前缀: {prefix})")
return []
def get_file_url(self, remote_key: str, expires_in: int = 3600, **kwargs) -> str:
"""
获取COS文件的访问URL
Args:
remote_key: COS中的键名
expires_in: URL过期时间
**kwargs: 额外的URL生成参数
Returns:
str: 文件访问URL
"""
try:
# 生成预签名URL
url = self.client.get_presigned_url(
Method="GET",
Bucket=self.bucket_name,
Key=remote_key,
Expired=expires_in,
**kwargs,
)
return url
except Exception as e:
loguru.logger.error(f"COS URL生成失败: {e} (键: {remote_key})")
raise
class COSStorageFactory(StorageFactory):
"""
COS存储工厂实现
负责创建COS存储提供者实例处理配置验证和初始化。
"""
def create_provider(self, config_dict: Dict[str, Any]) -> StorageProvider:
"""
创建COS存储提供者实例
Args:
config_dict: COS配置字典
Returns:
StorageProvider: COS存储提供者实例
"""
# 如果没有提供配置,尝试从全局配置获取
if not config_dict or not config_dict.get("secret_id"):
if config.has_cos_config():
cos_config = config.get_cos_config()
config_dict = {
"secret_id": cos_config["secret_id"],
"secret_key": cos_config["secret_key"],
"region": cos_config["region"],
"bucket_name": cos_config.get("bucket_name"),
**config_dict,
}
else:
raise ValueError("未提供有效的COS配置且全局配置中也没有COS配置")
return COSStorageProvider(config_dict)
def get_supported_types(self) -> List[str]:
"""
获取支持的存储类型列表
Returns:
List[str]: 支持的存储类型
"""
return ["cos", "qcloud", "tencent"]

View File

@@ -0,0 +1,502 @@
"""
AWS S3存储提供者实现
本模块实现了AWS S3的具体存储操作继承自抽象存储接口。
提供完整的S3文件上传、下载、删除等功能。
特性:
- 支持文件、字节数据、PyTorch张量的上传
- 支持预签名URL生成
- 完整的错误处理和重试机制
- 统一的日志记录
"""
import os
import boto3
from typing import Optional, Dict, Any, List
import torch
import loguru
from ..storage_interface import (
StorageProvider,
StorageFactory,
UploadResult,
DownloadResult,
)
from ...config_utils import config
class S3StorageProvider(StorageProvider):
"""
AWS S3存储提供者实现
实现了StorageProvider接口的所有方法提供完整的S3存储功能。
采用懒加载模式初始化S3客户端提高性能。
"""
def __init__(self, config: Dict[str, Any]):
"""
初始化S3存储提供者
Args:
config: S3配置字典必须包含access_key_id和secret_access_key
"""
super().__init__(config)
self.bucket_name = config.get("bucket_name", "modal-media-cache")
self.region = config.get("region", "us-east-1")
self.cdn_base_url = config.get("cdn_base_url", "https://cdn.roasmax.cn")
self._client = None
def _validate_config(self) -> None:
"""
验证S3配置的完整性
Raises:
ValueError: 配置信息缺失时抛出异常
"""
required_keys = ["access_key_id", "secret_access_key"]
missing_keys = [key for key in required_keys if not self.config.get(key)]
if missing_keys:
raise ValueError(
f"S3配置缺失必要参数: {missing_keys}. " f"请检查配置文件或环境变量"
)
@property
def client(self):
"""
获取S3客户端实例懒加载模式
Returns:
boto3.client: S3客户端实例
"""
if self._client is None:
try:
self._client = boto3.client(
"s3",
aws_access_key_id=self.config["access_key_id"],
aws_secret_access_key=self.config["secret_access_key"],
region_name=self.region,
)
loguru.logger.info(f"S3客户端初始化成功区域: {self.region}")
except Exception as e:
loguru.logger.error(f"S3客户端初始化失败: {e}")
raise
return self._client
def upload_file(
self,
local_path: str,
remote_key: str,
content_type: Optional[str] = None,
**kwargs,
) -> UploadResult:
"""
上传本地文件到S3
Args:
local_path: 本地文件路径
remote_key: S3中的键名
content_type: 文件内容类型
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
if not os.path.exists(local_path):
error_msg = f"本地文件不存在: {local_path}"
loguru.logger.error(error_msg)
return UploadResult(
success=False,
key=remote_key,
message=error_msg,
error=FileNotFoundError(error_msg),
)
file_size = os.path.getsize(local_path)
loguru.logger.info(
f"开始上传文件到S3: {local_path} -> s3://{self.bucket_name}/{remote_key} "
f"({file_size} bytes)"
)
extra_args = kwargs.copy()
if content_type:
extra_args["ContentType"] = content_type
self.client.upload_file(
local_path,
self.bucket_name,
remote_key,
ExtraArgs=extra_args if extra_args else None,
)
url = f"{self.cdn_base_url}/{remote_key}"
success_msg = f"文件上传成功: s3://{self.bucket_name}/{remote_key}"
loguru.logger.info(success_msg)
return UploadResult(
success=True,
key=remote_key,
url=url,
size=file_size,
message=success_msg,
)
except Exception as e:
error_msg = f"S3文件上传失败: {str(e)}"
loguru.logger.error(f"{error_msg} (文件: {local_path}, 键: {remote_key})")
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def upload_bytes(
self, data: bytes, remote_key: str, content_type: Optional[str] = None, **kwargs
) -> UploadResult:
"""
上传字节数据到S3
Args:
data: 字节数据
remote_key: S3中的键名
content_type: 文件内容类型
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
loguru.logger.info(
f"开始上传字节数据到S3: s3://{self.bucket_name}/{remote_key} "
f"({len(data)} bytes)"
)
extra_args = kwargs.copy()
if content_type:
extra_args["ContentType"] = content_type
self.client.put_object(
Bucket=self.bucket_name, Key=remote_key, Body=data, **extra_args
)
url = f"{self.cdn_base_url}/{remote_key}"
success_msg = f"字节数据上传成功: s3://{self.bucket_name}/{remote_key}"
loguru.logger.info(success_msg)
return UploadResult(
success=True,
key=remote_key,
url=url,
size=len(data),
message=success_msg,
)
except Exception as e:
error_msg = f"S3字节数据上传失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 大小: {len(data)} bytes)"
)
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def upload_tensor(
self, tensor: torch.Tensor, remote_key: str, format: str = "PNG", **kwargs
) -> UploadResult:
"""
上传PyTorch张量作为图像到S3
Args:
tensor: PyTorch张量
remote_key: S3中的键名
format: 图像格式PNG, JPEG等
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
try:
from ...image_utils import tensor_to_tempfile
loguru.logger.info(
f"开始上传张量到S3: s3://{self.bucket_name}/{remote_key} "
f"(形状: {tensor.shape}, 格式: {format})"
)
# 将张量转换为临时文件
temp_file = tensor_to_tempfile(tensor, format=format)
temp_path = temp_file.name
try:
# 设置内容类型
content_type = f"image/{format.lower()}"
if format.upper() == "JPEG":
content_type = "image/jpeg"
# 上传临时文件
result = self.upload_file(
temp_path, remote_key, content_type=content_type, **kwargs
)
if result.success:
loguru.logger.info(
f"张量上传成功: s3://{self.bucket_name}/{remote_key}"
)
return result
finally:
# 清理临时文件
if os.path.exists(temp_path):
os.unlink(temp_path)
loguru.logger.debug(f"临时文件已清理: {temp_path}")
except Exception as e:
error_msg = f"S3张量上传失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 张量形状: {tensor.shape})"
)
return UploadResult(
success=False, key=remote_key, message=error_msg, error=e
)
def download_file(
self, remote_key: str, local_path: str, **kwargs
) -> DownloadResult:
"""
从S3下载文件到本地
Args:
remote_key: S3中的键名
local_path: 本地保存路径
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果
"""
try:
loguru.logger.info(
f"开始从S3下载文件: s3://{self.bucket_name}/{remote_key} -> {local_path}"
)
# 确保本地目录存在
os.makedirs(os.path.dirname(local_path), exist_ok=True)
# 检查文件是否存在
if not self.file_exists(remote_key):
error_msg = f"S3文件不存在: s3://{self.bucket_name}/{remote_key}"
loguru.logger.error(error_msg)
return DownloadResult(
success=False, message=error_msg, error=FileNotFoundError(error_msg)
)
self.client.download_file(self.bucket_name, remote_key, local_path)
success_msg = f"文件下载成功: {local_path}"
loguru.logger.info(success_msg)
return DownloadResult(
success=True, local_path=local_path, message=success_msg
)
except Exception as e:
error_msg = f"S3文件下载失败: {str(e)}"
loguru.logger.error(
f"{error_msg} (键: {remote_key}, 本地路径: {local_path})"
)
return DownloadResult(success=False, message=error_msg, error=e)
def download_bytes(self, remote_key: str, **kwargs) -> DownloadResult:
"""
从S3下载文件为字节数据
Args:
remote_key: S3中的键名
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果数据包含在data字段中
"""
try:
loguru.logger.info(
f"开始从S3下载字节数据: s3://{self.bucket_name}/{remote_key}"
)
# 检查文件是否存在
if not self.file_exists(remote_key):
error_msg = f"S3文件不存在: s3://{self.bucket_name}/{remote_key}"
loguru.logger.error(error_msg)
return DownloadResult(
success=False, message=error_msg, error=FileNotFoundError(error_msg)
)
response = self.client.get_object(Bucket=self.bucket_name, Key=remote_key)
data = response["Body"].read()
success_msg = f"字节数据下载成功: {len(data)} bytes"
loguru.logger.info(success_msg)
return DownloadResult(success=True, data=data, message=success_msg)
except Exception as e:
error_msg = f"S3字节数据下载失败: {str(e)}"
loguru.logger.error(f"{error_msg} (键: {remote_key})")
return DownloadResult(success=False, message=error_msg, error=e)
def delete_file(self, remote_key: str, **kwargs) -> bool:
"""
删除S3中的文件
Args:
remote_key: S3中的键名
**kwargs: 额外的删除参数
Returns:
bool: 删除是否成功
"""
try:
loguru.logger.info(f"删除S3文件: s3://{self.bucket_name}/{remote_key}")
self.client.delete_object(Bucket=self.bucket_name, Key=remote_key)
loguru.logger.info(f"文件删除成功: s3://{self.bucket_name}/{remote_key}")
return True
except Exception as e:
loguru.logger.error(f"S3文件删除失败: {e} (键: {remote_key})")
return False
def file_exists(self, remote_key: str, **kwargs) -> bool:
"""
检查S3中文件是否存在
Args:
remote_key: S3中的键名
**kwargs: 额外的检查参数
Returns:
bool: 文件是否存在
"""
try:
self.client.head_object(Bucket=self.bucket_name, Key=remote_key)
return True
except Exception:
return False
def list_files(
self, prefix: str = "", max_keys: int = 1000, **kwargs
) -> List[Dict[str, Any]]:
"""
列出S3中的文件
Args:
prefix: 文件前缀过滤
max_keys: 最大返回数量
**kwargs: 额外的列表参数
Returns:
List[Dict[str, Any]]: 文件信息列表
"""
try:
loguru.logger.info(
f"列出S3文件: s3://{self.bucket_name}/{prefix} (最大: {max_keys})"
)
response = self.client.list_objects_v2(
Bucket=self.bucket_name, Prefix=prefix, MaxKeys=max_keys, **kwargs
)
files = []
if "Contents" in response:
for obj in response["Contents"]:
files.append(
{
"key": obj["Key"],
"size": obj["Size"],
"last_modified": obj["LastModified"],
"etag": obj.get("ETag", "").strip('"'),
"storage_class": obj.get("StorageClass", "STANDARD"),
}
)
loguru.logger.info(f"找到 {len(files)} 个文件")
return files
except Exception as e:
loguru.logger.error(f"S3文件列表获取失败: {e} (前缀: {prefix})")
return []
def get_file_url(self, remote_key: str, expires_in: int = 3600, **kwargs) -> str:
"""
获取S3文件的访问URL
Args:
remote_key: S3中的键名
expires_in: URL过期时间
**kwargs: 额外的URL生成参数
Returns:
str: 文件访问URL
"""
try:
# 如果配置了CDN优先返回CDN URL
if self.cdn_base_url and expires_in > 3600: # 长期URL使用CDN
return f"{self.cdn_base_url}/{remote_key}"
# 生成预签名URL
url = self.client.generate_presigned_url(
"get_object",
Params={"Bucket": self.bucket_name, "Key": remote_key},
ExpiresIn=expires_in,
**kwargs,
)
return url
except Exception as e:
loguru.logger.error(f"S3 URL生成失败: {e} (键: {remote_key})")
# 如果预签名URL生成失败返回CDN URL作为备选
if self.cdn_base_url:
return f"{self.cdn_base_url}/{remote_key}"
raise
class S3StorageFactory(StorageFactory):
"""
S3存储工厂实现
负责创建S3存储提供者实例处理配置验证和初始化。
"""
def create_provider(self, config_dict: Dict[str, Any]) -> StorageProvider:
"""
创建S3存储提供者实例
Args:
config_dict: S3配置字典
Returns:
StorageProvider: S3存储提供者实例
"""
# 如果没有提供配置,尝试从全局配置获取
if not config_dict or not config_dict.get("access_key_id"):
if config.has_aws_config():
aws_config = config.get_aws_config()
config_dict = {
"access_key_id": aws_config["access_key_id"],
"secret_access_key": aws_config["secret_access_key"],
**config_dict,
}
else:
raise ValueError("未提供有效的S3配置且全局配置中也没有AWS配置")
return S3StorageProvider(config_dict)
def get_supported_types(self) -> List[str]:
"""
获取支持的存储类型列表
Returns:
List[str]: 支持的存储类型
"""
return ["s3", "aws", "amazon"]

View File

@@ -0,0 +1,304 @@
"""
腾讯云VOD存储提供者
提供腾讯云视频点播(VOD)服务的统一接口实现。
支持媒体文件的查询和下载功能。
"""
import os
from typing import Dict, Any, Optional
import requests
import loguru
from tencentcloud.common import credential
from tencentcloud.common.profile.client_profile import ClientProfile
from tencentcloud.common.profile.http_profile import HttpProfile
from tencentcloud.vod.v20180717 import vod_client, models
from ..storage_interface import StorageProvider, StorageFactory, DownloadResult
class VODProvider(StorageProvider):
"""
腾讯云VOD存储提供者
提供腾讯云视频点播服务的统一接口,包括:
- 媒体文件信息查询
- 媒体文件下载
- 媒体文件管理
"""
def __init__(self, config: Dict[str, Any]):
"""
初始化VOD提供者
Args:
config: VOD配置字典包含
- secret_id: 腾讯云密钥ID
- secret_key: 腾讯云密钥Key
- region: 区域默认ap-beijing
- sub_app_id: 子应用ID可选
"""
super().__init__(config)
# 验证必需的配置项
required_keys = ["secret_id", "secret_key"]
for key in required_keys:
if not config.get(key):
raise ValueError(f"VOD配置缺少必需项: {key}")
self.secret_id = config["secret_id"]
self.secret_key = config["secret_key"]
self.region = config.get("region", "ap-beijing")
self.sub_app_id = config.get("sub_app_id")
# 初始化VOD客户端
self.vod_client = self._init_vod_client()
loguru.logger.info(f"VOD提供者初始化成功区域: {self.region}")
def _init_vod_client(self):
"""初始化VOD客户端"""
try:
http_profile = HttpProfile(endpoint="vod.tencentcloudapi.com")
client_profile = ClientProfile(httpProfile=http_profile)
cred = credential.Credential(self.secret_id, self.secret_key)
return vod_client.VodClient(cred, self.region, client_profile)
except Exception as e:
raise RuntimeError(f"VOD客户端初始化失败: {e}")
def get_media_info(
self, file_id: str, sub_app_id: Optional[str] = None
) -> Dict[str, Any]:
"""
获取媒体文件信息
Args:
file_id: 文件ID
sub_app_id: 子应用ID为None时使用默认配置
Returns:
Dict: 媒体文件信息
Raises:
ValueError: 文件不存在或参数错误
RuntimeError: API调用失败
"""
try:
if not file_id or not file_id.strip():
raise ValueError("文件ID不能为空")
req = models.DescribeMediaInfosRequest()
req.FileIds = [file_id.strip()]
# 使用传入的sub_app_id或默认配置
app_id = sub_app_id or self.sub_app_id
if app_id:
req.SubAppId = int(app_id)
resp = self.vod_client.DescribeMediaInfos(req)
if not resp.MediaInfoSet:
raise ValueError(f"文件不存在: {file_id}")
media_info = resp.MediaInfoSet[0]
# 构造返回信息
result = {
"file_id": file_id,
"name": media_info.BasicInfo.Name or "",
"size": media_info.BasicInfo.Size or 0,
"duration": media_info.BasicInfo.Duration or 0,
"media_url": media_info.BasicInfo.MediaUrl or "",
"cover_url": media_info.BasicInfo.CoverUrl or "",
"create_time": media_info.BasicInfo.CreateTime or "",
"update_time": media_info.BasicInfo.UpdateTime or "",
}
loguru.logger.info(f"获取媒体信息成功: {file_id}")
return result
except Exception as e:
if isinstance(e, ValueError):
raise
loguru.logger.error(f"获取媒体信息失败: {e}")
raise RuntimeError(f"腾讯云VOD API错误: {e}")
def get_download_url(self, file_id: str, sub_app_id: Optional[str] = None) -> str:
"""
获取媒体文件下载URL
Args:
file_id: 文件ID
sub_app_id: 子应用ID
Returns:
str: 下载URL
Raises:
ValueError: 文件不存在或没有下载URL
"""
media_info = self.get_media_info(file_id, sub_app_id)
if not media_info["media_url"]:
raise ValueError(f"文件 {file_id} 没有可用的下载URL")
return media_info["media_url"]
def download_file(
self,
file_id: str,
local_path: str,
sub_app_id: Optional[str] = None,
timeout: int = 300,
) -> DownloadResult:
"""
下载VOD媒体文件到本地
Args:
file_id: 文件ID这里用作远程键名
local_path: 本地保存路径
sub_app_id: 子应用ID
timeout: 下载超时时间(秒)
Returns:
DownloadResult: 下载结果
"""
try:
# 获取下载URL
download_url = self.get_download_url(file_id, sub_app_id)
loguru.logger.info(f"开始下载VOD文件: {file_id} -> {local_path}")
# 确保本地目录存在
os.makedirs(os.path.dirname(local_path), exist_ok=True)
# 下载文件
with requests.get(download_url, stream=True, timeout=timeout) as response:
response.raise_for_status()
with open(local_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
if chunk:
f.write(chunk)
# 验证文件是否下载成功
if not os.path.exists(local_path) or os.path.getsize(local_path) == 0:
raise RuntimeError("下载的文件为空或不存在")
loguru.logger.info(f"VOD文件下载成功: {local_path}")
return DownloadResult(
success=True, local_path=local_path, message="下载成功"
)
except Exception as e:
error_msg = f"VOD文件下载失败: {str(e)}"
loguru.logger.error(error_msg)
return DownloadResult(success=False, message=error_msg)
def upload_file(self, local_path: str, remote_key: str, **kwargs) -> Any:
"""
VOD不支持直接文件上传请使用腾讯云VOD控制台或专用上传接口
Raises:
NotImplementedError: VOD不支持此操作
"""
raise NotImplementedError("VOD提供者不支持文件上传请使用腾讯云VOD控制台")
def upload_bytes(self, file_content: bytes, remote_key: str, **kwargs) -> Any:
"""
VOD不支持直接字节上传
Raises:
NotImplementedError: VOD不支持此操作
"""
raise NotImplementedError("VOD提供者不支持字节上传")
def upload_tensor(self, tensor: Any, remote_key: str, **kwargs) -> Any:
"""
VOD不支持张量上传
Raises:
NotImplementedError: VOD不支持此操作
"""
raise NotImplementedError("VOD提供者不支持张量上传")
def delete_file(self, remote_key: str, **kwargs) -> bool:
"""
删除VOD媒体文件
Args:
remote_key: 文件ID
Returns:
bool: 是否删除成功
Note:
此功能需要额外的权限配置
"""
# 这里可以实现删除逻辑,但需要谨慎使用
loguru.logger.warning("VOD文件删除功能未实现请使用控制台操作")
return False
def list_files(self, prefix: str = "", **kwargs) -> list:
"""
列出VOD媒体文件
Args:
prefix: 过滤前缀在VOD中可能是分类或标签
Returns:
list: 文件列表
Note:
此功能需要调用搜索接口实现
"""
loguru.logger.warning("VOD文件列表功能未实现")
return []
class VODStorageFactory(StorageFactory):
"""
腾讯云VOD存储工厂
负责创建和管理VOD存储提供者实例
"""
def get_supported_types(self) -> list:
"""获取支持的存储类型"""
return ["vod", "tencent_vod", "qcloud_vod"]
def create_provider(self, storage_type: str, config: Dict[str, Any]) -> VODProvider:
"""
创建VOD存储提供者
Args:
storage_type: 存储类型
config: 配置信息
Returns:
VODProvider: VOD存储提供者实例
"""
if storage_type not in self.get_supported_types():
raise ValueError(f"不支持的存储类型: {storage_type}")
return VODProvider(config)
def validate_config(self, config: Dict[str, Any]) -> bool:
"""
验证VOD配置
Args:
config: 配置字典
Returns:
bool: 配置是否有效
"""
required_keys = ["secret_id", "secret_key"]
for key in required_keys:
if not config.get(key):
return False
return True

View File

@@ -0,0 +1,387 @@
"""
对象存储接口抽象层
本模块定义了对象存储服务的抽象接口,支持多种云存储服务的统一操作。
采用抽象工厂模式和策略模式,确保系统的可扩展性和可维护性。
支持的存储类型:
- AWS S3
- 腾讯云COS
- 其他云存储服务(可扩展)
设计原则:
- 开闭原则:对扩展开放,对修改关闭
- 依赖倒置原则:依赖抽象而非具体实现
- 单一职责原则:每个类只负责一个职责
"""
from abc import ABC, abstractmethod
from typing import Optional, Dict, Any, List
from dataclasses import dataclass
import torch
@dataclass
class UploadResult:
"""上传结果封装类"""
success: bool
key: str
url: Optional[str] = None
size: Optional[int] = None
message: Optional[str] = None
error: Optional[Exception] = None
@dataclass
class DownloadResult:
"""下载结果封装类"""
success: bool
local_path: Optional[str] = None
data: Optional[bytes] = None
message: Optional[str] = None
error: Optional[Exception] = None
class StorageProvider(ABC):
"""
对象存储提供者抽象基类
定义了所有对象存储服务必须实现的基本操作接口。
遵循里氏替换原则,所有子类都可以替换基类使用。
"""
def __init__(self, config: Dict[str, Any]):
"""
初始化存储提供者
Args:
config: 存储配置字典,包含认证信息和其他设置
"""
self.config = config
self._validate_config()
@abstractmethod
def _validate_config(self) -> None:
"""
验证配置信息的完整性和有效性
Raises:
ValueError: 配置信息缺失或无效时抛出异常
"""
pass
@abstractmethod
def upload_file(self, local_path: str, remote_key: str, **kwargs) -> UploadResult:
"""
上传本地文件到远程存储
Args:
local_path: 本地文件路径
remote_key: 远程存储中的键名
**kwargs: 额外的上传参数(如内容类型、元数据等)
Returns:
UploadResult: 上传操作结果
"""
pass
@abstractmethod
def upload_bytes(self, data: bytes, remote_key: str, **kwargs) -> UploadResult:
"""
上传字节数据到远程存储
Args:
data: 字节数据
remote_key: 远程存储中的键名
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
pass
@abstractmethod
def upload_tensor(
self, tensor: torch.Tensor, remote_key: str, format: str = "PNG", **kwargs
) -> UploadResult:
"""
上传PyTorch张量作为图像到远程存储
Args:
tensor: PyTorch张量
remote_key: 远程存储中的键名
format: 图像格式PNG, JPEG等
**kwargs: 额外的上传参数
Returns:
UploadResult: 上传操作结果
"""
pass
@abstractmethod
def download_file(
self, remote_key: str, local_path: str, **kwargs
) -> DownloadResult:
"""
从远程存储下载文件到本地
Args:
remote_key: 远程存储中的键名
local_path: 本地保存路径
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果
"""
pass
@abstractmethod
def download_bytes(self, remote_key: str, **kwargs) -> DownloadResult:
"""
从远程存储下载文件为字节数据
Args:
remote_key: 远程存储中的键名
**kwargs: 额外的下载参数
Returns:
DownloadResult: 下载操作结果数据包含在data字段中
"""
pass
@abstractmethod
def delete_file(self, remote_key: str, **kwargs) -> bool:
"""
删除远程存储中的文件
Args:
remote_key: 远程存储中的键名
**kwargs: 额外的删除参数
Returns:
bool: 删除是否成功
"""
pass
@abstractmethod
def file_exists(self, remote_key: str, **kwargs) -> bool:
"""
检查远程存储中文件是否存在
Args:
remote_key: 远程存储中的键名
**kwargs: 额外的检查参数
Returns:
bool: 文件是否存在
"""
pass
@abstractmethod
def list_files(
self, prefix: str = "", max_keys: int = 1000, **kwargs
) -> List[Dict[str, Any]]:
"""
列出远程存储中的文件
Args:
prefix: 文件前缀过滤
max_keys: 最大返回数量
**kwargs: 额外的列表参数
Returns:
List[Dict[str, Any]]: 文件信息列表
"""
pass
@abstractmethod
def get_file_url(self, remote_key: str, expires_in: int = 3600, **kwargs) -> str:
"""
获取文件的访问URL
Args:
remote_key: 远程存储中的键名
expires_in: URL过期时间
**kwargs: 额外的URL生成参数
Returns:
str: 文件访问URL
"""
pass
def get_provider_name(self) -> str:
"""
获取存储提供者名称
Returns:
str: 提供者名称
"""
return self.__class__.__name__
class StorageFactory(ABC):
"""
存储提供者工厂抽象基类
采用抽象工厂模式,负责创建具体的存储提供者实例。
支持不同类型的存储服务创建。
"""
@abstractmethod
def create_provider(self, config: Dict[str, Any]) -> StorageProvider:
"""
创建存储提供者实例
Args:
config: 存储配置字典
Returns:
StorageProvider: 存储提供者实例
Raises:
ValueError: 配置无效时抛出异常
NotImplementedError: 不支持的存储类型时抛出异常
"""
pass
@abstractmethod
def get_supported_types(self) -> List[str]:
"""
获取支持的存储类型列表
Returns:
List[str]: 支持的存储类型
"""
pass
class StorageManager:
"""
存储管理器
采用策略模式,统一管理不同的存储提供者。
提供统一的存储操作接口,支持动态切换存储提供者。
"""
def __init__(self):
"""初始化存储管理器"""
self._factories: Dict[str, StorageFactory] = {}
self._providers: Dict[str, StorageProvider] = {}
self._default_provider: Optional[str] = None
def register_factory(self, storage_type: str, factory: StorageFactory) -> None:
"""
注册存储工厂
Args:
storage_type: 存储类型标识符(如:'s3', 'cos'
factory: 存储工厂实例
"""
self._factories[storage_type] = factory
def create_provider(
self,
storage_type: str,
config: Dict[str, Any],
provider_id: Optional[str] = None,
) -> StorageProvider:
"""
创建存储提供者
Args:
storage_type: 存储类型
config: 存储配置
provider_id: 提供者唯一标识符默认使用storage_type
Returns:
StorageProvider: 存储提供者实例
Raises:
ValueError: 不支持的存储类型时抛出异常
"""
if storage_type not in self._factories:
available_types = list(self._factories.keys())
raise ValueError(
f"不支持的存储类型: {storage_type}. " f"可用类型: {available_types}"
)
factory = self._factories[storage_type]
provider = factory.create_provider(config)
# 缓存提供者实例
provider_id = provider_id or storage_type
self._providers[provider_id] = provider
# 设置默认提供者
if self._default_provider is None:
self._default_provider = provider_id
return provider
def get_provider(self, provider_id: Optional[str] = None) -> StorageProvider:
"""
获取存储提供者
Args:
provider_id: 提供者标识符,默认返回默认提供者
Returns:
StorageProvider: 存储提供者实例
Raises:
ValueError: 提供者不存在时抛出异常
"""
if provider_id is None:
provider_id = self._default_provider
if provider_id is None or provider_id not in self._providers:
available_providers = list(self._providers.keys())
raise ValueError(
f"存储提供者不存在: {provider_id}. "
f"可用提供者: {available_providers}"
)
return self._providers[provider_id]
def set_default_provider(self, provider_id: str) -> None:
"""
设置默认存储提供者
Args:
provider_id: 提供者标识符
Raises:
ValueError: 提供者不存在时抛出异常
"""
if provider_id not in self._providers:
available_providers = list(self._providers.keys())
raise ValueError(
f"存储提供者不存在: {provider_id}. "
f"可用提供者: {available_providers}"
)
self._default_provider = provider_id
def get_available_providers(self) -> List[str]:
"""
获取所有可用的提供者列表
Returns:
List[str]: 提供者标识符列表
"""
return list(self._providers.keys())
def get_supported_types(self) -> List[str]:
"""
获取所有支持的存储类型
Returns:
List[str]: 存储类型列表
"""
return list(self._factories.keys())
# 全局存储管理器实例
storage_manager = StorageManager()