feat: 添加便捷小工具页面 - AI检索图片/数据清洗功能
- 新增便捷工具页面 (/tools),提供AI检索图片/数据清洗功能 - 支持JSONL格式数据的URI匹配去重处理 - 实现实时进度显示和批量数据处理 - 添加完整的错误处理和用户反馈机制 - 遵循Tauri开发规范和UI/UX设计标准 - 包含测试数据和功能文档 技术实现: - 后端: Rust异步处理,流式文件读取,进度事件发送 - 前端: React + TypeScript,文件选择对话框,进度条显示 - 导航: 新增便捷工具菜单项,集成到主导航栏
This commit is contained in:
100
apps/desktop/TOOLS_FEATURE.md
Normal file
100
apps/desktop/TOOLS_FEATURE.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
# 便捷小工具功能 - v0.1.33
|
||||||
|
|
||||||
|
## 功能概述
|
||||||
|
|
||||||
|
新增便捷小工具页面,提供AI检索图片/数据清洗功能,支持JSONL格式数据的去重处理。
|
||||||
|
|
||||||
|
## 功能特性
|
||||||
|
|
||||||
|
### AI检索图片/数据清洗工具
|
||||||
|
|
||||||
|
- **文件格式支持**: JSONL (JSON Lines) 格式
|
||||||
|
- **数据去重**: 基于URI字段进行精确匹配去重
|
||||||
|
- **进度显示**: 实时显示处理进度和状态
|
||||||
|
- **批量处理**: 支持大数据量文件处理
|
||||||
|
- **结果统计**: 显示原始数据量、去除数量、最终结果数量
|
||||||
|
|
||||||
|
## 使用方法
|
||||||
|
|
||||||
|
1. **选择全部数据文件**: 包含所有数据的JSONL文件
|
||||||
|
2. **选择要去除的数据文件**: 包含需要去除数据的JSONL文件
|
||||||
|
3. **选择输出文件**: 指定处理结果的保存位置
|
||||||
|
4. **开始处理**: 点击"开始处理"按钮执行数据清洗
|
||||||
|
|
||||||
|
## 数据格式说明
|
||||||
|
|
||||||
|
### 全部数据文件格式
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"kind": "storage#object",
|
||||||
|
"id": "...",
|
||||||
|
"uri": "gs://bucket/path/to/file.jpg",
|
||||||
|
// 其他字段...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 要去除的数据文件格式
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "uuid",
|
||||||
|
"schema_id": "default_schema",
|
||||||
|
"json_data": "{\"uri\":\"gs://bucket/path/to/file.jpg\", ...}"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 处理逻辑
|
||||||
|
|
||||||
|
1. **解析要去除的数据**: 从`json_data`字段中提取`uri`值
|
||||||
|
2. **构建URI集合**: 将所有要去除的URI存储在HashSet中
|
||||||
|
3. **过滤全部数据**: 遍历全部数据文件,过滤掉URI匹配的记录
|
||||||
|
4. **保存结果**: 将过滤后的数据保存到输出文件
|
||||||
|
|
||||||
|
## 技术实现
|
||||||
|
|
||||||
|
### 后端 (Rust)
|
||||||
|
- **命令**: `clean_jsonl_data`
|
||||||
|
- **文件**: `src-tauri/src/presentation/commands/tools_commands.rs`
|
||||||
|
- **特性**:
|
||||||
|
- 异步处理
|
||||||
|
- 进度事件发送
|
||||||
|
- 内存优化的流式处理
|
||||||
|
- 错误处理和恢复
|
||||||
|
|
||||||
|
### 前端 (React + TypeScript)
|
||||||
|
- **页面**: `src/pages/Tools.tsx`
|
||||||
|
- **路由**: `/tools`
|
||||||
|
- **特性**:
|
||||||
|
- 文件选择对话框
|
||||||
|
- 实时进度显示
|
||||||
|
- 结果统计展示
|
||||||
|
- 错误处理和用户反馈
|
||||||
|
|
||||||
|
## 导航集成
|
||||||
|
|
||||||
|
在主导航栏中新增"便捷工具"菜单项,使用扳手图标,提供快速访问入口。
|
||||||
|
|
||||||
|
## 测试数据
|
||||||
|
|
||||||
|
项目包含测试数据文件:
|
||||||
|
- `test_data/sample_all_data.jsonl`: 示例全部数据文件
|
||||||
|
- `test_data/sample_remove_data.jsonl`: 示例要去除的数据文件
|
||||||
|
|
||||||
|
## 性能优化
|
||||||
|
|
||||||
|
- **流式处理**: 逐行读取文件,避免内存溢出
|
||||||
|
- **进度更新**: 每处理100行更新一次进度,平衡性能和用户体验
|
||||||
|
- **错误容错**: 解析失败的行会被跳过并记录警告
|
||||||
|
|
||||||
|
## 开发规范遵循
|
||||||
|
|
||||||
|
- 遵循Tauri开发规范的四层架构设计
|
||||||
|
- 使用TypeScript确保类型安全
|
||||||
|
- 遵循UI/UX设计标准,提供优雅的用户界面
|
||||||
|
- 实现完整的错误处理和用户反馈机制
|
||||||
|
|
||||||
|
## 未来扩展
|
||||||
|
|
||||||
|
- 支持更多数据格式 (CSV, JSON等)
|
||||||
|
- 添加更多数据清洗规则
|
||||||
|
- 支持自定义匹配字段
|
||||||
|
- 添加数据预览功能
|
||||||
@@ -253,7 +253,9 @@ pub fn run() {
|
|||||||
commands::test_commands::test_logging,
|
commands::test_commands::test_logging,
|
||||||
// 调试命令
|
// 调试命令
|
||||||
commands::debug_commands::test_parse_draft_file,
|
commands::debug_commands::test_parse_draft_file,
|
||||||
commands::debug_commands::validate_template_structure
|
commands::debug_commands::validate_template_structure,
|
||||||
|
// 便捷工具命令
|
||||||
|
commands::tools_commands::clean_jsonl_data
|
||||||
])
|
])
|
||||||
.setup(|app| {
|
.setup(|app| {
|
||||||
// 初始化日志系统
|
// 初始化日志系统
|
||||||
|
|||||||
@@ -16,3 +16,4 @@ pub mod material_matching_commands;
|
|||||||
pub mod template_matching_result_commands;
|
pub mod template_matching_result_commands;
|
||||||
pub mod export_record_commands;
|
pub mod export_record_commands;
|
||||||
pub mod video_generation_commands;
|
pub mod video_generation_commands;
|
||||||
|
pub mod tools_commands;
|
||||||
|
|||||||
@@ -0,0 +1,222 @@
|
|||||||
|
use tauri::{command, Emitter};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io::{BufRead, BufReader, Write};
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct DataCleaningProgress {
|
||||||
|
pub current: usize,
|
||||||
|
pub total: usize,
|
||||||
|
pub percentage: f64,
|
||||||
|
pub status: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct DataCleaningResult {
|
||||||
|
pub success: bool,
|
||||||
|
pub message: String,
|
||||||
|
pub original_count: usize,
|
||||||
|
pub removed_count: usize,
|
||||||
|
pub final_count: usize,
|
||||||
|
pub output_file: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct StorageObject {
|
||||||
|
pub uri: String,
|
||||||
|
// 其他字段可以忽略,我们只关心 uri
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct ProcessedData {
|
||||||
|
pub json_data: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// AI检索图片/数据清洗 - JSONL格式处理
|
||||||
|
#[command]
|
||||||
|
pub async fn clean_jsonl_data(
|
||||||
|
all_data_file: String,
|
||||||
|
remove_data_file: String,
|
||||||
|
output_file: String,
|
||||||
|
window: tauri::Window,
|
||||||
|
) -> Result<DataCleaningResult, String> {
|
||||||
|
// 验证文件存在
|
||||||
|
if !Path::new(&all_data_file).exists() {
|
||||||
|
return Err(format!("全部数据文件不存在: {}", all_data_file));
|
||||||
|
}
|
||||||
|
|
||||||
|
if !Path::new(&remove_data_file).exists() {
|
||||||
|
return Err(format!("要去除的数据文件不存在: {}", remove_data_file));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 发送开始处理的进度
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: 0,
|
||||||
|
total: 0,
|
||||||
|
percentage: 0.0,
|
||||||
|
status: "开始处理...".to_string(),
|
||||||
|
});
|
||||||
|
|
||||||
|
// 第一步:读取要去除的数据文件,提取所有URI
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: 0,
|
||||||
|
total: 0,
|
||||||
|
percentage: 10.0,
|
||||||
|
status: "读取要去除的数据文件...".to_string(),
|
||||||
|
});
|
||||||
|
|
||||||
|
let remove_uris = match extract_remove_uris(&remove_data_file).await {
|
||||||
|
Ok(uris) => uris,
|
||||||
|
Err(e) => return Err(format!("读取要去除的数据文件失败: {}", e)),
|
||||||
|
};
|
||||||
|
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: 0,
|
||||||
|
total: 0,
|
||||||
|
percentage: 30.0,
|
||||||
|
status: format!("找到 {} 个要去除的URI", remove_uris.len()),
|
||||||
|
});
|
||||||
|
|
||||||
|
// 第二步:处理全部数据文件
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: 0,
|
||||||
|
total: 0,
|
||||||
|
percentage: 40.0,
|
||||||
|
status: "开始处理全部数据文件...".to_string(),
|
||||||
|
});
|
||||||
|
|
||||||
|
let result = match process_all_data_file(&all_data_file, &remove_uris, &output_file, &window).await {
|
||||||
|
Ok(result) => result,
|
||||||
|
Err(e) => return Err(format!("处理全部数据文件失败: {}", e)),
|
||||||
|
};
|
||||||
|
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: result.final_count,
|
||||||
|
total: result.original_count,
|
||||||
|
percentage: 100.0,
|
||||||
|
status: "处理完成".to_string(),
|
||||||
|
});
|
||||||
|
|
||||||
|
Ok(result)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 从要去除的数据文件中提取所有URI
|
||||||
|
async fn extract_remove_uris(file_path: &str) -> Result<HashSet<String>, String> {
|
||||||
|
let file = File::open(file_path)
|
||||||
|
.map_err(|e| format!("无法打开文件: {}", e))?;
|
||||||
|
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut uris = HashSet::new();
|
||||||
|
|
||||||
|
for line in reader.lines() {
|
||||||
|
let line = line.map_err(|e| format!("读取行失败: {}", e))?;
|
||||||
|
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 解析JSON
|
||||||
|
let processed_data: ProcessedData = serde_json::from_str(&line)
|
||||||
|
.map_err(|e| format!("解析JSON失败: {} - 行内容: {}", e, line))?;
|
||||||
|
|
||||||
|
// 从json_data字段中解析内部JSON
|
||||||
|
let inner_json: serde_json::Value = serde_json::from_str(&processed_data.json_data)
|
||||||
|
.map_err(|e| format!("解析内部JSON失败: {}", e))?;
|
||||||
|
|
||||||
|
// 提取uri字段
|
||||||
|
if let Some(uri) = inner_json.get("uri").and_then(|v| v.as_str()) {
|
||||||
|
uris.insert(uri.to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(uris)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 处理全部数据文件,过滤掉要去除的URI
|
||||||
|
async fn process_all_data_file(
|
||||||
|
input_file: &str,
|
||||||
|
remove_uris: &HashSet<String>,
|
||||||
|
output_file: &str,
|
||||||
|
window: &tauri::Window,
|
||||||
|
) -> Result<DataCleaningResult, String> {
|
||||||
|
let input_file_handle = File::open(input_file)
|
||||||
|
.map_err(|e| format!("无法打开输入文件: {}", e))?;
|
||||||
|
|
||||||
|
let reader = BufReader::new(input_file_handle);
|
||||||
|
|
||||||
|
let mut output_file_handle = File::create(output_file)
|
||||||
|
.map_err(|e| format!("无法创建输出文件: {}", e))?;
|
||||||
|
|
||||||
|
let mut original_count = 0;
|
||||||
|
let mut removed_count = 0;
|
||||||
|
let mut final_count = 0;
|
||||||
|
|
||||||
|
// 首先计算总行数用于进度显示
|
||||||
|
let total_lines = count_lines(input_file)?;
|
||||||
|
|
||||||
|
for (line_number, line) in reader.lines().enumerate() {
|
||||||
|
let line = line.map_err(|e| format!("读取行失败: {}", e))?;
|
||||||
|
|
||||||
|
if line.trim().is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
original_count += 1;
|
||||||
|
|
||||||
|
// 解析JSON
|
||||||
|
let storage_object: StorageObject = match serde_json::from_str(&line) {
|
||||||
|
Ok(obj) => obj,
|
||||||
|
Err(e) => {
|
||||||
|
// 如果解析失败,跳过这一行但记录警告
|
||||||
|
eprintln!("警告: 解析JSON失败,跳过行 {}: {} - 内容: {}", line_number + 1, e, line);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// 检查URI是否在要去除的列表中
|
||||||
|
if remove_uris.contains(&storage_object.uri) {
|
||||||
|
removed_count += 1;
|
||||||
|
} else {
|
||||||
|
// 写入输出文件
|
||||||
|
writeln!(output_file_handle, "{}", line)
|
||||||
|
.map_err(|e| format!("写入输出文件失败: {}", e))?;
|
||||||
|
final_count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 每处理100行发送一次进度更新
|
||||||
|
if line_number % 100 == 0 {
|
||||||
|
let percentage = 40.0 + (line_number as f64 / total_lines as f64) * 50.0;
|
||||||
|
let _ = window.emit("data-cleaning-progress", DataCleaningProgress {
|
||||||
|
current: line_number,
|
||||||
|
total: total_lines,
|
||||||
|
percentage,
|
||||||
|
status: format!("已处理 {} 行,已去除 {} 个重复项", line_number, removed_count),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 确保文件被正确写入
|
||||||
|
output_file_handle.flush()
|
||||||
|
.map_err(|e| format!("刷新输出文件失败: {}", e))?;
|
||||||
|
|
||||||
|
Ok(DataCleaningResult {
|
||||||
|
success: true,
|
||||||
|
message: "数据清洗完成".to_string(),
|
||||||
|
original_count,
|
||||||
|
removed_count,
|
||||||
|
final_count,
|
||||||
|
output_file: output_file.to_string(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 计算文件行数
|
||||||
|
fn count_lines(file_path: &str) -> Result<usize, String> {
|
||||||
|
let file = File::open(file_path)
|
||||||
|
.map_err(|e| format!("无法打开文件计算行数: {}", e))?;
|
||||||
|
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let count = reader.lines().count();
|
||||||
|
Ok(count)
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@ import ModelDetail from './pages/ModelDetail';
|
|||||||
import AiClassificationSettings from './pages/AiClassificationSettings';
|
import AiClassificationSettings from './pages/AiClassificationSettings';
|
||||||
import TemplateManagement from './pages/TemplateManagement';
|
import TemplateManagement from './pages/TemplateManagement';
|
||||||
import { MaterialModelBinding } from './pages/MaterialModelBinding';
|
import { MaterialModelBinding } from './pages/MaterialModelBinding';
|
||||||
|
import Tools from './pages/Tools';
|
||||||
import Navigation from './components/Navigation';
|
import Navigation from './components/Navigation';
|
||||||
import { NotificationSystem, useNotifications } from './components/NotificationSystem';
|
import { NotificationSystem, useNotifications } from './components/NotificationSystem';
|
||||||
import { useProjectStore } from './store/projectStore';
|
import { useProjectStore } from './store/projectStore';
|
||||||
@@ -78,6 +79,7 @@ function App() {
|
|||||||
<Route path="/ai-classification-settings" element={<AiClassificationSettings />} />
|
<Route path="/ai-classification-settings" element={<AiClassificationSettings />} />
|
||||||
<Route path="/templates" element={<TemplateManagement />} />
|
<Route path="/templates" element={<TemplateManagement />} />
|
||||||
<Route path="/material-model-binding" element={<MaterialModelBinding />} />
|
<Route path="/material-model-binding" element={<MaterialModelBinding />} />
|
||||||
|
<Route path="/tools" element={<Tools />} />
|
||||||
</Routes>
|
</Routes>
|
||||||
</div>
|
</div>
|
||||||
</main>
|
</main>
|
||||||
|
|||||||
@@ -5,7 +5,8 @@ import {
|
|||||||
UserGroupIcon,
|
UserGroupIcon,
|
||||||
CpuChipIcon,
|
CpuChipIcon,
|
||||||
DocumentDuplicateIcon,
|
DocumentDuplicateIcon,
|
||||||
LinkIcon
|
LinkIcon,
|
||||||
|
WrenchScrewdriverIcon
|
||||||
} from '@heroicons/react/24/outline';
|
} from '@heroicons/react/24/outline';
|
||||||
|
|
||||||
const Navigation: React.FC = () => {
|
const Navigation: React.FC = () => {
|
||||||
@@ -41,6 +42,12 @@ const Navigation: React.FC = () => {
|
|||||||
href: '/ai-classification-settings',
|
href: '/ai-classification-settings',
|
||||||
icon: CpuChipIcon,
|
icon: CpuChipIcon,
|
||||||
description: '管理AI视频分类规则'
|
description: '管理AI视频分类规则'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: '便捷工具',
|
||||||
|
href: '/tools',
|
||||||
|
icon: WrenchScrewdriverIcon,
|
||||||
|
description: 'AI检索图片/数据清洗工具'
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|||||||
354
apps/desktop/src/pages/Tools.tsx
Normal file
354
apps/desktop/src/pages/Tools.tsx
Normal file
@@ -0,0 +1,354 @@
|
|||||||
|
import React, { useState } from 'react';
|
||||||
|
import {
|
||||||
|
Wrench,
|
||||||
|
FileText,
|
||||||
|
Upload,
|
||||||
|
Download,
|
||||||
|
Loader2,
|
||||||
|
CheckCircle,
|
||||||
|
AlertCircle,
|
||||||
|
Info,
|
||||||
|
Trash2
|
||||||
|
} from 'lucide-react';
|
||||||
|
import { invoke } from '@tauri-apps/api/core';
|
||||||
|
import { open, save } from '@tauri-apps/plugin-dialog';
|
||||||
|
import { listen } from '@tauri-apps/api/event';
|
||||||
|
import { useNotifications } from '../components/NotificationSystem';
|
||||||
|
|
||||||
|
interface DataCleaningProgress {
|
||||||
|
current: number;
|
||||||
|
total: number;
|
||||||
|
percentage: number;
|
||||||
|
status: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface DataCleaningResult {
|
||||||
|
success: boolean;
|
||||||
|
message: string;
|
||||||
|
original_count: number;
|
||||||
|
removed_count: number;
|
||||||
|
final_count: number;
|
||||||
|
output_file: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 便捷小工具页面
|
||||||
|
* 遵循 Tauri 开发规范和 UI/UX 设计标准
|
||||||
|
*/
|
||||||
|
const Tools: React.FC = () => {
|
||||||
|
const [allDataFile, setAllDataFile] = useState<string>('');
|
||||||
|
const [removeDataFile, setRemoveDataFile] = useState<string>('');
|
||||||
|
const [outputFile, setOutputFile] = useState<string>('');
|
||||||
|
const [isProcessing, setIsProcessing] = useState(false);
|
||||||
|
const [progress, setProgress] = useState<DataCleaningProgress | null>(null);
|
||||||
|
const [result, setResult] = useState<DataCleaningResult | null>(null);
|
||||||
|
|
||||||
|
const { success, error } = useNotifications();
|
||||||
|
|
||||||
|
// 选择全部数据文件
|
||||||
|
const selectAllDataFile = async () => {
|
||||||
|
try {
|
||||||
|
const selected = await open({
|
||||||
|
multiple: false,
|
||||||
|
filters: [{
|
||||||
|
name: 'JSONL Files',
|
||||||
|
extensions: ['jsonl']
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
|
||||||
|
if (selected && typeof selected === 'string') {
|
||||||
|
setAllDataFile(selected);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
error('文件选择失败', '无法选择全部数据文件');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// 选择要去除的数据文件
|
||||||
|
const selectRemoveDataFile = async () => {
|
||||||
|
try {
|
||||||
|
const selected = await open({
|
||||||
|
multiple: false,
|
||||||
|
filters: [{
|
||||||
|
name: 'JSONL Files',
|
||||||
|
extensions: ['jsonl']
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
|
||||||
|
if (selected && typeof selected === 'string') {
|
||||||
|
setRemoveDataFile(selected);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
error('文件选择失败', '无法选择要去除的数据文件');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// 选择输出文件
|
||||||
|
const selectOutputFile = async () => {
|
||||||
|
try {
|
||||||
|
const selected = await save({
|
||||||
|
filters: [{
|
||||||
|
name: 'JSONL Files',
|
||||||
|
extensions: ['jsonl']
|
||||||
|
}],
|
||||||
|
defaultPath: 'cleaned_data.jsonl'
|
||||||
|
});
|
||||||
|
|
||||||
|
if (selected) {
|
||||||
|
setOutputFile(selected);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
error('文件选择失败', '无法选择输出文件');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// 开始数据清洗
|
||||||
|
const startDataCleaning = async () => {
|
||||||
|
if (!allDataFile || !removeDataFile || !outputFile) {
|
||||||
|
error('参数错误', '请选择所有必需的文件');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
setIsProcessing(true);
|
||||||
|
setProgress(null);
|
||||||
|
setResult(null);
|
||||||
|
|
||||||
|
try {
|
||||||
|
// 监听进度事件
|
||||||
|
const unlisten = await listen<DataCleaningProgress>('data-cleaning-progress', (event) => {
|
||||||
|
setProgress(event.payload);
|
||||||
|
});
|
||||||
|
|
||||||
|
// 调用后端命令
|
||||||
|
const cleaningResult = await invoke<DataCleaningResult>('clean_jsonl_data', {
|
||||||
|
allDataFile,
|
||||||
|
removeDataFile,
|
||||||
|
outputFile
|
||||||
|
});
|
||||||
|
|
||||||
|
setResult(cleaningResult);
|
||||||
|
|
||||||
|
if (cleaningResult.success) {
|
||||||
|
success('数据清洗完成',
|
||||||
|
`原始数据: ${cleaningResult.original_count} 条,` +
|
||||||
|
`去除重复: ${cleaningResult.removed_count} 条,` +
|
||||||
|
`最终结果: ${cleaningResult.final_count} 条`
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
error('数据清洗失败', cleaningResult.message);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 清理事件监听器
|
||||||
|
unlisten();
|
||||||
|
} catch (err) {
|
||||||
|
const errorMessage = err instanceof Error ? err.message : '未知错误';
|
||||||
|
error('数据清洗失败', errorMessage);
|
||||||
|
} finally {
|
||||||
|
setIsProcessing(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// 重置表单
|
||||||
|
const resetForm = () => {
|
||||||
|
setAllDataFile('');
|
||||||
|
setRemoveDataFile('');
|
||||||
|
setOutputFile('');
|
||||||
|
setProgress(null);
|
||||||
|
setResult(null);
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="space-y-6">
|
||||||
|
{/* 页面标题 */}
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
<div className="w-10 h-10 bg-gradient-to-br from-purple-500 to-purple-600 rounded-lg flex items-center justify-center shadow-sm">
|
||||||
|
<Wrench className="w-5 h-5 text-white" />
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<h1 className="text-2xl font-bold text-gray-900">便捷小工具</h1>
|
||||||
|
<p className="text-gray-600">AI检索图片/数据清洗工具集</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* AI检索图片/数据清洗工具 */}
|
||||||
|
<div className="bg-white rounded-xl shadow-sm border border-gray-200 overflow-hidden">
|
||||||
|
<div className="p-6 border-b border-gray-200">
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
<FileText className="w-6 h-6 text-purple-600" />
|
||||||
|
<div>
|
||||||
|
<h2 className="text-lg font-semibold text-gray-900">AI检索图片/数据清洗</h2>
|
||||||
|
<p className="text-sm text-gray-600">JSONL格式数据去重处理</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="p-6 space-y-6">
|
||||||
|
{/* 使用说明 */}
|
||||||
|
<div className="bg-blue-50 border border-blue-200 rounded-lg p-4">
|
||||||
|
<div className="flex items-start gap-3">
|
||||||
|
<Info className="w-5 h-5 text-blue-600 mt-0.5 flex-shrink-0" />
|
||||||
|
<div className="text-sm text-blue-800">
|
||||||
|
<p className="font-medium mb-2">使用说明:</p>
|
||||||
|
<ul className="space-y-1 list-disc list-inside">
|
||||||
|
<li>选择包含所有数据的JSONL文件(全部数据)</li>
|
||||||
|
<li>选择包含要去除数据的JSONL文件(要去除的数据)</li>
|
||||||
|
<li>系统将根据URI字段进行匹配,去除重复项</li>
|
||||||
|
<li>处理结果将保存到指定的输出文件</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* 文件选择区域 */}
|
||||||
|
<div className="grid grid-cols-1 md:grid-cols-2 gap-6">
|
||||||
|
{/* 全部数据文件 */}
|
||||||
|
<div className="space-y-2">
|
||||||
|
<label className="block text-sm font-medium text-gray-700">
|
||||||
|
全部数据文件 <span className="text-red-500">*</span>
|
||||||
|
</label>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<button
|
||||||
|
onClick={selectAllDataFile}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="flex items-center gap-2 px-4 py-2 bg-gray-100 hover:bg-gray-200 disabled:bg-gray-50 disabled:text-gray-400 text-gray-700 rounded-lg border border-gray-300 transition-colors"
|
||||||
|
>
|
||||||
|
<Upload className="w-4 h-4" />
|
||||||
|
选择文件
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{allDataFile && (
|
||||||
|
<p className="text-xs text-gray-600 break-all bg-gray-50 p-2 rounded">
|
||||||
|
{allDataFile}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* 要去除的数据文件 */}
|
||||||
|
<div className="space-y-2">
|
||||||
|
<label className="block text-sm font-medium text-gray-700">
|
||||||
|
要去除的数据文件 <span className="text-red-500">*</span>
|
||||||
|
</label>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<button
|
||||||
|
onClick={selectRemoveDataFile}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="flex items-center gap-2 px-4 py-2 bg-gray-100 hover:bg-gray-200 disabled:bg-gray-50 disabled:text-gray-400 text-gray-700 rounded-lg border border-gray-300 transition-colors"
|
||||||
|
>
|
||||||
|
<Upload className="w-4 h-4" />
|
||||||
|
选择文件
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{removeDataFile && (
|
||||||
|
<p className="text-xs text-gray-600 break-all bg-gray-50 p-2 rounded">
|
||||||
|
{removeDataFile}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* 输出文件选择 */}
|
||||||
|
<div className="space-y-2">
|
||||||
|
<label className="block text-sm font-medium text-gray-700">
|
||||||
|
输出文件 <span className="text-red-500">*</span>
|
||||||
|
</label>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<button
|
||||||
|
onClick={selectOutputFile}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="flex items-center gap-2 px-4 py-2 bg-gray-100 hover:bg-gray-200 disabled:bg-gray-50 disabled:text-gray-400 text-gray-700 rounded-lg border border-gray-300 transition-colors"
|
||||||
|
>
|
||||||
|
<Download className="w-4 h-4" />
|
||||||
|
选择保存位置
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{outputFile && (
|
||||||
|
<p className="text-xs text-gray-600 break-all bg-gray-50 p-2 rounded">
|
||||||
|
{outputFile}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* 进度显示 */}
|
||||||
|
{progress && (
|
||||||
|
<div className="bg-gray-50 rounded-lg p-4">
|
||||||
|
<div className="flex items-center gap-3 mb-3">
|
||||||
|
<Loader2 className="w-5 h-5 text-purple-600 animate-spin" />
|
||||||
|
<span className="text-sm font-medium text-gray-900">处理中...</span>
|
||||||
|
</div>
|
||||||
|
<div className="space-y-2">
|
||||||
|
<div className="flex justify-between text-sm text-gray-600">
|
||||||
|
<span>{progress.status}</span>
|
||||||
|
<span>{progress.percentage.toFixed(1)}%</span>
|
||||||
|
</div>
|
||||||
|
<div className="w-full bg-gray-200 rounded-full h-2">
|
||||||
|
<div
|
||||||
|
className="bg-purple-600 h-2 rounded-full transition-all duration-300"
|
||||||
|
style={{ width: `${progress.percentage}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
{progress.total > 0 && (
|
||||||
|
<div className="text-xs text-gray-500">
|
||||||
|
{progress.current.toLocaleString()} / {progress.total.toLocaleString()}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* 结果显示 */}
|
||||||
|
{result && (
|
||||||
|
<div className={`rounded-lg p-4 ${result.success ? 'bg-green-50 border border-green-200' : 'bg-red-50 border border-red-200'}`}>
|
||||||
|
<div className="flex items-start gap-3">
|
||||||
|
{result.success ? (
|
||||||
|
<CheckCircle className="w-5 h-5 text-green-600 mt-0.5 flex-shrink-0" />
|
||||||
|
) : (
|
||||||
|
<AlertCircle className="w-5 h-5 text-red-600 mt-0.5 flex-shrink-0" />
|
||||||
|
)}
|
||||||
|
<div className="flex-1">
|
||||||
|
<p className={`font-medium ${result.success ? 'text-green-800' : 'text-red-800'}`}>
|
||||||
|
{result.message}
|
||||||
|
</p>
|
||||||
|
{result.success && (
|
||||||
|
<div className="mt-2 text-sm text-green-700 space-y-1">
|
||||||
|
<p>原始数据条数: {result.original_count.toLocaleString()}</p>
|
||||||
|
<p>去除重复条数: {result.removed_count.toLocaleString()}</p>
|
||||||
|
<p>最终结果条数: {result.final_count.toLocaleString()}</p>
|
||||||
|
<p className="break-all">输出文件: {result.output_file}</p>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* 操作按钮 */}
|
||||||
|
<div className="flex gap-3 pt-4 border-t border-gray-200">
|
||||||
|
<button
|
||||||
|
onClick={startDataCleaning}
|
||||||
|
disabled={!allDataFile || !removeDataFile || !outputFile || isProcessing}
|
||||||
|
className="flex items-center gap-2 px-6 py-2 bg-purple-600 hover:bg-purple-700 disabled:bg-gray-300 disabled:cursor-not-allowed text-white rounded-lg font-medium transition-colors"
|
||||||
|
>
|
||||||
|
{isProcessing ? (
|
||||||
|
<Loader2 className="w-4 h-4 animate-spin" />
|
||||||
|
) : (
|
||||||
|
<FileText className="w-4 h-4" />
|
||||||
|
)}
|
||||||
|
{isProcessing ? '处理中...' : '开始处理'}
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<button
|
||||||
|
onClick={resetForm}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="flex items-center gap-2 px-6 py-2 bg-gray-100 hover:bg-gray-200 disabled:bg-gray-50 disabled:text-gray-400 text-gray-700 rounded-lg font-medium transition-colors"
|
||||||
|
>
|
||||||
|
<Trash2 className="w-4 h-4" />
|
||||||
|
重置
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
export default Tools;
|
||||||
5
apps/desktop/test_data/sample_all_data.jsonl
Normal file
5
apps/desktop/test_data/sample_all_data.jsonl
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
{"kind":"storage#object","id":"fashion_image_block/gallery_v2/models/test_image_1.jpg/1752665521922040","selfLink":"https://www.googleapis.com/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_1.jpg","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_1.jpg?generation=1752665521922040&alt=media","name":"gallery_v2/models/test_image_1.jpg","bucket":"fashion_image_block","generation":"1752665521922040","metageneration":"1","contentType":"image/jpeg","storageClass":"STANDARD","size":310505,"md5Hash":"WmzNIJHZxALLVSfn0esl5Q==","crc32c":"6skf3A==","etag":"CPi/4O6jwY4DEAE=","timeCreated":"2025-07-16T11:32:01.981000Z","updated":"2025-07-16T11:32:01.981000Z","uri":"gs://fashion_image_block/gallery_v2/models/test_image_1.jpg"}
|
||||||
|
{"kind":"storage#object","id":"fashion_image_block/gallery_v2/models/test_image_2.jpg/1752665524250767","selfLink":"https://www.googleapis.com/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_2.jpg","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_2.jpg?generation=1752665524250767&alt=media","name":"gallery_v2/models/test_image_2.jpg","bucket":"fashion_image_block","generation":"1752665524250767","metageneration":"1","contentType":"image/jpeg","storageClass":"STANDARD","size":488328,"md5Hash":"ksnQY8Wv6g5Q+LkmaxBV+A==","crc32c":"u4zm9A==","etag":"CI/R7u+jwY4DEAE=","timeCreated":"2025-07-16T11:32:04.317000Z","updated":"2025-07-16T11:32:04.317000Z","uri":"gs://fashion_image_block/gallery_v2/models/test_image_2.jpg"}
|
||||||
|
{"kind":"storage#object","id":"fashion_image_block/gallery_v2/models/test_image_3.jpg/1752665522362206","selfLink":"https://www.googleapis.com/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_3.jpg","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_3.jpg?generation=1752665522362206&alt=media","name":"gallery_v2/models/test_image_3.jpg","bucket":"fashion_image_block","generation":"1752665522362206","metageneration":"1","contentType":"image/jpeg","storageClass":"STANDARD","size":719213,"md5Hash":"fPmRuc/nFZnN6g5u9pJ99g==","crc32c":"r4eLmg==","etag":"CN6u++6jwY4DEAE=","timeCreated":"2025-07-16T11:32:02.420000Z","updated":"2025-07-16T11:32:02.420000Z","uri":"gs://fashion_image_block/gallery_v2/models/test_image_3.jpg"}
|
||||||
|
{"kind":"storage#object","id":"fashion_image_block/gallery_v2/models/test_image_4.jpg/1752665523165046","selfLink":"https://www.googleapis.com/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_4.jpg","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_4.jpg?generation=1752665523165046&alt=media","name":"gallery_v2/models/test_image_4.jpg","bucket":"fashion_image_block","generation":"1752665523165046","metageneration":"1","contentType":"image/jpeg","storageClass":"STANDARD","size":463800,"md5Hash":"G0EjX9ZdkMEXoDR3h+zNJQ==","crc32c":"+wH9Bg==","etag":"CPaurO+jwY4DEAE=","timeCreated":"2025-07-16T11:32:03.230000Z","updated":"2025-07-16T11:32:03.230000Z","uri":"gs://fashion_image_block/gallery_v2/models/test_image_4.jpg"}
|
||||||
|
{"kind":"storage#object","id":"fashion_image_block/gallery_v2/models/test_image_5.jpg/1752665521595009","selfLink":"https://www.googleapis.com/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_5.jpg","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/fashion_image_block/o/gallery_v2%2Fmodels%2Ftest_image_5.jpg?generation=1752665521595009&alt=media","name":"gallery_v2/models/test_image_5.jpg","bucket":"fashion_image_block","generation":"1752665521595009","metageneration":"1","contentType":"image/jpeg","storageClass":"STANDARD","size":435049,"md5Hash":"vw3s4DXgN06HPdSnizhPrQ==","crc32c":"kHGLOA==","etag":"CIHFzO6jwY4DEAE=","timeCreated":"2025-07-16T11:32:01.663000Z","updated":"2025-07-16T11:32:01.663000Z","uri":"gs://fashion_image_block/gallery_v2/models/test_image_5.jpg"}
|
||||||
2
apps/desktop/test_data/sample_remove_data.jsonl
Normal file
2
apps/desktop/test_data/sample_remove_data.jsonl
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
{"id":"0fc38c97-e8a7-4fc2-8228-7598d1b993e9","schema_id":"default_schema","json_data":"{\"dress_color_pattern\":{\"Hue\":0.08,\"Saturation\":0.05,\"Value\":0.95,\"rgb_hex\":\"#f2ebe6\"},\"style_description\":\"测试样例1\",\"environment_tags\":[\"Indoor\",\"Test\"],\"products\":[{\"category\":\"测试\",\"color_pattern\":{\"Hue\":0.08,\"Saturation\":0.05,\"Value\":0.95,\"rgb_hex\":\"#f2ebe6\"},\"design_styles\":[\"Test\"],\"design_styles_embedding\":[]}],\"uri\":\"gs://fashion_image_block/gallery_v2/models/test_image_2.jpg\",\"description\":\"\",\"runtime\":\"\",\"releaseDate\":\"2025-07-16T11:31:48.092583Z\",\"title\":\"model\",\"categories\":[\"测试\"]}"}
|
||||||
|
{"id":"e32fcc8d-166c-4da5-991d-422f7c5a47e7","schema_id":"default_schema","json_data":"{\"dress_color_pattern\":{\"Hue\":0.08,\"Saturation\":0.05,\"Value\":0.95,\"rgb_hex\":\"#f2ebe6\"},\"style_description\":\"测试样例2\",\"environment_tags\":[\"Indoor\",\"Test\"],\"products\":[{\"category\":\"测试\",\"color_pattern\":{\"Hue\":0.08,\"Saturation\":0.05,\"Value\":0.95,\"rgb_hex\":\"#f2ebe6\"},\"design_styles\":[\"Test\"],\"design_styles_embedding\":[]}],\"uri\":\"gs://fashion_image_block/gallery_v2/models/test_image_4.jpg\",\"description\":\"\",\"runtime\":\"\",\"releaseDate\":\"2025-07-16T11:31:48.092583Z\",\"title\":\"model\",\"categories\":[\"测试\"]}"}
|
||||||
Reference in New Issue
Block a user