Home Esplora Aiuto
Registrati Accedi
wxcz_admin
/
project0802shenji
1
0
Forka 0
File Problemi 0 Pull Requests 0 Wiki
Ramo (Branch): master
Rami (Branch) Tag
project0802shen...
/
logs
/
training_20250718_000815.log

training_20250718_000815.log 6.6 KB
Permalink Cronologia Originale

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162 
  1. 2025-07-18 00:08:15 - root - INFO - 日志系统已初始化，日志文件: /qwen/logs/training_20250718_000815.log
    2. 

  2. 2025-07-18 00:08:15 - TrainingMonitor - INFO - 训练监控器已初始化，监控间隔: 30秒
    3. 

  3. 2025-07-18 00:08:15 - SecurityModelApp - INFO - 网络安全模型应用程序已初始化
    4. 

  4. 2025-07-18 00:08:15 - SecurityModelApp - INFO - 开始新的模型训练...
    5. 

  5. 2025-07-18 00:08:15 - TrainingMonitor - INFO - 开始训练监控
    6. 

  6. 2025-07-18 00:08:15 - SecurityModelApp - INFO - 设置模型...
    7. 

  7. 2025-07-18 00:08:15 - ModelDownloader - INFO - 模型已存在: /qwen/models/Qwen_Qwen2.5-1.5B-Instruct
    8. 

  8. 2025-07-18 00:08:15 - SecurityModelTrainer - INFO - 加载模型: /qwen/models/Qwen_Qwen2.5-1.5B-Instruct
    9. 

  9. 2025-07-18 00:08:17 - accelerate.utils.modeling - INFO - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
    10. 

  10. 2025-07-18 00:08:21 - SecurityModelTrainer - INFO - 模型已加载并配置8位量化
    11. 

  11. 2025-07-18 00:08:22 - SecurityModelTrainer - INFO - LoRA配置已应用
    12. 

  12. 2025-07-18 00:08:22 - SecurityModelTrainer - INFO - 模型设置完成
    13. 

  13. 2025-07-18 00:08:22 - SecurityModelApp - INFO - 准备训练数据...
    14. 

  14. 2025-07-18 00:08:22 - DataLoader - INFO - 未指定数据文件，将自动加载目录下所有JSON文件
    15. 

  15. 2025-07-18 00:08:22 - DataLoader - INFO - 发现 5 个JSON文件和 14 个JSONL文件
    16. 

  16. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSON文件: additional_data.json
    17. 

  17. 2025-07-18 00:08:22 - DataLoader - INFO - 从 additional_data.json 加载了 2 条数据
    18. 

  18. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSON文件: enhanced_test.json
    19. 

  19. 2025-07-18 00:08:22 - DataLoader - INFO - 从 enhanced_test.json 加载了 6 条数据
    20. 

  20. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSON文件: cleaned_data.json
    21. 

  21. 2025-07-18 00:08:22 - DataLoader - INFO - 从 cleaned_data.json 加载了 12 条数据
    22. 

  22. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSON文件: training_data.json
    23. 

  23. 2025-07-18 00:08:22 - DataLoader - INFO - 从 training_data.json 加载了 12 条数据
    24. 

  24. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSON文件: test_data.json
    25. 

  25. 2025-07-18 00:08:22 - DataLoader - INFO - 从 test_data.json 加载了 2 条数据
    26. 

  26. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: dedup_modelscope_dataset_sample.jsonl
    27. 

  27. 2025-07-18 00:08:22 - DataLoader - INFO - 从 dedup_modelscope_dataset_sample.jsonl 加载了 100 条数据
    28. 

  28. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: security_only_training_dataset.jsonl
    29. 

  29. 2025-07-18 00:08:22 - DataLoader - INFO - 从 security_only_training_dataset.jsonl 加载了 14322 条数据
    30. 

  30. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: dedup_alpaca_with_cot.jsonl
    31. 

  31. 2025-07-18 00:08:22 - DataLoader - INFO - 从 dedup_alpaca_with_cot.jsonl 加载了 2951 条数据
    32. 

  32. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: dedup_poc_security_training.jsonl
    33. 

  33. 2025-07-18 00:08:22 - DataLoader - INFO - 从 dedup_poc_security_training.jsonl 加载了 1131 条数据
    34. 

  34. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: webshell_dataset_sample.jsonl
    35. 

  35. 2025-07-18 00:08:22 - DataLoader - INFO - 从 webshell_dataset_sample.jsonl 加载了 10 条数据
    36. 

  36. 2025-07-18 00:08:22 - DataLoader - INFO - 正在加载JSONL文件: dedup_modelscope_security_training.jsonl
    37. 

  37. 2025-07-18 00:08:23 - DataLoader - INFO - 从 dedup_modelscope_security_training.jsonl 加载了 11044 条数据
    38. 

  38. 2025-07-18 00:08:23 - DataLoader - INFO - 正在加载JSONL文件: dedup_eval_data.jsonl
    39. 

  39. 2025-07-18 00:08:23 - DataLoader - INFO - 从 dedup_eval_data.jsonl 加载了 478 条数据
    40. 

  40. 2025-07-18 00:08:23 - DataLoader - INFO - 正在加载JSONL文件: dedup_hf_dataset_sample.jsonl
    41. 

  41. 2025-07-18 00:08:23 - DataLoader - INFO - 从 dedup_hf_dataset_sample.jsonl 加载了 100 条数据
    42. 

  42. 2025-07-18 00:08:23 - DataLoader - INFO - 正在加载JSONL文件: final_security_training_dataset.jsonl
    43. 

  43. 2025-07-18 00:08:24 - DataLoader - INFO - 从 final_security_training_dataset.jsonl 加载了 21653 条数据
    44. 

  44. 2025-07-18 00:08:24 - DataLoader - INFO - 正在加载JSONL文件: enhanced_test.jsonl
    45. 

  45. 2025-07-18 00:08:24 - DataLoader - INFO - 从 enhanced_test.jsonl 加载了 2 条数据
    46. 

  46. 2025-07-18 00:08:24 - DataLoader - INFO - 正在加载JSONL文件: dedup_cve_dataset_sample.jsonl
    47. 

  47. 2025-07-18 00:08:24 - DataLoader - INFO - 从 dedup_cve_dataset_sample.jsonl 加载了 100 条数据
    48. 

  48. 2025-07-18 00:08:24 - DataLoader - INFO - 正在加载JSONL文件: dedup_train_data.jsonl
    49. 

  49. 2025-07-18 00:08:24 - DataLoader - INFO - 从 dedup_train_data.jsonl 加载了 4380 条数据
    50. 

  50. 2025-07-18 00:08:24 - DataLoader - INFO - 正在加载JSONL文件: webshell_security_training.jsonl
    51. 

  51. 2025-07-18 00:08:24 - DataLoader - INFO - 从 webshell_security_training.jsonl 加载了 1480 条数据
    52. 

  52. 2025-07-18 00:08:24 - DataLoader - INFO - 正在加载JSONL文件: dedup_cve_security_training.jsonl
    53. 

  53. 2025-07-18 00:08:24 - DataLoader - INFO - 从 dedup_cve_security_training.jsonl 加载了 667 条数据
    54. 

  54. 2025-07-18 00:08:24 - DataLoader - INFO - 总共加载了 58452 条训练数据
    55. 

  55. 2025-07-18 00:08:24 - DataLoader - INFO - 已加载的文件: additional_data.json (2条), enhanced_test.json (6条), cleaned_data.json (12条), training_data.json (12条), test_data.json (2条), dedup_modelscope_dataset_sample.jsonl (100条), security_only_training_dataset.jsonl (14322条), dedup_alpaca_with_cot.jsonl (2951条), dedup_poc_security_training.jsonl (1131条), webshell_dataset_sample.jsonl (10条), dedup_modelscope_security_training.jsonl (11044条), dedup_eval_data.jsonl (478条), dedup_hf_dataset_sample.jsonl (100条), final_security_training_dataset.jsonl (21653条), enhanced_test.jsonl (2条), dedup_cve_dataset_sample.jsonl (100条), dedup_train_data.jsonl (4380条), webshell_security_training.jsonl (1480条), dedup_cve_security_training.jsonl (667条)
    56. 

  56. 2025-07-18 00:08:24 - DataLoader - INFO - 数据分割: 训练集 52606 条，验证集 5846 条
    57. 

  57. 2025-07-18 00:08:24 - DataLoader - INFO - 创建数据集: 52606 个样本
    58. 

  58. 2025-07-18 00:08:24 - DataLoader - INFO - 创建数据集: 5846 个样本
    59. 

  59. 2025-07-18 00:08:24 - SecurityModelTrainer - INFO - 训练准备完成
    60. 

  60. 2025-07-18 00:08:24 - SecurityModelApp - INFO - 训练配置: {'model_loaded': True, 'tokenizer_loaded': True, 'trainer_ready': True, 'config': {'base_model': 'qwen/Qwen2.5-1.5B-Instruct', 'use_lora': True, 'fp16': True, 'batch_size': 1, 'learning_rate': 0.0002, 'num_epochs': 3}, 'train_dataset_size': 52606, 'eval_dataset_size': 5846}
    61. 

  61. 2025-07-18 00:08:24 - SecurityModelApp - INFO - 开始训练...
    62. 

  62. 2025-07-18 00:08:24 - SecurityModelTrainer - INFO - 开始模型训练...
    63.