系统背景

集群上运行大规模 COAWST（wrf roms swan） 耦合算例

问题

• 不同程序的 编译项 与 参数配置 过多且复杂
• 对于一个 IO 项问题，是没设置正确的环境变量导致_没编译功能_**，**还是没设置好参数导致_没启用功能_，无从得知

IO 相关参数

wrf

# IO 方式 2，11，13，102 ...
io_form_history     
io_form_restart      

# Quilt I/O Server
nio_tasks_per_group 
nio_groups

编译设置 (环境变量)：

PNETCDF_QUILT  # 启用基于 PnetCDF 的 Quilt I/O Server 实现

# PNETCDF_QUILT “值得开启”，但前提是 PnetCDF 编译链完整且稳定
# 与 nio_tasks_per_group = n 同时使用， 
# 不启用 也能使用 Quilt  Server（配置：nio_tasks_per_group nio_groups

roms

PIO 相关

编译设置：

• 编译参数参考说明：https://www.myroms.org/wiki/IO
• 宏参考说明：https://www.myroms.org/wiki/Options#PIO_LIB

分析

通过输出文件

通过分析输出文件的 大小/数量 初步判断瓶颈

# 文件数量
find . -maxdepth 1 -type f -name 'ocean_his_*' | wc -l

# 文件大小
du -ch ocean_his_* | tail -1

其中三个程序的输出有：

swan：swaninit*

Roms: ocean_his_* ocean_avg_*

Wrf: wrfout_*

输出占比集中在roms 与 WRF

LOG 分析

• Slrum log 能得到很多信息
• 同时可以使用 darshan 评估 并行IO

基本信息

不同程序分配的节点数

Model Coupling: 

       Ocean Model MPI nodes: 00000 - 00255

       Waves Model MPI nodes: 00256 - 00511

       Atmos Model MPI nodes: 00512 - 01535

Roms 网格切分

Resolution, Grid 01: 970x1198x50,  Parallel Nodes: 256,  Tiling: 16x16

Wrf 网格切分

Ntasks in X           31 , ntasks in Y           32

同时能算出分配到 io 的节点 wrf-nodes - NtasksX*NtasksY

耗时指标

Swan 是耗时最小的部分，不做研究

WRF

• Timing for Writing （写文件耗时)
• Timing for main (计算步长耗时)

Timing for main: time 2026-04-03_23:50:00 on domain   1:    3.69922 elapsed seconds
Timing for Writing wrfout_d01_2026-04-04_00:00:00 for domain        1:    8.47534 elapsed seconds

ROMS

程序运行时间 ≈ Total / roms 分配进程 （log 中的 Average ）

Average:               578.752
 Minimum:               577.427
 Maximum:               578.860

 Nonlinear model elapsed CPU time profile, Grid: 01

  Allocation and array initialization ..............        20.930  ( 0.0188 %)
  Ocean state initialization .......................        51.577  ( 0.0464 %)
  Reading of input data ............................       400.877  ( 0.3608 %)
  Processing of input data .........................        29.871  ( 0.0269 %)
  Processing of output time averaged data ..........        67.112  ( 0.0604 %)
  Computation of vertical boundary conditions ......        21.002  ( 0.0189 %)
  Computation of global information integrals ......        17.770  ( 0.0160 %)
  Writing of output data ...........................     14673.920  (13.2054 %)
  Model 2D kernel ..................................      5109.128  ( 4.5978 %)
  2D/3D coupling, vertical metrics .................       405.235  ( 0.3647 %)
  Omega vertical velocity ..........................       285.565  ( 0.2570 %)
  Equation of state for seawater ...................       961.461  ( 0.8652 %)
  Atmosphere-Ocean bulk flux parameterization ......        62.334  ( 0.0561 %)
  KPP vertical mixing parameterization .............      2940.584  ( 2.6463 %)
  3D equations right-side terms ....................       856.016  ( 0.7704 %)
  3D equations predictor step ......................      1448.507  ( 1.3035 %)
  Pressure gradient ................................       444.194  ( 0.3997 %)
  Harmonic mixing of tracers, geopotentials ........       816.935  ( 0.7352 %)
  Biharmonic mixing of tracers, geopotentials ......       393.041  ( 0.3537 %)
  Harmonic stress tensor, geopotentials ............      2031.193  ( 1.8279 %)
  Corrector time-step for 3D momentum ..............       921.772  ( 0.8295 %)
  Corrector time-step for tracers ..................       937.017  ( 0.8432 %)
  Unused 07 ........................................     41156.609  (37.0379 %)
                                              Total:     74052.650   66.6419 %

  Unique kernel(s) regions profiled ................     74052.650   66.6419 %
  Residual, non-profiled code ......................     37067.639   33.3581 %


 All percentages are with respect to total time =       111120.289


 MPI communications profile, Grid: 01

  Message Passage: 2D halo exchanges ...............      3098.897  ( 2.7888 %)
  Message Passage: 3D halo exchanges ...............      1388.548  ( 1.2496 %)
  Message Passage: 4D halo exchanges ...............       539.617  ( 0.4856 %)
  Message Passage: data broadcast ..................     13393.653  (12.0533 %)
  Message Passage: data reduction ..................       253.303  ( 0.2280 %)
  Message Passage: data gathering ..................      3194.237  ( 2.8746 %)
  Message Passage: data scattering..................      2440.529  ( 2.1963 %)
  Message Passage: point data gathering ............         0.254  ( 0.0002 %)
  Message Passage: synchronization barrier .........         7.261  ( 0.0065 %)
                                              Total:     24316.300   21.8829 %

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

其中需要关注的有：

• Writing of output data
• Unused 07 （闲置状态）
• MPI data broadcast

LOG 分析以及优化思路

优化思路

1. 线性扩展
2. wrf 历史模式：102
3. pnetcdf 并行 io

进行一次完整运行

ROM：

Computation of global information integrals ......      8438.950  ( 0.0370 %)
  Writing of output data ...........................   2007518.132  ( 8.8010 %)
  Model 2D kernel ..................................   2813099.155  (12.3327 %)

 All percentages are with respect to total time =     22810024.664
 
 MPI communications profile, Grid: 01
  Message Passage: 2D halo exchanges ...............   2572137.150  (11.2763 %)
  Message Passage: data broadcast ..................   1064456.268  ( 4.6666 %)
                                              Total:   5866359.029   25.7183 %

1. Unused （ROMS 闲置时间） ≈ 60% ，ROMS 等待 WRF 的结果 --> 参数分配上还有优化空间

WRF：

(使用脚本统计数据指标)

• Timing for main 包括计算、等待 IO 进程、等待耦合的时间
• 这里平均每步等待 20s

设计 30 分钟迭代的快速测试

线性扩展

darshan log

# darshan log version: 3.41
# compression method: ZLIB
# exe: ./coawstM.backup coupling.in 
# uid: 66756
# jobid: 1471163
# start_time: 1779956254
# start_time_asci: Thu May 28 16:17:34 2026
# end_time: 1779957375
# end_time_asci: Thu May 28 16:36:15 2026
# nprocs: 128
# run time: 1120.9576
# metadata: lib_ver = 3.5.0
# metadata: h = romio_no_indep_rw=true;cb_nodes=4

# log file regions
# -------------------------------------------------------
# header: 1328 bytes (uncompressed)
# job data: 243 bytes (compressed)
# record table: 24007 bytes (compressed)
# POSIX module: 33758 bytes (compressed), ver=4
# LUSTRE module: 6917 bytes (compressed), ver=2
# STDIO module: 773 bytes (compressed), ver=2
# HEATMAP module: 9885 bytes (compressed), ver=1

# mounted file systems (mount point and fs type)
# -------------------------------------------------------
# mount entry:        /proc/sys/fs/binfmt_misc        autofs
# mount entry:        /sys/fs/cgroup/unified        cgroup2
# mount entry:        /sys/kernel/tracing        tracefs
# mount entry:        /sys/kernel/config        configfs
# mount entry:        /sys/fs/bpf        bpf
# mount entry:        /dev/mqueue        mqueue
# mount entry:        /vol8        lustre
# mount entry:        /dev        devtmpfs

# *******************************************************
# POSIX module data
# *******************************************************

# description of POSIX counters:
#   POSIX_*: posix operation counts.
#   READS,WRITES,OPENS,SEEKS,STATS,MMAPS,SYNCS,FILENOS,DUPS are types of operations.
#   POSIX_RENAME_SOURCES/TARGETS: total count file was source or target of a rename operation
#   POSIX_RENAMED_FROM: Darshan record ID of the first rename source, if file was a rename target
#   POSIX_MODE: mode that file was opened in.
#   POSIX_BYTES_*: total bytes read and written.
#   POSIX_MAX_BYTE_*: highest offset byte read and written.
#   POSIX_CONSEC_*: number of exactly adjacent reads and writes.
#   POSIX_SEQ_*: number of reads and writes from increasing offsets.
#   POSIX_RW_SWITCHES: number of times access alternated between read and write.
#   POSIX_*_ALIGNMENT: memory and file alignment.
#   POSIX_*_NOT_ALIGNED: number of reads and writes that were not aligned.
#   POSIX_MAX_*_TIME_SIZE: size of the slowest read and write operations.
#   POSIX_SIZE_*_*: histogram of read and write access sizes.
#   POSIX_STRIDE*_STRIDE: the four most common strides detected.
#   POSIX_STRIDE*_COUNT: count of the four most common strides.
#   POSIX_ACCESS*_ACCESS: the four most common access sizes.
#   POSIX_ACCESS*_COUNT: count of the four most common access sizes.
#   POSIX_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).
#   POSIX_*_RANK_BYTES: bytes transferred by the fastest and slowest ranks (for shared files).
#   POSIX_F_*_START_TIMESTAMP: timestamp of first open/read/write/close.
#   POSIX_F_*_END_TIMESTAMP: timestamp of last open/read/write/close.
#   POSIX_F_READ/WRITE/META_TIME: cumulative time spent in read, write, or metadata operations.
#   POSIX_F_MAX_*_TIME: duration of the slowest read and write operations.
#   POSIX_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).
#   POSIX_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).

# WARNING: POSIX_OPENS counter includes both POSIX_FILENOS and POSIX_DUPS counts

# WARNING: POSIX counters related to file offsets may be incorrect if a file is simultaneously accessed by both POSIX and STDIO (e.g., using fileno())
#         - Affected counters include: MAX_BYTE_{READ|WRITTEN}, CONSEC_{READS|WRITES}, SEQ_{READS|WRITES}, {MEM|FILE}_NOT_ALIGNED, STRIDE*_STRIDE

total_POSIX_OPENS: 1167
total_POSIX_FILENOS: 13
total_POSIX_DUPS: 0
total_POSIX_READS: 107443
total_POSIX_WRITES: 759720
total_POSIX_SEEKS: 5894
total_POSIX_STATS: 1477
total_POSIX_MMAPS: -1
total_POSIX_FSYNCS: 0
total_POSIX_FDSYNCS: 0
total_POSIX_RENAME_SOURCES: 0
total_POSIX_RENAME_TARGETS: 0
total_POSIX_RENAMED_FROM: 0
total_POSIX_MODE: 0
total_POSIX_BYTES_READ: 4642184152
total_POSIX_BYTES_WRITTEN: 12596970403
total_POSIX_MAX_BYTE_READ: 6624238151
total_POSIX_MAX_BYTE_WRITTEN: 5010776075
total_POSIX_CONSEC_READS: 103785
total_POSIX_CONSEC_WRITES: 753463
total_POSIX_SEQ_READS: 105843
total_POSIX_SEQ_WRITES: 758835
total_POSIX_RW_SWITCHES: 2439
total_POSIX_MEM_NOT_ALIGNED: 0
total_POSIX_MEM_ALIGNMENT: 8
total_POSIX_FILE_NOT_ALIGNED: 862089
total_POSIX_FILE_ALIGNMENT: 1048576
total_POSIX_MAX_READ_TIME_SIZE: 28834614
total_POSIX_MAX_WRITE_TIME_SIZE: 28996093
total_POSIX_SIZE_READ_0_100: 3548
total_POSIX_SIZE_READ_100_1K: 915
total_POSIX_SIZE_READ_1K_10K: 101545
total_POSIX_SIZE_READ_10K_100K: 144
total_POSIX_SIZE_READ_100K_1M: 572
total_POSIX_SIZE_READ_1M_4M: 667
total_POSIX_SIZE_READ_4M_10M: 11
total_POSIX_SIZE_READ_10M_100M: 41
total_POSIX_SIZE_READ_100M_1G: 0
total_POSIX_SIZE_READ_1G_PLUS: 0
total_POSIX_SIZE_WRITE_0_100: 2234
total_POSIX_SIZE_WRITE_100_1K: 2425
total_POSIX_SIZE_WRITE_1K_10K: 751992
total_POSIX_SIZE_WRITE_10K_100K: 969
total_POSIX_SIZE_WRITE_100K_1M: 643
total_POSIX_SIZE_WRITE_1M_4M: 1276
total_POSIX_SIZE_WRITE_4M_10M: 29
total_POSIX_SIZE_WRITE_10M_100M: 152
total_POSIX_SIZE_WRITE_100M_1G: 0
total_POSIX_SIZE_WRITE_1G_PLUS: 0
total_POSIX_STRIDE1_STRIDE: 4194304
total_POSIX_STRIDE2_STRIDE: 4
total_POSIX_STRIDE3_STRIDE: 4096
total_POSIX_STRIDE4_STRIDE: 1
total_POSIX_STRIDE1_COUNT: 1218
total_POSIX_STRIDE2_COUNT: 288
total_POSIX_STRIDE3_COUNT: 124
total_POSIX_STRIDE4_COUNT: 24
total_POSIX_ACCESS1_ACCESS: 4189
total_POSIX_ACCESS2_ACCESS: 4163
total_POSIX_ACCESS3_ACCESS: 4194304
total_POSIX_ACCESS4_ACCESS: 8192
total_POSIX_ACCESS1_COUNT: 619252
total_POSIX_ACCESS2_COUNT: 109913
total_POSIX_ACCESS3_COUNT: 1849
total_POSIX_ACCESS4_COUNT: 99909
total_POSIX_FASTEST_RANK: -1
total_POSIX_FASTEST_RANK_BYTES: -1
total_POSIX_SLOWEST_RANK: -1
total_POSIX_SLOWEST_RANK_BYTES: -1
total_POSIX_F_OPEN_START_TIMESTAMP: 0.011120
total_POSIX_F_READ_START_TIMESTAMP: 0.014384
total_POSIX_F_WRITE_START_TIMESTAMP: 1.191473
total_POSIX_F_CLOSE_START_TIMESTAMP: 0.021221
total_POSIX_F_OPEN_END_TIMESTAMP: 1035.282613
total_POSIX_F_READ_END_TIMESTAMP: 1035.839145
total_POSIX_F_WRITE_END_TIMESTAMP: 1122.177334
total_POSIX_F_CLOSE_END_TIMESTAMP: 1122.201477
total_POSIX_F_READ_TIME: 7.116889
total_POSIX_F_WRITE_TIME: 56.739679
total_POSIX_F_META_TIME: 6.663915
total_POSIX_F_MAX_READ_TIME: 0.119348
total_POSIX_F_MAX_WRITE_TIME: 0.111346
total_POSIX_F_FASTEST_RANK_TIME: 0.000000
total_POSIX_F_SLOWEST_RANK_TIME: 0.000000
total_POSIX_F_VARIANCE_RANK_TIME: 0.000000
total_POSIX_F_VARIANCE_RANK_BYTES: 0.000000

# *******************************************************
# STDIO module data
# *******************************************************

# description of STDIO counters:
#   STDIO_{OPENS|FDOPENS|WRITES|READS|SEEKS|FLUSHES} are types of operations.
#   STDIO_BYTES_*: total bytes read and written.
#   STDIO_MAX_BYTE_*: highest offset byte read and written.
#   STDIO_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).
#   STDIO_*_RANK_BYTES: bytes transferred by the fastest and slowest ranks (for shared files).
#   STDIO_F_*_START_TIMESTAMP: timestamp of the first call to that type of function.
#   STDIO_F_*_END_TIMESTAMP: timestamp of the completion of the last call to that type of function.
#   STDIO_F_*_TIME: cumulative time spent in different types of functions.
#   STDIO_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).
#   STDIO_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).

# WARNING: STDIO_OPENS counter includes STDIO_FDOPENS count

# WARNING: STDIO counters related to file offsets may be incorrect if a file is simultaneously accessed by both STDIO and POSIX (e.g., using fdopen())
#         - Affected counters include: MAX_BYTE_{READ|WRITTEN}

total_STDIO_OPENS: 19
total_STDIO_FDOPENS: 0
total_STDIO_READS: 16
total_STDIO_WRITES: 0
total_STDIO_SEEKS: 26
total_STDIO_FLUSHES: 0
total_STDIO_BYTES_WRITTEN: 0
total_STDIO_BYTES_READ: 54526051
total_STDIO_MAX_BYTE_READ: 4194303
total_STDIO_MAX_BYTE_WRITTEN: 0
total_STDIO_FASTEST_RANK: -1
total_STDIO_FASTEST_RANK_BYTES: -1
total_STDIO_SLOWEST_RANK: -1
total_STDIO_SLOWEST_RANK_BYTES: -1
total_STDIO_F_META_TIME: 0.025318
total_STDIO_F_WRITE_TIME: 0.000000
total_STDIO_F_READ_TIME: 0.107740
total_STDIO_F_OPEN_START_TIMESTAMP: 2.131395
total_STDIO_F_CLOSE_START_TIMESTAMP: 2.132253
total_STDIO_F_WRITE_START_TIMESTAMP: 0.000000
total_STDIO_F_READ_START_TIMESTAMP: 2.136211
total_STDIO_F_OPEN_END_TIMESTAMP: 1035.230478
total_STDIO_F_CLOSE_END_TIMESTAMP: 1035.235342
total_STDIO_F_WRITE_END_TIMESTAMP: 0.000000
total_STDIO_F_READ_END_TIMESTAMP: 1035.233100
total_STDIO_F_FASTEST_RANK_TIME: 0.000000
total_STDIO_F_SLOWEST_RANK_TIME: 0.000000
total_STDIO_F_VARIANCE_RANK_TIME: 0.000000
total_STDIO_F_VARIANCE_RANK_BYTES: 0.000000

# darshan log version: 3.41
# compression method: ZLIB
# exe: ./coawstM.backup coupling.in 
# uid: 66756
# jobid: 1134026
# start_time: 1779960614
# start_time_asci: Thu May 28 17:30:14 2026
# end_time: 1779961290
# end_time_asci: Thu May 28 17:41:30 2026
# nprocs: 1024
# run time: 675.4815
# metadata: lib_ver = 3.5.0
# metadata: h = romio_no_indep_rw=true;cb_nodes=4

# log file regions
# -------------------------------------------------------
# header: 1328 bytes (uncompressed)
# job data: 243 bytes (compressed)
# record table: 192120 bytes (compressed)
# POSIX module: 252901 bytes (compressed), ver=4
# LUSTRE module: 54221 bytes (compressed), ver=2
# STDIO module: 758 bytes (compressed), ver=2
# HEATMAP module: 68184 bytes (compressed), ver=1

# mounted file systems (mount point and fs type)
# -------------------------------------------------------
# mount entry:        /proc/sys/fs/binfmt_misc        autofs
# mount entry:        /sys/fs/cgroup/unified        cgroup2
# mount entry:        /sys/kernel/tracing        tracefs
# mount entry:        /sys/kernel/config        configfs
# mount entry:        /sys/fs/bpf        bpf
# mount entry:        /dev/mqueue        mqueue
# mount entry:        /vol8        lustre
# mount entry:        /dev        devtmpfs

# *******************************************************
# POSIX module data
# *******************************************************

# description of POSIX counters:
#   POSIX_*: posix operation counts.
#   READS,WRITES,OPENS,SEEKS,STATS,MMAPS,SYNCS,FILENOS,DUPS are types of operations.
#   POSIX_RENAME_SOURCES/TARGETS: total count file was source or target of a rename operation
#   POSIX_RENAMED_FROM: Darshan record ID of the first rename source, if file was a rename target
#   POSIX_MODE: mode that file was opened in.
#   POSIX_BYTES_*: total bytes read and written.
#   POSIX_MAX_BYTE_*: highest offset byte read and written.
#   POSIX_CONSEC_*: number of exactly adjacent reads and writes.
#   POSIX_SEQ_*: number of reads and writes from increasing offsets.
#   POSIX_RW_SWITCHES: number of times access alternated between read and write.
#   POSIX_*_ALIGNMENT: memory and file alignment.
#   POSIX_*_NOT_ALIGNED: number of reads and writes that were not aligned.
#   POSIX_MAX_*_TIME_SIZE: size of the slowest read and write operations.
#   POSIX_SIZE_*_*: histogram of read and write access sizes.
#   POSIX_STRIDE*_STRIDE: the four most common strides detected.
#   POSIX_STRIDE*_COUNT: count of the four most common strides.
#   POSIX_ACCESS*_ACCESS: the four most common access sizes.
#   POSIX_ACCESS*_COUNT: count of the four most common access sizes.
#   POSIX_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).
#   POSIX_*_RANK_BYTES: bytes transferred by the fastest and slowest ranks (for shared files).
#   POSIX_F_*_START_TIMESTAMP: timestamp of first open/read/write/close.
#   POSIX_F_*_END_TIMESTAMP: timestamp of last open/read/write/close.
#   POSIX_F_READ/WRITE/META_TIME: cumulative time spent in read, write, or metadata operations.
#   POSIX_F_MAX_*_TIME: duration of the slowest read and write operations.
#   POSIX_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).
#   POSIX_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).

# WARNING: POSIX_OPENS counter includes both POSIX_FILENOS and POSIX_DUPS counts

# WARNING: POSIX counters related to file offsets may be incorrect if a file is simultaneously accessed by both POSIX and STDIO (e.g., using fileno())
#         - Affected counters include: MAX_BYTE_{READ|WRITTEN}, CONSEC_{READS|WRITES}, SEQ_{READS|WRITES}, {MEM|FILE}_NOT_ALIGNED, STRIDE*_STRIDE

total_POSIX_OPENS: 8943
total_POSIX_FILENOS: 13
total_POSIX_DUPS: 0
total_POSIX_READS: 830835
total_POSIX_WRITES: 899851
total_POSIX_SEEKS: 5606
total_POSIX_STATS: 11461
total_POSIX_MMAPS: -1
total_POSIX_FSYNCS: 0
total_POSIX_FDSYNCS: 0
total_POSIX_RENAME_SOURCES: 0
total_POSIX_RENAME_TARGETS: 0
total_POSIX_RENAMED_FROM: 0
total_POSIX_MODE: 0
total_POSIX_BYTES_READ: 10420613385
total_POSIX_BYTES_WRITTEN: 13183610642
total_POSIX_MAX_BYTE_READ: 6624238151
total_POSIX_MAX_BYTE_WRITTEN: 5010776075
total_POSIX_CONSEC_READS: 820249
total_POSIX_CONSEC_WRITES: 893034
total_POSIX_SEQ_READS: 822019
total_POSIX_SEQ_WRITES: 898406
total_POSIX_RW_SWITCHES: 2439
total_POSIX_MEM_NOT_ALIGNED: 0
total_POSIX_MEM_ALIGNMENT: 8
total_POSIX_FILE_NOT_ALIGNED: 1712797
total_POSIX_FILE_ALIGNMENT: 1048576
total_POSIX_MAX_READ_TIME_SIZE: 8192
total_POSIX_MAX_WRITE_TIME_SIZE: 28834614
total_POSIX_SIZE_READ_0_100: 17852
total_POSIX_SIZE_READ_100_1K: 915
total_POSIX_SIZE_READ_1K_10K: 810585
total_POSIX_SIZE_READ_10K_100K: 99
total_POSIX_SIZE_READ_100K_1M: 665
total_POSIX_SIZE_READ_1M_4M: 667
total_POSIX_SIZE_READ_4M_10M: 11
total_POSIX_SIZE_READ_10M_100M: 41
total_POSIX_SIZE_READ_100M_1G: 0
total_POSIX_SIZE_READ_1G_PLUS: 0
total_POSIX_SIZE_WRITE_0_100: 2221
total_POSIX_SIZE_WRITE_100_1K: 2448
total_POSIX_SIZE_WRITE_1K_10K: 892017
total_POSIX_SIZE_WRITE_10K_100K: 1065
total_POSIX_SIZE_WRITE_100K_1M: 643
total_POSIX_SIZE_WRITE_1M_4M: 1276
total_POSIX_SIZE_WRITE_4M_10M: 29
total_POSIX_SIZE_WRITE_10M_100M: 152
total_POSIX_SIZE_WRITE_100M_1G: 0
total_POSIX_SIZE_WRITE_1G_PLUS: 0
total_POSIX_STRIDE1_STRIDE: 4194304
total_POSIX_STRIDE2_STRIDE: 4096
total_POSIX_STRIDE3_STRIDE: 1
total_POSIX_STRIDE4_STRIDE: 26
total_POSIX_STRIDE1_COUNT: 1218
total_POSIX_STRIDE2_COUNT: 124
total_POSIX_STRIDE3_COUNT: 24
total_POSIX_STRIDE4_COUNT: 20
total_POSIX_ACCESS1_ACCESS: 4189
total_POSIX_ACCESS2_ACCESS: 8192
total_POSIX_ACCESS3_ACCESS: 4530
total_POSIX_ACCESS4_ACCESS: 2334
total_POSIX_ACCESS1_COUNT: 676442
total_POSIX_ACCESS2_COUNT: 761680
total_POSIX_ACCESS3_COUNT: 2193
total_POSIX_ACCESS4_COUNT: 2176
total_POSIX_FASTEST_RANK: -1
total_POSIX_FASTEST_RANK_BYTES: -1
total_POSIX_SLOWEST_RANK: -1
total_POSIX_SLOWEST_RANK_BYTES: -1
total_POSIX_F_OPEN_START_TIMESTAMP: 0.090209
total_POSIX_F_READ_START_TIMESTAMP: 0.097283
total_POSIX_F_WRITE_START_TIMESTAMP: 0.853514
total_POSIX_F_CLOSE_START_TIMESTAMP: 0.101990
total_POSIX_F_OPEN_END_TIMESTAMP: 565.882341
total_POSIX_F_READ_END_TIMESTAMP: 570.081440
total_POSIX_F_WRITE_END_TIMESTAMP: 675.602195
total_POSIX_F_CLOSE_END_TIMESTAMP: 675.626367
total_POSIX_F_READ_TIME: 29.532315
total_POSIX_F_WRITE_TIME: 64.455771
total_POSIX_F_META_TIME: 286.078577
total_POSIX_F_MAX_READ_TIME: 0.102825
total_POSIX_F_MAX_WRITE_TIME: 0.078899
total_POSIX_F_FASTEST_RANK_TIME: 0.000000
total_POSIX_F_SLOWEST_RANK_TIME: 0.000000
total_POSIX_F_VARIANCE_RANK_TIME: 0.000000
total_POSIX_F_VARIANCE_RANK_BYTES: 0.000000

# *******************************************************
# STDIO module data
# *******************************************************

# description of STDIO counters:
#   STDIO_{OPENS|FDOPENS|WRITES|READS|SEEKS|FLUSHES} are types of operations.
#   STDIO_BYTES_*: total bytes read and written.
#   STDIO_MAX_BYTE_*: highest offset byte read and written.
#   STDIO_*_RANK: rank of the processes that were the fastest and slowest at I/O (for shared files).
#   STDIO_*_RANK_BYTES: bytes transferred by the fastest and slowest ranks (for shared files).
#   STDIO_F_*_START_TIMESTAMP: timestamp of the first call to that type of function.
#   STDIO_F_*_END_TIMESTAMP: timestamp of the completion of the last call to that type of function.
#   STDIO_F_*_TIME: cumulative time spent in different types of functions.
#   STDIO_F_*_RANK_TIME: fastest and slowest I/O time for a single rank (for shared files).
#   STDIO_F_VARIANCE_RANK_*: variance of total I/O time and bytes moved for all ranks (for shared files).

# WARNING: STDIO_OPENS counter includes STDIO_FDOPENS count

# WARNING: STDIO counters related to file offsets may be incorrect if a file is simultaneously accessed by both STDIO and POSIX (e.g., using fdopen())
#         - Affected counters include: MAX_BYTE_{READ|WRITTEN}

total_STDIO_OPENS: 19
total_STDIO_FDOPENS: 0
total_STDIO_READS: 16
total_STDIO_WRITES: 0
total_STDIO_SEEKS: 26
total_STDIO_FLUSHES: 0
total_STDIO_BYTES_WRITTEN: 0
total_STDIO_BYTES_READ: 54526051
total_STDIO_MAX_BYTE_READ: 4194303
total_STDIO_MAX_BYTE_WRITTEN: 0
total_STDIO_FASTEST_RANK: -1
total_STDIO_FASTEST_RANK_BYTES: -1
total_STDIO_SLOWEST_RANK: -1
total_STDIO_SLOWEST_RANK_BYTES: -1
total_STDIO_F_META_TIME: 0.019924
total_STDIO_F_WRITE_TIME: 0.000000
total_STDIO_F_READ_TIME: 0.106629
total_STDIO_F_OPEN_START_TIMESTAMP: 1.386794
total_STDIO_F_CLOSE_START_TIMESTAMP: 1.387560
total_STDIO_F_WRITE_START_TIMESTAMP: 0.000000
total_STDIO_F_READ_START_TIMESTAMP: 1.391299
total_STDIO_F_OPEN_END_TIMESTAMP: 565.481624
total_STDIO_F_CLOSE_END_TIMESTAMP: 565.487266
total_STDIO_F_WRITE_END_TIMESTAMP: 0.000000
total_STDIO_F_READ_END_TIMESTAMP: 565.484678
total_STDIO_F_FASTEST_RANK_TIME: 0.000000
total_STDIO_F_SLOWEST_RANK_TIME: 0.000000
total_STDIO_F_VARIANCE_RANK_TIME: 0.000000
total_STDIO_F_VARIANCE_RANK_BYTES: 0.000000

核心指标直观对比表

• 可以看出因为串行模型下，扩展核数导致 Lustre 的元数据服务器（MDS）拥堵

WRF IO MODE 102

该模式让 WRF 每个进程分别负责各自的文件，避免了上千进程的写文件的拥堵；该方法也是 WRF 论坛中对边 >500 规模网格的推荐方法

ROM

Average:               257.028
 Minimum:               253.560
 Maximum:               257.158


  Writing of output data ...........................     14544.184  (29.4719 %)
  Model 2D kernel ..................................      4990.394  (10.1124 %)
  Unused 07 ........................................       330.811  ( 0.6703 %)
                                              Total:     33027.832   66.9266 %

  Unique kernel(s) regions profiled ................     33027.832   66.9266 %
  Residual, non-profiled code ......................     16321.481   33.0734 %


 All percentages are with respect to total time =        49349.313


 MPI communications profile, Grid: 01

  Message Passage: 2D halo exchanges ...............      2917.842  ( 5.9126 %)
  Message Passage: 3D halo exchanges ...............      1340.847  ( 2.7171 %)
  Message Passage: 4D halo exchanges ...............       515.030  ( 1.0436 %)
  Message Passage: data broadcast ..................     13709.517  (27.7806 %)
  Message Passage: data reduction ..................       286.763  ( 0.5811 %)
  Message Passage: data gathering ..................      2557.551  ( 5.1825 %)
  Message Passage: data scattering..................      2209.861  ( 4.4780 %)
  Message Passage: point data gathering ............         0.976  ( 0.0020 %)
  Message Passage: synchronization barrier .........         3.564  ( 0.0072 %)
                                              Total:     23541.949   47.7047 %

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

WRF

ROM 中的 Unused 07 （空闲时间）与 WRF 中的 writing 耗时都大幅缩短

• 该方法会导致大量碎片文件；需要后处理 (可以另起程序一边生成新的数据，一边合并) 总速度时也有显著提升

看完了？说点什么呢

| 一般HPC集群中自带的mpi4py netcdf4 或通过上传的whl安装的库，是指向库本身的netcdf4.so，不支持并行功能

重点： 需要手动编译、安装：mpi4py netcdf4

前置准备：

服务器依赖

使用服务器 mpi，netCDF，hdf5

module add ...

必须确认服务器上的 netcdf 开启并行

nc-config --has-parallel -> YES

Pip whl

建议使用虚拟环境

• Cython
• Cffi
• pyyaml

# 联网服务器
pip download -d ./python_whl --python-version 3.10 --platform manylinux2014_aarch64 --only-binary :all: **xxx**

# 离线服务器
pip install --no-index --find-links=./python_whl **xxx**

安装包

tar -zxvf netcdf4...
tar -zxvf mpi4py...

本次使用：

netcdf4: 1.7.4

mpi4py: 4.1.2

环境变量

export CC=mpicc
export USE_NCCONFIG=1
export NETCDF4_DIR=...
export HDF5_DIR=...

编译

mpi4py

cd mpi4py...
python setup.py build --mpicc=$(which mpicc)

yhrun --mpi=pmix -n 2 -p mt_module python3 -c "from mpi4py import MPI; print(f'Rank {MPI.COMM_WORLD.Get_rank()}')"

NETCDF4

cd netcdf4...

python setup.py clean --all
python setup.py build
python setup.py install

python3 -c "import netCDF4; print(netCDF4.__version__)"

ERROR记录

NETCDF 编译过程

python setup.py install

ERROR

ValueError: invalid pyproject.toml config: `project.license`.
configuration error: `project.license` must be valid exactly by one definition (2 matches found):

    - keys:
        'file': {type: string}
      required: ['file']
    - keys:
        'text': {type: string}
      required: ['text']

在较新的 Python 打包标准中，project.license 只能选择 file 或 text 其中之一，但 netCDF4 的某些版本在 pyproject.toml 中同时写了这两项，导致校验失败。

解决

license = "MIT"
license-files = ["LICENSE"]

# 改为：

license = {file = "LICENSE"}

看完了？说点什么呢

本文以一个典型的三层多层感知机（MLP）为例，梳理神经网络前向传播与反向传播的数学基础，并在此基础上，从系统视角剖析单次训练迭代中的 GPU 显存占用情况及相应的显存优化策略。

一、 数学基础：基于链式法则的反向传播

一个三层 MLP 模型，其参数矩阵分布为 W_1, W_2, W_3，输入为 x，真实标签为 y。

1. 前向传播 (Forward Propagation)

前向传播的核心是计算每一层的中间激活值（Activations）以及最终的预测值 \hat{y}。为保持表达严谨且简洁，省略偏置项并合并激活函数，各层计算过程可表示为：

• 第一层： h_1 = f_1(W_1, x)
• 第二层： h_2 = f_2(W_2, h_1)
• 第三层： \hat{y} = f_3(W_3, h_2)

从全局视角来看，整个前向传播是一个深度的嵌套复合函数，其最终的损失函数（Loss）计算如下：

$L = \text{Loss}(f_3(W_3, f_2(W_2, f_1(W_1, x))))$

2. 反向传播 (Backpropagation)

训练的核心目标是最小化误差 L。我们需要求解损失 L 对各层参数 W 的偏导数（梯度），即量化参数微小变化对最终误差的边际影响。该过程严格依赖微积分中的链式法则（Chain Rule）。

• 第三层梯度： 误差直接对 W_3 求导，等于总误差对预测值的偏导乘以预测值对 W_3 的偏导。

  $ \frac{\partial L}{\partial W_3} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial W_3} $

• 第二层梯度： 误差需先传导至第二层的输出 h_2，再对 W_2 求导。

  $ \frac{\partial L}{\partial W_2} = \underbrace{\frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_2}}_{\text{传导至 } h_2 \text{ 的误差}} \cdot \frac{\partial h_2}{\partial W_2} $

• 第一层梯度： 同理，误差依次反向传播至第一层。

  $ \frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_2} \cdot \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1} $

3. 参数更新 (Weight Update)

获取各层梯度后，通过梯度下降（Gradient Descent）算法，结合学习率 \eta 对模型参数进行迭代更新：

• W_3 \leftarrow W_3 - \eta \cdot \frac{\partial L}{\partial W_3}
• W_2 \leftarrow W_2 - \eta \cdot \frac{\partial L}{\partial W_2}
• W_1 \leftarrow W_1 - \eta \cdot \frac{\partial L}{\partial W_1}

二、 系统视角：单次迭代的 GPU 显存剖析

在上述数学过程转化为工程代码并在 GPU 上执行时，显存（VRAM）的占用是制约模型规模的核心瓶颈。在一次 Forward + Backward 循环中，GPU 显存主要被以下两类数据占据：

1. 静态与输入数据

• 模型参数 (Model Parameters): 即上述的 W_1, W_2, W_3。
• 输入与标签 (Inputs & Labels): 维度通常为 Batch_Size \times 数据维度的大小。

2. 动态生成数据（训练时激增的开销）

• 中间激活值 (Activations, H): 在计算链式法则时，反向传播需要用到前向传播的中间结果（如 h_1, h_2）。因此这些状态必须驻留在显存中 (直到计算出 n-1 层才会释放内存)，其占用大小与 Batch_Size 输入序列长度 呈严格的线性正相关关系。
• 参数梯度 (Gradients, \nabla W): 大小与模型参数 W 完全一致。
• 优化器状态 (Optimizer States): 如果使用 Adam 等具有动量机制的优化器，需要额外记录每个参数的过去一阶动量（Momentum）和二阶方差（Variance）。并为了保证精度使用 32位，这将消耗 4 \times 模型参数大小的额外显存。

以目前最热门的模型llama、qwen等，最小规模 7B 至少 14GB，一次训练单次batch size \times 6=84GB （模型参数[1] 梯度[1] 优化器[4]），已经 超出一张A100的大小，所以如今大模型训练设计出了各种显存优化策略。

三、 显存优化：存算置换与高效训练策略

针对上述显存瓶颈，工程上常采用“以计算时间换取显存空间”或“降低数值精度”的策略。

1. 核心“存算置换”策略概览

(以下提及的存算置换策略，其底层原理与工程实现机制将在后续的专栏文章中进行详细深度解析。)

• 小批次 + 梯度累加 (Small Batch + Gradient Accumulation): 在时间维度上拆分大 Batch，通过多次小 Batch 的前向/反向传播累加梯度，绕过单次激活值过大的显存限制。（但依旧会 OOM）
• ZeRO-Offload (算时加载): 将优化器状态或梯度等暂时卸载至 CPU 内存（RAM），在需要计算时再通过 PCIe 调度至 GPU显存。
• 梯度检查点 (Gradient Checkpointing): 前向传播时主动丢弃部分中间激活值 H，在反向传播经过该层时重新计算。这是一种典型的以增加计算量（约30%）换取显著显存节省的策略。

2. 其他正交优化技术

工业实践上更为常用

• 混合精度与量化 (Mixed Precision / Quantization): 将传统的 FP32 运算降阶为 FP16/BF16，甚至 INT8/INT4，成倍削减显存占用并提升计算吞吐。
• 参数高效微调 (PEFT, 如 LoRA): 在微调阶段冻结主干网络（不保存其激活值与优化器状态），仅注入极少量的可训练参数矩阵，将训练显存开销降低至全量微调的零头。

看完了？说点什么呢

最近使用AI Agent, 框架开发web应用。然而设计数据库或前后端交互时，还需自行设计后端。当然过于跨域等老生常谈的问题不用过多赘述，因为Nginx解决了大部分。在开发过程中总结一些关于Nginx配置踩坑并额外补充的一些前端知识。

一个简单的配置

## web
    location /tb/ {
    alias D:/Projects/xxx/dist/;
    index index.html;
    try_files $uri $uri/ /tallybook/index.html;
    }
## api
    location /tbapi/ {
        proxy_pass http://127.0.0.1:8000/;
        proxy_redirect off;
    }

root 与 alias

localtion 模块中用来指定映射服务器静态资源文件的命令

• root 是追加：它把请求的 URL 拼接到 root 指定的路径后面。
• alias 是替换：它用 alias 指定的路径 替换掉 location 匹配的部分。

当浏览器访问：*http://site.com/images/cat.jpg 时*

location /images/ {
    root /var/www/html;
}

→ /var/www/html/images/cat.jpg

location /images/ {
    alias /var/www/html/;
}

→ /var/www/html/cat.jpg

注意：如果 location 后面有 /（例如 /images/），那么 alias 后面必须也要加 /。

proxy_pass 反向代理的路径透传

对于nginx设置

    location /downloader/data/ {
        proxy_pass http://127.0.0.1:8000/;

如果设置了autoindex on; location 建议以 / 结尾

注意，此时只会路由到/downloader/data 或 /

🕳 python http.server

最好的debug方式就是在 do_GET中打印地址

    def do_GET(self):
        print(self.path)
        # ...
        super().do_GET()

启用 http.server 需要注意的坑
场景：将http.server作为静态资源服务器，指定资源目录

如果希望指定目录. 则需要在python后端重写**path** ，这种方法在需要修改目录的情况下最方便。

# 将 Nginx 传来的虚拟路径替换为本地实际目录
self.path = self.path.replace('/downloader/data', f'/{DIRECTORY}')
# 指定 http.server 访问的目录
# self.path = f'/{DIRECTORY}' + self.path

try_files 单页应用（SPA）的部署策略

try_files $uri $uri/ /index.html;

为什么这样写：

现代 SPA（单页应用）

现在的前端框架（如 Vue/React），整个网站其实只有一个 HTML 文件（通常是 index.html）。页面的切换（路由）是由浏览器里的 JavaScript 控制的，而不是服务器。

问题来了：

1. 用户访问首页 http://site.com/ -> Nginx 返回 index.html -> JS 加载 -> 此时用户点击按钮跳转到 /user/profile。
   • 注意：这时候浏览器地址栏变了，但浏览器并没有向 Nginx 发送请求，是 JS 把页面内容变了。一切正常。
2. 但是！ 用户在 /user/profile 这个页面，按了一下 F5 刷新。
3. 浏览器向 Nginx 发送请求：GET /user/profile。
4. Nginx 傻眼了：我去哪里找 /user/profile 这个文件？根目录下根本没有 user 文件夹，更没有 profile 文件。
5. Nginx 默认行为：直接报 404 错误。

作用 （三步走）

先尝试访问路由文件→ 访问目录 + index/html（如果存在）→ 指定的html (兜底)

一种优雅的**“静态资源优先，应用入口兜底”** 的设计模式

看完了？说点什么呢

控制移动

基础移动

创建一个物体

进入蓝图 （蓝图关系）

蓝图设置

F7 编译蓝图

设置移动性

预览

Alt+P

非线性移动

添加时间轴

编辑时间轴

接入时间轴

Spline 移动

通过 spline 设计物体运动路线，实现复杂的运动路线

• 创建蓝图 -> 赋予 spline 组件
• 场景中编辑线条 （拖动、alt 添加节点）
• 运动物体引用 spline 位置

创建 spline 蓝图

蓝图

最右侧为 Ftransform 变量，可视为 位置、旋转、缩放 的结构体

设置路径

从内容菜单拖出对象

Alt + 🖱️ 左键端点拉出新的路径点

移动物体引用路径

创建一个新的蓝图类

打开蓝图为其添加一个 spline 蓝图的引用，（相当于一个指向该类对象的指针）

记得编译

绑定对象

编辑蓝图

预览

看完了？说点什么呢

过程

开启c++编程后，编译报错

Expecting to find a type to be declared in a module rules named 'VisualStudioTools' in 'UE5Rules, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.  This type must derive from the 'ModuleRules' type defined by UnrealBuildTool.

缺少 VisualStudioTools , 疑似新建c++项目，初始化时安装插件时选择 安装到驱动 导致。 应该选择安装到项目

解决方法

项目目录新建Plugins目录， 将 ...\UE_5.x\Engine\Plugins\VisualStudioTools 拷贝到该目录下。

其他

类似问题，同理解决方法

Expecting to find a type to be declared in a module rules named 'XXX' in 'UE5Rules

看完了？说点什么呢

突然接到任务，做一款基于UE5的项目，大概两个月时间， 这不可能完成的吧。从头开始学估计要花不少时间，虽然项目上只需要关注 C++ 侧的编程，只能先学再说了。
迅速了解了下UE5的作品。貌似可以开发VR 项目。刚好用上在角落吃灰的quest3，原本也有计划在3年内做款 VR Game，这下提前计划了。最近各种学习计划被提前 等于没提前

针对项目做计划

近期目标就是实现运动与事件交互吧 （2025.10.25）

• C++ Class 如何使用 （如何体现到blueprint）
• 实现物体移动
• 物体状态触发事件
• 物体与物体间的交互

看完了？说点什么呢

天津神户园，纪念中日的两个港口城市友好交往而建。可惜到时天时以晚，加上手机电量告急。匆匆拍上几张照片便乘车返回了。
下午到天津市区开会，刚好在水上公园边上，会议结束时便顺路散步到了神户园。 听闻这个中日蜜月期建成的园林很久了，但地方太偏了（没一条顺路的地铁线到达）

从里面拍摄，经典的日式入口

内部小道和建筑都极具日式风格⛩️ 。虽然看着挺亮，全靠手动快门救活

、

看完了？说点什么呢

记一次wsl下的环境踩坑，折腾一个多小时，发现了奇怪的解决方法

发现问题

先上环境：

import onnxruntime as ort
ort.get_available_providers()

显示可用

['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']

报错段，识别不到 gpu ，但环境下pytorch能使用gpu

session = ort.InferenceSession("yolov8n/best.onnx", providers=["CUDAExecutionProvider"])

*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:129 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:121 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, common::Status> = void] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=DESKTOP-LBBLQ7H ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=282 ; expr=cudaSetDevice(info_.device_id); 

 when using ['CUDAExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************

尝试 debug

查阅资料，认为的 onnxruntime 找不到 cudnn, 而 pytorch 是自带 cudnn 的。

wsl 中的 cuda toolkit 都是调用window下的，在win下安装好后一般不必在wsl重新安装

apt 安装库，修改环境变量 ... 之后便报了 not found 错误, 然而继续把缺少的安装上并修改环境变量后，在终端执行 nvidia-smi 发现无效了, 赶紧把环境变量注释掉。

意外解决

原本打算就这样算了，服务器用 CPU （i7-13700） 也足够，在中间需要数据转换时 import torch 报错 undefined symbol,一开始以为还是把环境搞崩了，但开了新的 .py 发现能正常使用。之后尝试在使用 onnx 前多加一行

import torch # add
import onnxruntime as ort
session = ort.InferenceSession("yolov8n/best.onnx", providers=["CUDAExecutionProvider"])

没想到真就通过了...

记录引起报错的代码

onnxruntime: 无法识别 GPU

import onnxruntime as ort
session = ort.InferenceSession("yolov8n/best.onnx", providers=["CUDAExecutionProvider"])

pytorch: undefined symbol 报错

import onnxruntime
import torch

看完了？说点什么呢

使用hashcat掩码模式破解文档时，一开始以为设计mask，会和正则一样，虽然复杂但使用GPT之类的工具能快速构建出规则。但可能是版本原因或其他问题，GPT、deepseek提供的方案都无效

看完了？说点什么呢