光谱工具集:采集、回放、寻峰、波长标定

- spectro_core.py: 共享核心(加载/标定/基线/寻峰/高斯拟合/双峰联合拟合/CSV-JSON)

- peak_finder.py: 寻峰与峰信息导出,--pick 交互选峰(含 RAW 预览)

- wavelength_calibration.py: 改用 core 寻峰,汞灯 577/579 双峰可分

- wavelength_player.py / hq_average.py: 复用 core 的标定与解码加载
This commit is contained in:
starcells committed 2026-11-14 13:48:24 +08:00
commit 45249b59f9
24 files changed
+11449

No files matched your search

+14
View File
@@ -0,0 +1,14 @@
# 行尾统一:仓库内文本一律 LF(Windows 检出仍可用)
* text=auto
*.py text eol=lf
*.md text eol=lf
*.txt text eol=lf
*.csv text eol=lf
*.json text eol=lf
*.ps1 text eol=crlf
# 二进制,不做行尾转换
*.png binary
*.jpg binary
*.npy binary
*.bin binary
+33
View File
@@ -0,0 +1,33 @@
# ---- 采集数据:体积大,不入库 -------------------------------------------
recordings/
*.bin
*.jsonl
# ---- 运行产物 -----------------------------------------------------------
peak_exports/
*.npy
*.log
# ---- Python -------------------------------------------------------------
__pycache__/
*.py[cod]
*.so
.venv/
venv/
env/
.pytest_cache/
.mypy_cache/
.ruff_cache/
# ---- 编辑器 / 系统 ------------------------------------------------------
.vscode/
.idea/
.DS_Store
Thumbs.db
desktop.ini
# ---- 临时文件 -----------------------------------------------------------
*.tmp
*.bak
*.orig
*.rej
+283
View File
@@ -0,0 +1,283 @@
# 寻峰与峰信息导出(peak_finder.py)
`peak_finder.py` 在 `hq_record.py` 录制的 RAW 数据里找谱线峰:用 `hq_spectrum.py`
的解码和降维方式得到 R/G/B 三条谱,扣除局部基线后寻峰、拟合、给出峰形和**像素坐标**,
最后把**选中的峰**导出为图片、CSV、JSON 和原始像素块。
**本程序不做标定。** 它测的是传感器列坐标(`sensor_x_px`);只有当你在命令行里显式给出
`--calibration <某个 calibration.json>` 时,才会按该文件里的系数把列坐标换算成波长
(`wavelength_nm = polyval(coefficients_descending, sensor_x_px)`)。不给就只输出像素,
波长列留空。这会避免把某一次实验的标定悄悄套到别的录制上——标定是否适用由你判断,
依据(系数、确认范围、参考录制)会原样写进 `peaks.json`。
支持的数据是 `hq_record.py` 录制的 RAW(`BGGR_PISP_COMP1`、2028×1520、uint8 容器)。
逐帧元数据里的黑电平(`SensorBlackLevels`,本数据集为 4096)会被自动使用。
依赖:Python 3 + NumPy + Matplotlib,**不需要 SciPy**。脚本放在 `hq_spectrum.py`
同目录即可(本目录里那份 `hq_spectrum.py` 是 2026-11-14 版的副本,方便单独运行;
你自己的 `hq_spectrum.py` 更新后记得同步过来,或用 `--module-dir` 指过去)。
兼容性:与 2026-10-24 版和 2026-11-14 版 `hq_spectrum.py` 都测过。新版把 PC1B 解码器
改成了按需处理 qmode 3 + 直接写入 uint16(实测三份录制逐像素完全一致),并重写了
`make_preview`——后者现在要求**整数** Bayer 帧,本程序已改为把原始 uint16 帧(多帧
平均时是四舍五入后的均值)交给它。
## 快速开始
在 `peak_finder.py` 所在目录运行。先设两个路径变量(换成你自己的),下面直接引用:
```powershell
$rec = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417" # 录制目录
$cal = "C:\Users\peng0\Dev\Light\Programs\wavelength_calibration\calibration.json"
# 1. 先只看找到了哪些峰(不导出图片,速度快)
python peak_finder.py --input $rec --select none
# 2. 导出全部峰(每峰一张峰形图 + CSV + 原始像素块)
python peak_finder.py --input $rec --select all
# 3. 只导出 540–560 nm 的峰和最强的两条(按波长选就必须给 --calibration)
python peak_finder.py --input "C:\Users\peng0\Dev\Light\recordings\hq_20261110_230441" `
--calibration $cal --select nm:540-560,top:2
# 4. 按编号导出(编号以上一次输出的 P001… 为准)
python peak_finder.py --input $rec --select id:2,id:4-5
# 5. 在窗口里点选:左边全谱,右边跟着鼠标显示该峰的 RAW 像素块和放大谱形
# 悬停预览、点击切换选/不选,a=全选,n=全不选,e=导出,q=取消
python peak_finder.py --input $rec --pick
# 6. 平均多帧降噪;只取 ROI 中的一段行,减小谱线倾斜造成的展宽
python peak_finder.py --input "C:\Users\peng0\Dev\Light\recordings\hq_20261111_092116" `
--frame 5 --average 8 --rows 600,760
# 7. 顺带对照汞灯谱线表(需要波长,所以也给标定)
python peak_finder.py --input $rec --calibration $cal --match line_tables\mercury_hg1.csv
```
只有第 3 条和第 7 条给了 `--calibration`;其余只输出像素坐标,要波长自己补上。
给了 `--calibration` 时终端会打印:
```
id nm x_px ch SNR height FWHM px FWHM nm slit y flags
* P001 435.6458 302.489 RB 517 2983 18.08 5.151 805.571 extrapolated, Hg I -0.19 nm
* P002 545.8237 689.157 RGB 2925 13545 10.35 2.949 763.262 saturated, Hg I -0.25 nm
P003 579.5502 807.519 RGB 939 4346 14.74 4.2 771.807 Hg I +0.48 nm
```
(这三条相对 NIST 汞灯 435.8328 / 546.0735 / 579.0663 nm 的偏差为 −0.19 / −0.25 / +0.48 nm,
属于该标定本身约 0.7 nm RMS 的水平。)
## 交互选峰窗口(`--pick`)
```
┌───────────────────────────────┬──────────────────────┐
│ │ RAW 预览(整幅、与 │
│ 全谱(选中/未选中/聚焦) │ 播放器同款)+ROI 框 │
│ │ +峰值所在列 │
│ ├──────────────────────┤
│ │ 放大谱形 + 拟合曲线 │
│ │ + 半高宽标记 │
└───────────────────────────────┴──────────────────────┘
```
鼠标移到某条谱线上,右边两块立刻切换成那个峰。右上就是 `hq_spectrum` 播放器里那张
auto-stretched RAW 预览(缩放到 ROI 区域):橙色框是 ROI,青色竖线是当前峰的传感器列,
青圈是它在缝方向的行位置;右下是放大的净信号和拟合曲线。
点一下切换选中(红色为选中、灰色为未选中、线更粗的是当前聚焦)。`a` 全选、`n` 全不选、
`e` 导出并关闭、`q` 取消。见 `pick_window.png`。
窗口用固定的三块布局(不依赖 constrained_layout),可以随便拉伸;如果窗口比屏幕还大,
程序会先按屏幕尺寸缩一档,避免开出来就有一半在屏幕外。
> 导出的单峰 PNG 里那小块仍然是**未去马赛克的 Bayer 像素块**(`inferno` 色阶),
> 用来看像素级的线形和倾斜;想看"谱线在相机画面里的位置",看这个选峰窗口或
> `peaks_overview.png`。
## 参数
### 输入
| 参数 | 默认 | 说明 |
| --- | --- | --- |
| `--input` | 必填 | 录制目录(含 `meta.json`、`frames.bin`) |
| `--frame` | `0` | 帧号(0 起) |
| `--average` | `1` | 从 `--frame` 起平均多少帧;>1 时导出的像素块是均值(float32) |
| `--roi` | `auto` | `auto` 自动检测光谱带;`x0,y0,x1,y1` 手动;`calibration` 用标定 JSON 里保存的 ROI |
| `--rows` | 无 | 在 ROI 内再限制行范围 `y0,y1`(全传感器行号),可减小谱线倾斜展宽 |
| `--reduce` | `mean` | 沿缝方向的行合并方式 `mean`/`median` |
| `--black` / `--full-scale` | 自动 | 覆盖黑电平/满量程(解码后 ADU) |
| `--calibration` | 无 | 要换算波长时显式给出的标定 JSON;不给则只输出像素坐标(`wavelength_nm` 留空) |
| `--wavelength-offset` | `0` | 给所给标定加一个固定波长偏移(nm),仅用于已确认的系统偏移;本身就是标定假设,慎用 |
| `--module-dir` | 脚本目录 | `hq_spectrum.py` 所在目录 |
| `--quiet` | 关 | 不打印过程信息 |
### 寻峰
| 参数 | 默认 | 说明 |
| --- | --- | --- |
| `--baseline-window` | `100` | 形态学开运算基线的窗口(传感器像素);`0` 表示不减基线 |
| `--peak-distance` | `4` | 同通道两峰最小间距(传感器像素,2 px = 一个采样点),再近就只留高的;调大用于合并"勉强分开"的混合峰 |
| `--peak-height` | `0.02` | 最小峰高,占该通道最大净信号的比值 |
| `--peak-prominence` | `0.01` | 最小地形突出度,占该通道最大净信号的比值 |
| `--min-snr` | `15` | 最小峰高,单位是该通道噪声估计 |
| `--max-peaks` | `40` | 每通道最多峰数 |
| `--group-tol` | `4` | 跨 R/G/B 合并成同一条谱线的距离(像素) |
| `--fwhm-span` | `60` | 半高宽搜索的最大半程(像素);超出记为无半高宽 |
| `--fit-window` | `0` | 拟合窗半宽(像素);0 自动按半高宽定 |
| `--shape` | `gauss` | 线型模型 `gauss`/`lorentz` |
### 选择与导出
| 参数 | 默认 | 说明 |
| --- | --- | --- |
| `--select` | `all` | `all`、`none`、`3`(编号)、`3-7`(编号段)、`nm:540-560`(波长段)、`546.1`(波长±典型线距的一半)、`top:5`(SNR 前 5),可用逗号组合取并集 |
| `--pick` | 关 | 开窗口点选:左边全谱,右边实时显示鼠标所指峰的 RAW 像素块与放大谱形;悬停预览、点击切换选中 |
| `--output` | `peak_exports/<录制名>_f<帧号>` | 输出目录;单帧是 `_f0003`,`--average 8` 时是 `_f0003-0010_avg8` |
| `--patch-half-width` | `24` | 导出原始像素块的横向半宽(像素) |
| `--patch-format` | `npy` | 像素块格式 `npy`/`csv`/`both`/`none` |
| `--patch-csv` | 关 | 在 npy 之外再写一份长表 CSV |
| `--no-images` | 关 | 不生成每峰的 PNG |
| `--match` | 无 | 谱线表(CSV/JSON),用于给出谱线身份建议 |
| `--match-tolerance` | `1.5` | 匹配容差(nm) |
## 导出内容
```
peak_exports/hq_20261101_114417_f0000/
├─ peaks_overview.png 全谱:上图原始信号+基线,下图归一化净信号+峰标注
├─ peaks_all.csv 所有通道的全部候选峰(每行一个通道峰)
├─ selected_peaks.csv 选中的峰,一行一条物理谱线(只包含像素坐标等汇总)
├─ peaks.json 清单:录制、帧、ROI、黑电平、噪声、标定、阈值、每峰的逐通道细节
└─ peaks/
├─ peak_P002.png 峰形图:全谱定位 + RAW 像素块 + 放大曲线与拟合
├─ peak_P002_shape.csv 峰形数据:逐通道 raw / 基线 / net / 拟合值 / 残差
└─ peak_P002_raw_y572-978_x664-714.npy 该峰的 RAW 像素块(uint16;文件名即像素框)
```
`peak_Pxxx.png` 三块内容:上方是该峰在全谱中的位置;左下是把该峰附近的 RAW Bayer
像素块(未去马赛克,横轴/纵轴都是真实传感器行列号);右下是放大后的净信号、拟合曲线
(虚线)、半高宽标记,图例里的 `x±…` 是拟合中心的误差。标题给出波长、传感器列、通道、
SNR、FWHM;饱和峰带 `[SATURATED]`,标定范围外带 `[extrapolated]`,重叠双峰经联合拟合
分开的带 `[pair fit]`。
### selected_peaks.csv 主要字段
| 字段 | 含义 |
| --- | --- |
| `id` | 峰编号 `P001`…(按传感器列排序,同一数据同参数可复现) |
| `wavelength_nm` | 波长(空气,nm);无标定时为空 |
| `extrapolated` | 1 = 在标定确认范围之外,属于外推 |
| `channels` / `primary_channel` | 该线出现在哪些 CFA 通道 / 用哪个通道出指标 |
| `sensor_x_px` | 峰中心的全传感器列坐标(亚像素,加权合并各通道拟合中心) |
| `center_spread_px` | 各通道中心的最大差,判断跨通道一致性 |
| `sample_index` / `sample_step_px` | 该峰在降维谱数组里的下标 / 采样步长(2 px) |
| `dispersion_nm_per_px` | 该处色散 |
| `height_net_adu`、`snr`、`fwhm_px`、`fwhm_nm`、`area_net_adu_px` | 峰高(扣基线)、信噪比、半高宽(像素与 nm)、净面积 |
| `baseline_adu`、`raw_adu`、`saturated` | 基线值、峰顶原始值、是否饱和 |
| `slit_y_px` | 该谱线在缝方向(行方向)的强度加权中心行 |
| `roi_x0..roi_y1` | 使用的 ROI(全传感器坐标,半开区间) |
| `patch_x0..patch_y1` | 导出的 RAW 像素块框(全传感器坐标,半开区间) |
| `frame`、`frames_averaged` | 帧号、平均帧数 |
| `image_png`、`shape_csv`、`raw_npy`、`raw_csv` | 对应文件(相对输出目录) |
| `match_species`、`match_nm`、`match_delta_nm` | 谱线表匹配结果(未启用时为空) |
`peaks_all.csv` 另外给出逐通道的 `center_parabola_px`、`center_fit_px`、
`center_err_px`(拟合中心的形式误差)、`fwhm_halfmax_px`(半高宽直接测量值,可与拟合值
对照)、`fit_ok`、`fit_rms_adu`、`deblended`,便于判断某个峰的拟合是否可信。实际采用的
中心由 `center_source` 标明:`fit` = 高斯/洛伦兹拟合中心,`apex` = 拟合被拒后用峰顶三点
抛物线,`sample` = 平顶/饱和导致抛物线也不可用时的采样点。重叠双峰的两个分量共用同一个
`fwhm_px`(见下),因此两者线宽相同是正常的。
## 坐标约定
* 全部为 **0 起** 的全分辨率传感器坐标;像素 `x` 覆盖 `[x, x+1)`,第 0 行在画面顶部。
* 降维谱每 2 px 一个采样点(CFA),拟合中心可以落在采样点之间,因此 `sensor_x_px` 是小数。
* `roi`、`patch` 都是半开区间 `[x0,x1) × [y0,y1)`,与 `hq_spectrum.py` 的 ROI 定义一致。
* `peak_*_raw_y572-978_x664-714.npy` 的第 0 行 = 传感器第 572 行,第 0 列 = 第 664 列
(文件名里就是行列范围),可以直接 `np.load` 后与 `patch_x0…patch_y1` 对上。
## 方法说明
1. 用 `hq_spectrum.py` 的 `RecordingSource` 解码一帧(或若干帧均值):PC1B 解压成
uint16 Bayer 图。
2. `extract_planes` 按 BGGR 拆成 B、Gb+Gr(平均)、R 三个平面,沿缝方向(行)取
mean/median,得到一个像素一个采样点的三条谱;x 用 `channel_x_positions` 换算成
全传感器列号(G 在 `x0+0.5+2k`,R/B 在 `x0+dx+2k`)。
3. 基线:对每条谱做形态学开运算(先最小值滤波再最大值滤波,窗口默认 ±50 采样点),
再去掉开运算结果的高频毛刺。它只用于**测量峰高/峰宽**,原始值单独导出。
4. 噪声:中位绝对一阶差分 / (0.6745·√2),对孤立尖峰不敏感。
5. 候选峰:局部极大值(平台取中点),要求峰高 ≥ max(高度比×通道最大, SNR 阈×噪声),
且**地形突出度**(topographic prominence,向上走到比它更高的点为止,取两侧谷底较高者)
达标,并与更强的峰间距 ≥ `--peak-distance`。用真实突出度而不是固定窗口,肩峰会被
自动排除,而真正分得开的双峰不会被并掉。
6. 精化:加线性背景的高斯/洛伦兹拟合(自实现的 Levenberg–Marquardt,带边界投影),
同时给出参数的形式误差。拟合不合格(不收敛、宽度顶到边界、残差 RMS > 20% 峰高)时
**退回峰顶**(三点抛物线,`center_source=apex`),平顶/饱和导致抛物线也不可用时再用
检测到的采样点。**全程不使用质心**:质心是窗口的函数而不是峰的函数,同一个峰换个窗口
就换个值(本数据集里实测偏过 6 px 和 102 px),不能当峰位置用。
7. **双峰去混叠**:若两条相邻谱线的间距小于约 1.2 倍平均半高宽,就用「两个峰 + 一个
公共线宽 + 线性背景」做联合拟合。单峰拟合在重叠区会把两个中心都往中间拉(实测汞
577/579 会合成一条 579.5 nm),联合拟合才能把它们分开;标 `deblended=1`、`[pair fit]`。
8. 跨通道合并:按**检测到的极大值位置**聚类(容差 `--group-tol`),同一通道只留 SNR
最高者;同一条线在不同通道被检测到位置差几像素时,按拟合中心再合并一次
(容差 1.5 px);峰中心按净峰高加权平均。
9. 波长(**仅当你给了 `--calibration`**):`wavelength_nm = np.polyval(coefficients_descending,
sensor_x_px)`,与 `wavelength_player.py` 用的是同一个公式;超出该标定确认范围的峰标
`extrapolated`。不给标定时这一列与 nm 相关的量都留空,图上的 x 轴就是传感器列。
## 已知限制
* **标定只对同一光学结构和传感器模式有效。** 标定来自 `hq_20261101_114417`,其确认范围
是 436–580 nm;其它录制的同一根谱线位置可能相差几个像素(几百分之一 nm 到 1 nm)。
需要更准就重新跑 `wavelength_calibration.py`。`--wavelength-offset` 只能在已确认是
固定系统偏移时使用,不能代替重新标定。
* **谱线沿行方向不是一条直线(弯曲 + 漂移)。** 实测汞 546.1 nm(`hq_20261101_114417`):
照亮带(约 363 行)上谱线中心是一条 S 形曲线,峰峰值 9.6 px(≈2.7 nm),按强度加权的
1σ 是 1.76 px,而**净线性漂移只有 2.6 px(0.41°)**。因此沿全部 970 行平均得到
FWHM 10.35 px(2.95 nm),只用最亮的 70 行(`--rows 690,760`)得到 7.37 px(2.10 nm),
中心相差 1.76 px(0.50 nm)。要测真实线宽就用 `--rows` 取窄行窗口。
导出的 RAW 像素块只有 ~50 px 宽却横跨几百行,`imshow` 的 `aspect="auto"` 会把横向
放大约 10 倍,看着比实际"斜"得多——上面的数字才是真实几何。
* **饱和峰**:核心像素达到 65535 时标 `[SATURATED]`,此时峰高、线宽和中心都可能偏。
降低曝光/增益或缩短曝光时间重新录制更可靠。
* **重叠双峰要小心。** 汞 576.96/579.07 nm 这类双线在默认 ROI 下间距只有约 6 px,
而单条线 FWHM 约 7.5 px。程序会用公共线宽做联合拟合把它们分成两条(本例
P004≈578.0 nm / P005≈580.4 nm),但分量位置的不确定度比孤立线大一个量级:
`center_err_px` 约 0.2–0.5 px,而孤立线只要 0.02–0.3 px。间距小于约 1 倍半高宽时
结果对线型假设很敏感,把它当"参考位置"而不是精密测量值。
* **弱线/连续谱**:`--peak-height`、`--peak-prominence`、`--min-snr` 是相对的,暗弱录制
里需要适当调低;连成一片的宽峰会只报一个中心,或被 `--peak-distance` 合并。想把
靠得很近的线分开就把 `--peak-distance` 调小(最小 2 px = 一个采样点)。
* 基线是形态学开运算的结果,宽于基线窗口的发射带会被部分当作基线;需要时可加大
`--baseline-window` 或设为 0。
* 波长单位是空气波长,未做光谱响应(强度)校正,强度仍是相机 ADU。
## 与其它脚本的关系
| 脚本 | 作用 |
| --- | --- |
| `hq_spectrum.py` | 解码/实时显示/回放的核心,本脚本复用它的解码、ROI、CFA 分离 |
| `hq_record.py` | 录制 RAW |
| `wavelength_calibration.py` | 用汞灯重新做像素→波长标定,产出 `wavelength_calibration/`;peak_finder 只读它的 JSON,从不改它 |
| `wavelength_player.py` | 带标定的回放 |
| `hq_average.py` | 多帧平均成新录制(本脚本用 `--average` 做同样的事,只是不落盘) |
| `peak_finder.py` | 本脚本:寻峰 + 峰形/像素坐标导出 |
本脚本**只读取**标定,不会修改 `wavelength_calibration/` 里的任何文件。
## 测试
```powershell
# 录制不在程序目录附近时,用环境变量指过去,否则端到端用例会自动跳过
$env:PEAK_FINDER_TEST_RECORDING = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417"
$env:PEAK_FINDER_TEST_CALIBRATION = "C:\Users\peng0\Dev\Light\Programs\wavelength_calibration\calibration.json"
python -m unittest test_peak_finder -v
```
覆盖基线/噪声估计、亚像素中心、高斯/洛伦兹拟合、候选筛选、跨通道合并、选择语法、
标定校验、谱线表匹配,以及一次真实的端到端导出(录制不在时会自动跳过)。
+86
View File
@@ -0,0 +1,86 @@
# spectrometer
Raspberry Pi HQ Camera(IMX477)RAW 光谱工具集:录制、回放、寻峰、波长标定。
采集的是相机 **RAW** 数据(`BGGR_PISP_COMP1`,2028×1520,16 bit 容器),不做 ISP、
不去马赛克、不做平滑;所有分析都基于按 CFA 分离后的 R/G/B 三条一维谱。
## 文件
| 文件 | 作用 |
| --- | --- |
| `hq_spectrum.py` | 核心库 + 实时播放器:PC1B 解码、CFA 分离、ROI、降维、预览、交互窗口 |
| `hq_record.py` | 把相机 RAW 流录成 `recordings/<时间戳>/{frames.bin,frames.jsonl,meta.json}` |
| `hq_average.py` | 多帧平均成一条新录制(降噪) |
| `spectro_core.py` | **共享核心**:模块加载、标定解析、基线/噪声/寻峰/拟合/双峰分解、CSV/JSON 输出 |
| `peak_finder.py` | 寻峰与峰信息导出:峰形图、峰形数据、像素坐标、RAW 像素块;`--pick` 交互选峰 |
| `wavelength_calibration.py` | 用汞灯谱线标定 像素→波长,产出 `wavelength_calibration/` |
| `wavelength_player.py` | 带已存标定的回放(波长轴、瀑布图) |
| `PEAK_FINDER.md` / `WAVELENGTH_PLAYER.md` | 两个工具的详细说明 |
| `test_peak_finder.py` / `test_wavelength_player.py` | 单元测试(含真实录制的端到端用例) |
| `wavelength_calibration/` | 一次汞灯标定的结果:`calibration.json` 及残差、候选、逐像素波长表 |
`anal.py` 是 2026-10-23 的 `hq_spectrum.py` 旧副本,函数一一对应、无独有内容,属于
遗留文件,可删。
## 依赖
```
pip install -r requirements.txt
```
- Python ≥ 3.10,**NumPy + Matplotlib**,不需要 SciPy。
- 树莓派上采集还需要 `picamera2`(系统包)和 `python3-tk`(交互窗口)。
- 在 PC 上分析已有录制只需要 NumPy + Matplotlib。
## 快速开始
录制(树莓派):
```bash
python3 hq_record.py --output recordings --exposure 66351 --gain 16 --duration 10
```
回放 / 看谱(PC 或树莓派):
```bash
python3 hq_spectrum.py --input recordings/hq_20261101_114417 --fps 15
python3 wavelength_player.py --input recordings/hq_20261101_114417 \
--calibration wavelength_calibration/calibration.json
```
寻峰并导出(`--calibration` 只是**应用**已有的标定,不给就只输出像素坐标):
```bash
python peak_finder.py --input recordings/hq_20261101_114417 \
--calibration wavelength_calibration/calibration.json --select all
python peak_finder.py --input recordings/hq_20261101_114417 --pick
```
重新标定:
```bash
python wavelength_calibration.py --input recordings/hq_20261101_114417 --headless
# 看清 candidates.png 后指定:候选编号:波长
python wavelength_calibration.py --input recordings/hq_20261101_114417 --headless \
--assign "B1:404.6563,B2:435.8328,G1:546.0735,G2:576.9598,G3:579.0663"
```
## 测试
```bash
python -m unittest test_peak_finder test_wavelength_player -v
```
端到端用例需要录制与标定;录制不在脚本附近时用环境变量指过去,否则自动跳过:
```powershell
$env:PEAK_FINDER_TEST_RECORDING = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417"
$env:PEAK_FINDER_TEST_CALIBRATION = "wavelength_calibration\calibration.json"
```
## 仓库里不放什么
`recordings/`(原始录制,单个约 470 MB)、`peak_exports/`(导出结果)、`__pycache__/`
都在 `.gitignore` 里。标定结果 `wavelength_calibration/` 是**放**进来的,因为它决定了
所有波长读数。
+29
View File
@@ -0,0 +1,29 @@
# 使用标定结果播放录制
将 `wavelength_player.py` 放在 `hq_spectrum.py` 同目录,使用原来的 Python/NumPy/Matplotlib 环境。
原播放器不需要修改。默认标定路径是新脚本旁 `wavelength_calibration/calibration.json`。
在 Programs 目录运行,把下面的 `other_recording` 换成实际录制目录:
```powershell
python wavelength_player.py --input recordings/other_recording --calibration wavelength_calibration/calibration.json
```
默认暂停,沿用原播放器的播放/暂停、进度条、逐帧、倍速、循环、ROI 框选、`s` 保存和 `q` 退出。加 `--autoplay` 立即播放。默认在目标录制中自动检测 ROI;也可使用 `--roi calibration` 复用标定 ROI,或 `--roi 400,600,1400,900` 指定新 ROI。
```powershell
# 用已有汞灯录制检查标定回放
python wavelength_player.py --input recordings/hq_20261024_135709 --roi calibration
# 无窗口检查前两帧并导出最后一帧截图及光谱 CSV
python wavelength_player.py --input recordings/other_recording --headless --frames 2 --save-prefix replay_check
# 全部参数
python wavelength_player.py --help
```
直接使用 JSON 中的降幂系数,不重新拟合。R/G/B 各自完整传感器列坐标转换为 nm,ROI 改动不会把像素坐标重新从零开始。线性、二次及波长递减方向均支持;瀑布图显示时重采样到等距波长网格,导出的 R/G/B 光谱仍保留各自原始采样坐标。绿色背景表示确认点覆盖的波长范围,范围外为外推。
默认不平滑、不降噪、不扣背景,保留包括负值的黑电平校正信号。各帧采用录制自己的黑电平元数据,不沿用标定灯录制的 4096;可用 `--black` 手动覆盖(单位为解码后的 ADU)。归一化副本仅用于寻峰和辅助显示。导出 CSV 沿用原播放器字段,其中 `signal_adu` 和 `signal_smoothed` 在本程序中相同,均未经平滑。强度单位仍为相机 ADU,未做光谱响应校正。
标定要求光学结构、传感器模式及坐标几何保持一致。若 JSON 指向的原标定录制仍存在,会比较图像宽高,不一致即拒绝;若原录制已移走,会提示无法核对尺寸。相同宽高也不能保证光学位置未改变。
测试:`python -m unittest test_wavelength_player -v`。
+1507
View File
File diff suppressed because it is too large. Load diff
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""Average hq_record recording directories using the existing player's decoder."""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
import sys
import time
import uuid
import numpy as np
try: # shared core lives beside this script
import spectro_core as core
except ImportError as exc: # pragma: no cover - placement error
raise SystemExit('hq_average.py needs spectro_core.py in the same directory '
'(or on sys.path)') from exc
def load_player(source, module_dir=None):
"""Decoder module: --module-dir, then this folder, then the project above."""
for directory in ([Path(module_dir)] if module_dir
else [Path(__file__).resolve().parent, Path(source).parent.parent]):
if (directory / 'hq_spectrum.py').is_file():
return core.load_hq(directory)
return core.load_hq()
def write_json(path, value):
core.write_json(path, value)
def average(source, output, group=0, drop_last=False, module_dir=None):
source, output = Path(source).resolve(), Path(output).resolve()
if group < 0:
raise ValueError('group must be >= 0')
if source == output or source in output.parents or output in source.parents:
raise ValueError('Input and output must be separate, non-nested directories')
if output.exists():
raise ValueError(f'Output already exists: {output}')
meta = json.loads((source / 'meta.json').read_text(encoding='utf-8'))
if meta.get('kind') != 'hq_raw_recording' or meta.get('format_version') != 1:
raise ValueError('Expected hq_raw_recording format_version 1')
w, h, stride = (int(meta[k]) for k in ('width', 'height', 'stride'))
fb = int(meta['frame_bytes'])
if min(w, h, stride) <= 0 or fb != h * stride:
raise ValueError('Invalid dimensions/stride/frame_bytes')
size = (source / 'frames.bin').stat().st_size
count, remainder = divmod(size, fb)
if remainder or not count or int(meta.get('frames', count)) != count:
raise ValueError('Truncated recording or frame count mismatch; repair source separately')
if not meta.get('completed'):
raise ValueError('Source recording is not marked completed')
records = []
metadata_path = source / 'frames.jsonl'
if metadata_path.exists():
for i, line in enumerate(metadata_path.read_text(encoding='utf-8').splitlines()):
row = json.loads(line)
if not isinstance(row, dict):
raise ValueError(f'Invalid metadata row {i}')
nested = row.pop('metadata', None)
if isinstance(nested, dict):
row = {**nested, **row}
for key, expected in [('i', i), ('offset', i * fb), ('bytes', fb)]:
if key in row and row[key] != expected:
raise ValueError(f'Incorrect metadata {key} in row {i}')
records.append(row)
if len(records) != count:
raise ValueError('frames.jsonl count mismatch')
else:
records = [{} for _ in range(count)]
# Exposure/gain changes are retained as ranges in each averaged frame's provenance.
player = load_player(source, module_dir)
pattern = str(meta.get('pattern') or player.pattern_from_format(meta['raw_format']))
if pattern not in ('BGGR', 'RGGB', 'GBRG', 'GRBG'):
raise ValueError('Unsupported Bayer pattern')
config = {'size': (w, h), 'stride': stride}
n = group or count
groups = [(a, min(a + n, count)) for a in range(0, count, n)
if not (drop_last and min(a + n, count) - a < n)]
if not groups:
raise ValueError('No output frames; reduce group size or omit --drop-last')
output.parent.mkdir(parents=True, exist_ok=True)
temp = output.parent / (output.name + '.partial-' + uuid.uuid4().hex[:8])
temp.mkdir()
started = time.monotonic()
try:
depth = None
out_fb = w * h * 2
with (source / 'frames.bin').open('rb') as inp, (temp / 'frames.bin').open('wb') as raw_out, (temp / 'frames.jsonl').open('w', encoding='utf-8') as mout:
for oi, (a, b) in enumerate(groups):
acc = np.zeros((h, w), dtype=np.float64)
inp.seek(a * fb)
for index in range(a, b):
data = inp.read(fb)
if len(data) != fb:
raise ValueError('Input changed or was truncated while reading')
image, bits = player.decode_raw(np.frombuffer(data, dtype=np.uint8).reshape(h, stride), config, meta['raw_format'])
if depth is not None and depth != bits:
raise ValueError('Decoded bit depth changed')
depth = bits
acc += image
image = np.clip(np.rint(acc / (b - a)), 0, (1 << depth) - 1).astype('<u2')
raw_out.write(image.tobytes())
row = dict(records[a])
for key in ('_frame_index', '_frame_total'):
row.pop(key, None)
ranges = {}
for key in ('ExposureTime', 'AnalogueGain', 'DigitalGain'):
values = [r[key] for r in records[a:b] if isinstance(r.get(key), (float, int))]
if values:
ranges[key] = [min(values), max(values)]
row.update(i=oi, offset=oi * out_fb, bytes=out_fb,
averaging={'start_index': a, 'end_index_exclusive': b, 'count': b-a,
'metadata_policy': 'representative first source frame',
'last_source_metadata': records[b-1], 'parameter_ranges': ranges})
mout.write(json.dumps(row, ensure_ascii=False) + '\n')
print(f'Output {oi+1}/{len(groups)}: source [{a}, {b})', flush=True)
newmeta = dict(meta)
fps = float(meta.get('average_capture_fps') or meta.get('fps_requested') or 15) / n
newmeta.update(raw_format=f'S{pattern}{depth}', bit_depth=depth, stride=w*2,
frame_bytes=out_fb, storage_dtype='uint8', frames=len(groups),
completed=True, frames_file='frames.bin', metadata_file='frames.jsonl',
average_capture_fps=fps, fps_requested=fps,
duration_s=len(groups)/fps, frame_limit=len(groups), duration_limit_s=0,
averaging={'source_directory': str(source), 'source_meta': meta,
'mode': 'all' if not group else 'group', 'group_size': n,
'drop_last': drop_last, 'discarded_frames': count-groups[-1][1],
'rounding': 'nearest, ties to even', 'accumulator': 'float64',
'timing': 'nominal group cadence; last partial group may be shorter'})
write_json(temp / 'meta.json', newmeta)
# Keep every original per-frame field, including within-group variations.
if metadata_path.exists():
import shutil
shutil.copyfile(metadata_path, temp / 'source_frames.jsonl')
os.rename(temp, output)
print(f'Saved: {output} ({time.monotonic()-started:.1f}s)')
return output
except BaseException:
print(f'Incomplete output kept for inspection: {temp}', file=sys.stderr)
raise
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--input', type=Path, help='Recording directory; omit for folder picker')
p.add_argument('--output', type=Path, help='NEW output directory')
m = p.add_mutually_exclusive_group()
m.add_argument('--all', action='store_true', help='Average entire recording (default)')
m.add_argument('--group', type=int, default=0, help='Average each N consecutive frames')
p.add_argument('--drop-last', action='store_true', help='Discard final incomplete group')
p.add_argument('--module-dir', type=Path, help='Directory containing original hq_spectrum.py')
args = p.parse_args()
if args.input is None:
import tkinter as tk
from tkinter import filedialog
root = tk.Tk()
root.withdraw()
selected = filedialog.askdirectory(title='Select recording folder')
root.destroy()
if not selected:
return 0
args.input = Path(selected)
suffix = '_avg_all' if not args.group else f'_avg{args.group}'
output = args.output or args.input.resolve().with_name(args.input.name + suffix)
try:
average(args.input, output, args.group, args.drop_last, args.module_dir)
return 0
except (Exception, KeyboardInterrupt) as exc:
print(f'Error: {exc}', file=sys.stderr)
return 1
if __name__ == '__main__':
raise SystemExit(main())
+396
View File
@@ -0,0 +1,396 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
hq_record.py
============
Fast raw recorder for the Raspberry Pi HQ Camera.
This is the "record" half of the split workflow:
hq_record.py -> recordings/hq_YYYYmmdd_HHMMSS/
hq_spectrum.py --input REC_DIR -> replay / analysis
The recorder does not import Matplotlib and does not compute spectra. It only
pulls the Picamera2 RAW stream, queues each raw byte buffer, and writes it to
`frames.bin`. A background writer thread keeps disk I/O away from the camera
capture loop.
Recording layout
----------------
REC_DIR/
meta.json camera/format/stride/frame count
frames.bin raw byte buffers, one after another, no decoding
frames.jsonl one line of camera metadata per frame
For the usual Pi 5 HQ Camera configuration the raw stream is
`BGGR_PISP_COMP1` (PC1B). Each frame is height*stride bytes; at 2028x1520
that is typically about 3.1 MB/frame. At 15 fps that is about 47 MB/s, so a
USB SSD/NVMe is preferable to a slow SD card.
Examples
--------
# list modes first if needed
python3 hq_record.py --list-modes
# 10 seconds, 15 fps, manual exposure copied from rpicam-hello
python3 hq_record.py --duration 10 --fps 15 \
--exposure 33013 --gain 16 \
--record-dir recordings/hg_lamp
# unlimited until Ctrl+C
python3 hq_record.py --duration 0 --fps 15 \
--record-dir recordings/hg_long
# analyze later (on the Pi or on a PC)
python3 hq_spectrum.py --input recordings/hg_lamp --fps 15
The recorder is deliberately independent of the analyzer: a slow Matplotlib
redraw can no longer cause dropped frames or change the exposure timing.
"""
from __future__ import annotations
import argparse
import json
import queue
import shutil
import sys
import threading
import time
from datetime import datetime
from pathlib import Path
from types import SimpleNamespace
from typing import Optional
import numpy as np
try:
from hq_spectrum import CameraSource, list_camera_modes
except ImportError as exc: # pragma: no cover - import error path
raise SystemExit(
"hq_record.py must sit next to hq_spectrum.py "
"(run it from the hq_spectrum directory)."
) from exc
class RawWriter(threading.Thread):
"""Write queued raw buffers and metadata lines in the background."""
def __init__(self, frames_path: Path, metadata_path: Path, frame_bytes: int,
queue_size: int):
super().__init__(name="hq-raw-writer", daemon=False)
self.frames_path = frames_path
self.metadata_path = metadata_path
self.frame_bytes = frame_bytes
self.queue: queue.Queue = queue.Queue(maxsize=max(2, queue_size))
self.frames_written = 0
self.error: Optional[BaseException] = None
self.finished = threading.Event()
def put(self, raw: np.ndarray, metadata: dict, host_time_ns: int) -> None:
if self.error is not None:
raise RuntimeError(f"writer failed: {self.error}")
raw = np.ascontiguousarray(raw)
if raw.dtype != np.uint8 or raw.nbytes != self.frame_bytes:
raise ValueError(
f"unexpected raw buffer {raw.shape} {raw.dtype} {raw.nbytes}B; "
f"expected {self.frame_bytes}B uint8"
)
item = (raw, dict(metadata), int(host_time_ns))
while True:
try:
self.queue.put(item, timeout=0.5)
return
except queue.Full:
if self.error is not None:
raise RuntimeError(f"writer failed: {self.error}")
def run(self) -> None:
try:
with open(self.frames_path, "wb", buffering=4 * 1024 * 1024) as frames, \
open(self.metadata_path, "w", encoding="utf-8", buffering=1024 * 1024) as meta_out:
while True:
item = self.queue.get()
if item is None:
break
raw, camera_meta, host_time_ns = item
offset = self.frames_written * self.frame_bytes
record = dict(camera_meta)
record.update({
"i": self.frames_written,
"offset": offset,
"bytes": raw.nbytes,
"host_time_ns": host_time_ns,
})
frames.write(memoryview(raw).cast("B"))
meta_out.write(json.dumps(record, default=str, separators=(",", ":")) + "\n")
self.frames_written += 1
if self.frames_written % 30 == 0:
frames.flush()
meta_out.flush()
frames.flush()
meta_out.flush()
self.finished.set()
except BaseException as exc: # noqa: BLE001 - report writer failure to main
self.error = exc
self.finished.set()
def stop(self) -> None:
if self.is_alive():
try:
self.queue.put(None, timeout=2.0)
except queue.Full:
pass
self.join(timeout=30.0)
if self.is_alive():
raise RuntimeError("writer thread did not stop within 30 seconds")
if self.error is not None:
raise RuntimeError(f"writer failed: {self.error}")
def parse_args(argv: Optional[list[str]] = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Record Raspberry Pi HQ Camera RAW frames for later spectrum analysis.",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
parser.add_argument("--width", type=int, default=2028, help="raw stream width")
parser.add_argument("--height", type=int, default=1520, help="raw stream height")
parser.add_argument("--fps", type=float, default=15.0, help="requested frame rate")
parser.add_argument("--duration", type=float, default=10.0,
help="seconds to record; 0 means until Ctrl+C")
parser.add_argument("--frames", type=int, default=0,
help="optional frame limit; 0 means no limit")
parser.add_argument("--record-dir", default=None,
help="output directory (default recordings/hq_YYYYmmdd_HHMMSS)")
parser.add_argument("--queue-size", type=int, default=8,
help="raw frames buffered in RAM while the writer catches up")
parser.add_argument("--exposure", type=float, default=None,
help="manual exposure time in microseconds")
parser.add_argument("--gain", type=float, default=None, help="manual analogue gain")
parser.add_argument("--raw-format", default=None,
help="optional raw format, e.g. SBGGR12 or BGGR_PISP_COMP1")
parser.add_argument("--list-modes", action="store_true",
help="list camera raw modes and exit")
args = parser.parse_args(argv)
if args.width < 64 or args.height < 64:
parser.error("width and height must be at least 64")
if args.fps <= 0:
parser.error("--fps must be positive")
if args.duration < 0:
parser.error("--duration must be >= 0")
if args.frames < 0:
parser.error("--frames must be >= 0")
if args.queue_size < 2:
parser.error("--queue-size must be >= 2")
return args
def write_json(path: Path, data: dict) -> None:
tmp = path.with_suffix(path.suffix + ".tmp")
with open(tmp, "w", encoding="utf-8") as handle:
json.dump(data, handle, indent=2, default=str)
handle.write("\n")
tmp.replace(path)
def prepare_directory(path: Path) -> None:
if path.exists():
if not path.is_dir():
raise RuntimeError(f"{path} exists and is not a directory")
if any(path.iterdir()):
raise RuntimeError(
f"{path} is not empty; choose a new --record-dir "
f"(the recorder never overwrites old data)"
)
else:
path.mkdir(parents=True, exist_ok=True)
def main(argv: Optional[list[str]] = None) -> int:
args = parse_args(argv)
if args.list_modes:
list_camera_modes()
return 0
cam_args = SimpleNamespace(
width=args.width,
height=args.height,
fps=args.fps,
exposure=args.exposure,
gain=args.gain,
raw_format=args.raw_format,
)
source = None
writer = None
meta = None
record_dir = None
enqueued = 0
started = None
exit_code = 0
try:
source = CameraSource(cam_args)
print(f"camera : {source.description}")
print(f"raw stream : {source.raw_format} "
f"{source.full_size[0]}x{source.full_size[1]}")
print(f"stride : {source.stride}")
print(f"Bayer : {source.pattern} decoded {source.bit_depth}-bit buffer")
print(f"target rate : {args.fps:g} fps")
if "PISP_COMP" in source.raw_format.upper():
print("note : recording the compressed PC1B buffer; no decode on the Pi")
source.start()
last_meta = {}
# Discard warm-up frames; the recording should start with settled AE.
for _ in range(5):
_, last_meta = source.read_raw()
# Get one sample only to validate the actual buffer size.
sample_raw, sample_meta = source.read_raw()
last_meta = sample_meta
if sample_raw.dtype != np.uint8:
raise RuntimeError(f"unexpected raw dtype {sample_raw.dtype}; expected uint8 bytes")
frame_bytes = int(sample_raw.nbytes)
expected_bytes = source.stride * source.full_size[1]
if frame_bytes != expected_bytes:
raise RuntimeError(
f"raw buffer is {frame_bytes} B but stride*height is {expected_bytes} B; "
f"refusing to write a recording that the analyzer would misread"
)
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
record_dir = Path(args.record_dir).expanduser() if args.record_dir else \
Path("recordings") / f"hq_{stamp}"
prepare_directory(record_dir)
frames_path = record_dir / "frames.bin"
metadata_path = record_dir / "frames.jsonl"
meta_path = record_dir / "meta.json"
data_rate = frame_bytes * args.fps
free = shutil.disk_usage(record_dir).free
print(f"output : {record_dir}")
print(f"frame size : {frame_bytes / 1e6:.3f} MB")
print(f"data rate : {data_rate / 1e6:.1f} MB/s")
print(f"free space : {free / 1e9:.2f} GB")
planned_frames = None
if args.frames:
planned_frames = args.frames
if args.duration:
duration_frames = int(args.duration * args.fps)
planned_frames = duration_frames if planned_frames is None else min(planned_frames, duration_frames)
if planned_frames is not None:
needed = frame_bytes * planned_frames * 1.05
print(f"planned size : {needed / 1e9:.2f} GB for ~{planned_frames} frames")
if needed > free:
raise RuntimeError(
f"not enough free space: need ~{needed / 1e9:.2f} GB, "
f"have {free / 1e9:.2f} GB"
)
meta = {
"format_version": 1,
"kind": "hq_raw_recording",
"created_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"camera": source.description,
"raw_format": source.raw_format,
"width": source.full_size[0],
"height": source.full_size[1],
"stride": source.stride,
"frame_bytes": frame_bytes,
"storage_dtype": "uint8",
"pattern": source.pattern,
"bit_depth": source.bit_depth,
"fps_requested": args.fps,
"exposure_us": args.exposure,
"gain": args.gain,
"black_level": last_meta.get("BlackLevel"),
"duration_limit_s": args.duration,
"frame_limit": args.frames,
"frames": 0,
"completed": False,
}
write_json(meta_path, meta)
writer = RawWriter(frames_path, metadata_path, frame_bytes, args.queue_size)
writer.start()
# The first sample was taken to validate the format, so it is also the
# first frame of the recording.
writer.put(sample_raw, last_meta, time.time_ns())
enqueued = 1
started = time.monotonic()
last_report = started
while True:
if args.frames and enqueued >= args.frames:
break
elapsed = time.monotonic() - started
if args.duration and elapsed >= args.duration:
break
raw, camera_meta = source.read_raw()
writer.put(raw, camera_meta, time.time_ns())
enqueued += 1
now = time.monotonic()
if now - last_report >= 1.0:
written = writer.frames_written
rate = enqueued / max(now - started, 1e-6)
print(f"recording : {enqueued} frames captured, {written} written, "
f"{rate:.1f} fps")
last_report = now
elapsed = time.monotonic() - started
writer.stop()
writer_error = writer.error
writer = None
if writer_error is not None:
raise RuntimeError(f"writer failed: {writer_error}")
meta.update({
"frames": int(enqueued),
"duration_s": round(elapsed, 3),
"average_capture_fps": round(enqueued / max(elapsed, 1e-6), 3),
"completed": True,
"ended_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"frames_file": "frames.bin",
"metadata_file": "frames.jsonl",
})
write_json(meta_path, meta)
size = frames_path.stat().st_size
print(f"done : {enqueued} frames, {size / 1e9:.2f} GB, "
f"{enqueued / max(elapsed, 1e-6):.1f} fps")
print(f"analyze with : python3 hq_spectrum.py --input \"{record_dir}\"")
return 0
except KeyboardInterrupt:
print("\ninterrupted; finalizing recording", file=sys.stderr)
exit_code = 130
return exit_code
except Exception as exc:
print(f"error: {exc}", file=sys.stderr)
exit_code = 1
return exit_code
finally:
if source is not None:
try:
source.close()
except Exception:
pass
if writer is not None:
try:
writer.stop()
except Exception as exc:
print(f"writer shutdown error: {exc}", file=sys.stderr)
exit_code = 1
if meta is not None and record_dir is not None:
try:
meta.update({
"frames": int(writer.frames_written) if writer is not None else int(enqueued),
"completed": exit_code == 0 and meta.get("completed", False),
"ended_at": datetime.now().astimezone().isoformat(timespec="seconds"),
})
write_json(record_dir / "meta.json", meta)
except Exception as exc:
print(f"warning: could not finalize meta.json: {exc}", file=sys.stderr)
if __name__ == "__main__":
raise SystemExit(main())
+2409
View File
File diff suppressed because it is too large. Load diff
+6
View File
@@ -0,0 +1,6 @@
wavelength_nm,species,label
404.6563,Hg I,Hg 405
435.8328,Hg I,Hg 436
546.0735,Hg I,Hg 546
576.9598,Hg I,Hg 577 (unresolved with 579 in this instrument)
579.0663,Hg I,Hg 579
1 wavelength_nm species label
2 404.6563 Hg I Hg 405
3 435.8328 Hg I Hg 436
4 546.0735 Hg I Hg 546
5 576.9598 Hg I Hg 577 (unresolved with 579 in this instrument)
6 579.0663 Hg I Hg 579
+1411
View File
File diff suppressed because it is too large. Load diff
+6
View File
@@ -0,0 +1,6 @@
# 分析端(PC)依赖:读录制、寻峰、画图、标定
numpy>=1.24
matplotlib>=3.6
# 树莓派采集端还需要(系统包,不走 pip;按需安装):
# sudo apt install python3-picamera2 python3-numpy python3-matplotlib python3-tk
+860
View File
@@ -0,0 +1,860 @@
#!/usr/bin/env python3
"""Shared core for the spectro tools: loading, calibration, peak detection, output.
Extracted from peak_finder.py so that wavelength_calibration.py, wavelength_player.py
and hq_average.py can call the same code instead of keeping their own copies.
No command line and no matplotlib at import time; plotting helpers stay in the
scripts that need them. Everything here is pure NumPy.
"""
from __future__ import annotations
import csv
import importlib.util
import json
import math
import sys
from pathlib import Path
from typing import Optional
import numpy as np
CORE_DIR = Path(__file__).resolve().parent
GAUSS_TO_FWHM = 2.3548200450309493
def load_hq(module_dir: Optional[str] = None) -> object:
"""Import hq_spectrum.py from this folder (or ``module_dir``) by path."""
candidates = [Path(module_dir).expanduser()] if module_dir else [CORE_DIR, Path.cwd()]
for directory in candidates:
path = directory / "hq_spectrum.py"
if path.is_file():
spec = importlib.util.spec_from_file_location("hq_spectrum_shared", path)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
try: # last resort: a normal import from sys.path
import hq_spectrum as module # type: ignore
except ImportError as exc:
raise RuntimeError("cannot find hq_spectrum.py; put it beside spectro_core.py "
"or use --module-dir") from exc
return module
def is_saturated(image: np.ndarray, roi, x_center: float, full_scale: float,
band: tuple[int, int] = (-2, 4), threshold: float = 0.01) -> bool:
"""True when a meaningful fraction of the pixels near ``x_center`` is clipped."""
x0 = max(0, min(int(x_center) + band[0], image.shape[1]))
x1 = max(0, min(int(x_center) + band[1], image.shape[1]))
block = image[roi.y0:roi.y1, x0:x1]
if block.size == 0:
return False
return float(np.mean(block >= 0.995 * full_scale)) > threshold
def _num(value, digits: int = 6) -> str:
"""Format a value for CSV output; non-finite values become empty cells."""
if value is None:
return ""
if isinstance(value, (bool, np.bool_)):
return "1" if value else "0"
if isinstance(value, str):
return value
try:
number = float(value)
except (TypeError, ValueError):
return str(value)
if not np.isfinite(number):
return ""
return f"{number:.{digits}g}"
def write_csv(path: Path, fields: list[str], rows) -> None:
with path.open("w", newline="", encoding="utf-8") as handle:
writer = csv.writer(handle)
writer.writerow(fields)
for row in rows:
writer.writerow(row)
def _jsonable(value):
"""Convert NumPy scalars/arrays and Paths into plain JSON values."""
if isinstance(value, dict):
return {str(k): _jsonable(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [_jsonable(v) for v in value]
if isinstance(value, np.ndarray):
return [_jsonable(v) for v in value.tolist()]
if isinstance(value, (np.floating, np.integer, np.bool_)):
return value.item()
if isinstance(value, Path):
return str(value)
return value
def write_json(path, value) -> None:
"""Write JSON with NumPy/Path values converted and a trailing newline."""
Path(path).write_text(json.dumps(_jsonable(value), indent=2) + "\n", encoding="utf-8")
class Calibration:
"""Polynomial sensor-pixel -> wavelength saved by wavelength_calibration.py."""
def __init__(self, path, offset_nm: float = 0.0):
self.path = Path(path).expanduser().resolve()
try:
self.data = json.loads(self.path.read_text(encoding="utf-8-sig"))
except OSError as exc:
raise ValueError(f"cannot read calibration {self.path}: {exc}") from exc
except json.JSONDecodeError as exc:
raise ValueError(f"calibration {self.path} is not valid JSON: {exc}") from exc
if not isinstance(self.data, dict):
raise ValueError("calibration JSON must contain an object")
coeff = np.asarray(self.data.get("coefficients_descending"), dtype=float)
if coeff.ndim != 1 or len(coeff) not in (2, 3):
raise ValueError("calibration needs 2 or 3 descending coefficients")
if not np.all(np.isfinite(coeff)) or coeff[0] == 0:
raise ValueError("calibration coefficients must be finite and the leading one nonzero")
if self.data.get("order", len(coeff) - 1) != len(coeff) - 1:
raise ValueError("calibration order does not match coefficient count")
self.coeff = coeff
span = np.asarray(self.data.get("calibrated_pixel_range"), dtype=float)
if span.shape != (2,) or not np.all(np.isfinite(span)) or span[0] >= span[1]:
raise ValueError("calibration has an invalid calibrated_pixel_range")
self.span = span
self.offset_nm = float(offset_nm)
if not np.isfinite(self.offset_nm):
raise ValueError("wavelength offset must be finite")
self.width: Optional[int] = None
self._grid_px = None
self._grid_nm = None
def __call__(self, x):
return np.polyval(self.coeff, np.asarray(x, dtype=float)) + self.offset_nm
def dispersion(self, x) -> np.ndarray:
"""Local nm per sensor pixel (never zero over a validated sensor)."""
return np.polyval(np.polyder(self.coeff), np.asarray(x, dtype=float))
def set_width(self, width: int) -> None:
self.width = int(width)
grid = np.arange(self.width, dtype=float)
values = np.asarray(self(grid), dtype=float)
order = np.argsort(values)
self._grid_px = grid[order]
self._grid_nm = values[order]
def inverse(self, wavelength):
"""Sensor pixel of a wavelength; used for the secondary plot axis only."""
if self._grid_nm is None:
raise ValueError("calibration width is unknown")
return np.interp(np.asarray(wavelength, dtype=float), self._grid_nm, self._grid_px)
@property
def nm_span(self) -> tuple[float, float]:
values = sorted(np.asarray(self(self.span), dtype=float).tolist())
return float(values[0]), float(values[1])
def describe(self) -> str:
low, high = self.nm_span
text = (f"JSON calibration ({self.data.get('wavelength_medium', 'air')}); "
f"confirmed {low:.2f}-{high:.2f} nm; outside = extrapolation")
if self.offset_nm:
text += f"; manual offset {self.offset_nm:+.3f} nm"
return text
def validate_geometry(self, source, tell=None) -> None:
"""Refuse a recording whose sensor geometry differs from the calibration."""
tell = tell or (lambda *args, **kwargs: None)
derivative = self.dispersion([0.0, float(source.full_size[0] - 1)])
if np.any(np.abs(derivative) < 1e-12) or derivative[0] * derivative[1] <= 0:
raise ValueError("calibration is not monotonic over this recording width")
reference = self.data.get("recording")
metadata = Path(reference) / "meta.json" if reference else None
if metadata is not None and metadata.is_file():
reference_meta = json.loads(metadata.read_text(encoding="utf-8-sig"))
size = (int(reference_meta["width"]), int(reference_meta["height"]))
if size != tuple(source.full_size):
raise ValueError("recording dimensions differ from the calibration recording; "
"recalibrate this sensor mode")
else:
tell("note : calibration reference recording unavailable; sensor size not checked")
if self.data.get("pattern") and self.data["pattern"] != source.pattern:
tell(f"note : calibration pattern {self.data['pattern']} differs from "
f"recording pattern {source.pattern}")
def _rank_filter(y: np.ndarray, width: int, reducer) -> np.ndarray:
"""Running min/max over the window [i - width, i + width] with edge padding."""
width = max(0, int(width))
values = np.asarray(y, dtype=float)
if width == 0 or values.size <= 1:
return values.copy()
n = values.size
padded = np.pad(values, (width, width), mode="edge")
out = padded[width:width + n].copy()
for shift in range(-width, width + 1):
start = width + shift
reducer(out, padded[start:start + n], out=out)
return out
def _min_filter(y: np.ndarray, width: int) -> np.ndarray:
return _rank_filter(y, width, np.minimum)
def _max_filter(y: np.ndarray, width: int) -> np.ndarray:
return _rank_filter(y, width, np.maximum)
def estimate_baseline(y: np.ndarray, window_samples: int, smooth) -> np.ndarray:
"""Morphological opening baseline; removes peaks narrower than the window."""
width = max(0, int(window_samples))
if width == 0 or np.size(y) <= 2:
return np.zeros_like(np.asarray(y, dtype=float))
opened = _max_filter(_min_filter(np.asarray(y, dtype=float), width), width)
return smooth(opened.astype(np.float32), max(1, width // 2)).astype(float)
def estimate_noise(net: np.ndarray) -> float:
"""Robust noise proxy from the median absolute first difference."""
if np.size(net) < 3:
return 0.0
diff = np.diff(np.asarray(net, dtype=float))
mad = float(np.median(np.abs(diff - np.median(diff))))
return mad / (0.6745 * math.sqrt(2.0))
def parabola_center(x: np.ndarray, y: np.ndarray, index: int) -> float:
"""Sub-sample vertex of the parabola through three samples (NaN if unusable)."""
if index <= 0 or index >= y.size - 1:
return float("nan")
y0, y1, y2 = float(y[index - 1]), float(y[index]), float(y[index + 1])
denominator = y0 - 2.0 * y1 + y2
if not np.isfinite(denominator) or denominator == 0.0:
return float("nan")
delta = 0.5 * (y0 - y2) / denominator
if not np.isfinite(delta) or abs(delta) > 1.0:
return float("nan")
step = float(x[index + 1] - x[index])
return float(x[index] + delta * step)
def half_max_crossings(x: np.ndarray, net: np.ndarray, index: int,
max_span: int) -> tuple[float, float]:
"""Linearly interpolated x positions where the net signal falls to half max."""
peak = float(net[index])
if not np.isfinite(peak) or peak <= 0.0:
return float("nan"), float("nan")
half = 0.5 * peak
span = max(1, int(max_span))
left = index
floor_index = max(0, index - span)
while left > floor_index and net[left] > half:
left -= 1
right = index
ceiling = min(net.size - 1, index + span)
while right < ceiling and net[right] > half:
right += 1
left_x = right_x = float("nan")
if left < index and net[left] <= half:
rise = float(net[left + 1] - net[left])
left_x = float(x[left] + (half - net[left]) / rise * (x[left + 1] - x[left])) \
if rise > 0 else float(x[left])
if right > index and net[right] <= half:
fall = float(net[right - 1] - net[right])
right_x = float(x[right] - (net[right] - half) / fall * (x[right] - x[right - 1])) \
if fall > 0 else float(x[right])
return left_x, right_x
def _core(x: np.ndarray, amplitude: float, center: float, width: float,
shape: str) -> np.ndarray:
offset = x - center
if shape == "lorentz":
return amplitude / (1.0 + (offset / width) ** 2)
return amplitude * np.exp(-0.5 * (offset / width) ** 2)
def _shape_model(x: np.ndarray, params: np.ndarray, xref: float, shape: str) -> np.ndarray:
amplitude, center, width, c0, c1 = params
return _core(x, amplitude, center, width, shape) + c0 + c1 * (x - xref)
def _multi_model(x: np.ndarray, params: np.ndarray, xref: float, shape: str,
peaks: int) -> np.ndarray:
"""1..N shape components plus a linear background; params = 3*N + 2."""
total = np.zeros_like(x, dtype=float)
for index in range(peaks):
amplitude, center, width = params[3 * index:3 * index + 3]
total = total + _core(x, amplitude, center, width, shape)
return total + params[3 * peaks] + params[3 * peaks + 1] * (x - xref)
def fit_profile(x: np.ndarray, y: np.ndarray, start: list[float], shape: str,
bounds: list[tuple[float, float]], max_iter: int = 200,
peaks: int = 1, shared_width: bool = False) -> dict:
"""Fit 1..N peak components + linear background; returns parameters and flags.
``shared_width`` fits one common line width for all components, which is the
physically sensible and numerically stable model for an unresolved doublet.
Parameter layout: per component ``A, mu(, sigma)`` then ``c0, c1``; with a
shared width the single ``sigma`` is appended before ``c0, c1``.
"""
x = np.asarray(x, dtype=float)
y = np.asarray(y, dtype=float)
peaks = int(peaks)
per_peak = 2 if shared_width else 3
background = per_peak * peaks
if x.size < background + 3 or not np.all(np.isfinite(y)):
return {"ok": False}
xref = float(np.mean(x))
lower = np.array([bound[0] for bound in bounds], dtype=float)
upper = np.array([bound[1] for bound in bounds], dtype=float)
params = np.clip(np.asarray(start, dtype=float), lower, upper)
def model(values, parameters):
shared = parameters[2 * peaks] if shared_width else None
total = np.zeros_like(values, dtype=float)
for index in range(peaks):
if shared_width:
amplitude, center = parameters[2 * index], parameters[2 * index + 1]
width = shared
else:
amplitude, center, width = parameters[3 * index:3 * index + 3]
total = total + _core(values, amplitude, center, width, shape)
return total + parameters[background] + parameters[background + 1] * (values - xref)
residual = y - model(x, params)
cost = float(np.sum(residual * residual))
if not np.isfinite(cost):
return {"ok": False}
lam = 1e-3
converged = False
for _ in range(max_iter):
base = model(x, params)
jacobian = np.empty((x.size, params.size))
for j in range(params.size):
step = 1e-6 * max(1.0, abs(params[j]))
probe = params.copy()
probe[j] += step
jacobian[:, j] = (model(x, probe) - base) / step
normal = jacobian.T @ jacobian
gradient = jacobian.T @ residual
diagonal = np.diag(np.diag(normal)) + 1e-12 * np.eye(params.size)
relative = 1.0
stepped = False
for _ in range(40):
try:
delta = np.linalg.solve(normal + lam * diagonal, gradient)
except np.linalg.LinAlgError:
lam *= 10.0
continue
trial = np.clip(params + delta, lower, upper)
trial_residual = y - model(x, trial)
trial_cost = float(np.sum(trial_residual * trial_residual))
if np.isfinite(trial_cost) and trial_cost < cost:
relative = (cost - trial_cost) / max(cost, 1e-30)
params, residual, cost = trial, trial_residual, trial_cost
lam = max(lam / 3.0, 1e-15)
stepped = True
break
lam *= 10.0
if not stepped or relative < 1e-12:
converged = True
break
rms = float(np.sqrt(cost / max(1, x.size)))
# Parameter errors from the normal matrix at the final parameters; scale the
# columns first because amplitude, centre and width differ by orders of magnitude.
try:
base = model(x, params)
jacobian = np.empty((x.size, params.size))
for j in range(params.size):
step = 1e-6 * max(1.0, abs(params[j]))
probe = params.copy()
probe[j] += step
jacobian[:, j] = (model(x, probe) - base) / step
normal = jacobian.T @ jacobian
scale = np.sqrt(np.maximum(np.diag(normal), 1e-300))
correlation = normal / np.outer(scale, scale)
try:
inverse = np.linalg.inv(correlation)
except np.linalg.LinAlgError:
inverse = np.linalg.pinv(correlation)
variance = cost / max(1, x.size - params.size)
errors = np.sqrt(np.clip(np.diag(inverse) * variance, 0.0, None)) / scale
except (np.linalg.LinAlgError, ValueError, FloatingPointError):
errors = np.full(params.size, float("nan"))
components = []
usable = bool(converged and np.isfinite(params).all())
for index in range(peaks):
if shared_width:
amplitude = float(params[2 * index])
center = float(params[2 * index + 1])
width = float(params[2 * peaks])
low, high = lower[2 * peaks], upper[2 * peaks]
amplitude_err = float(errors[2 * index])
center_err = float(errors[2 * index + 1])
width_err = float(errors[2 * peaks])
else:
amplitude = float(params[3 * index])
center = float(params[3 * index + 1])
width = float(params[3 * index + 2])
low, high = lower[3 * index + 2], upper[3 * index + 2]
amplitude_err = float(errors[3 * index])
center_err = float(errors[3 * index + 1])
width_err = float(errors[3 * index + 2])
width_at_bound = bool(width <= low * (1 + 1e-6) or width >= high * (1 - 1e-6))
if amplitude <= 0 or width <= 0 or width_at_bound:
usable = False
if not x[0] - 1.0 <= center <= x[-1] + 1.0:
usable = False
fwhm = 2.0 * width if shape == "lorentz" else GAUSS_TO_FWHM * width
components.append({"amplitude": amplitude, "center": center,
"width": width, "fwhm": float(fwhm),
"amplitude_err": amplitude_err, "center_err": center_err,
"width_err": width_err,
"width_at_bound": width_at_bound})
first = components[0]
return {
"ok": usable,
"peaks": peaks,
"params": params.tolist(),
"xref": xref,
"components": components,
"center": first["center"],
"width": first["width"],
"amplitude": first["amplitude"],
"fwhm": first["fwhm"],
"c0": float(params[background]),
"c1": float(params[background + 1]),
"rms": rms,
"converged": converged,
"at_bound": any(item["width_at_bound"] for item in components),
"n_points": int(x.size),
}
def model_values(x, params, xref: float, shape: str) -> np.ndarray:
if not params:
return np.full(np.shape(x), np.nan, dtype=float)
return _shape_model(np.asarray(x, dtype=float), np.asarray(params, dtype=float), xref, shape)
def peak_prominence(net: np.ndarray, index: int) -> tuple[float, float]:
"""True topographic prominence of a local maximum and the level it sits on.
Walk outwards until the profile rises above the peak or the array ends; the
prominence base is the higher of the two minima on the way. Unlike a fixed
window this keeps resolved doublets apart and rejects shoulders.
"""
height = float(net[index])
left = index
left_min = height
while left > 0:
left -= 1
value = float(net[left])
if value > height:
break
if value < left_min:
left_min = value
right = index
right_min = height
while right < net.size - 1:
right += 1
value = float(net[right])
if value > height:
break
if value < right_min:
right_min = value
base = max(left_min, right_min)
return height - base, base
def find_peak_candidates(net: np.ndarray, min_distance: int, min_height: float,
min_prominence: float, max_peaks: int) -> list[dict]:
"""Local maxima with plateau centring, true prominence and neighbour suppression."""
n = net.size
if n < 3 or max_peaks <= 0:
return []
candidates = []
index = 1
while index < n - 1:
if net[index] > net[index - 1]:
end = index
while end + 1 < n and net[end + 1] == net[index]:
end += 1
if end + 1 < n and net[end] > net[end + 1]:
candidates.append((index + end) // 2)
index = end + 1
else:
index += 1
if not candidates:
return []
candidates.sort(key=lambda i: float(net[i]), reverse=True)
kept: list[dict] = []
for index in candidates:
height = float(net[index])
if height < min_height:
continue
if any(abs(index - other["sample"]) < max(1, int(min_distance)) for other in kept):
continue
prominence, base = peak_prominence(net, index)
if prominence < min_prominence:
continue
kept.append({"sample": int(index), "height_adu": height,
"base_adu": base, "prominence_adu": float(prominence)})
if len(kept) >= max_peaks:
break
return kept
def refine_peak(x: np.ndarray, net: np.ndarray, raw: np.ndarray, candidate: dict,
noise: float, step_px: float, fwhm_span: int, fit_window: int,
shape: str, saturated: bool, baseline_value: float = 0.0) -> dict:
"""Measure one detected peak: centre, width, height, area and fit quality."""
index = candidate["sample"]
n = net.size
left_x, right_x = half_max_crossings(x, net, index, fwhm_span)
fwhm_measured = (right_x - left_x
if np.isfinite(left_x) and np.isfinite(right_x) else float("nan"))
if fit_window > 0:
window_px = float(fit_window) * step_px
elif np.isfinite(fwhm_measured):
window_px = float(np.clip(2.0 * fwhm_measured, 8.0 * step_px, 90.0 * step_px))
else:
window_px = 15.0 * step_px
lower = int(index)
while lower > 0 and x[index] - x[lower] < window_px:
lower -= 1
upper = int(index)
while upper < n - 1 and x[upper] - x[index] < window_px:
upper += 1
center_parabola = parabola_center(x, net, index)
start_center = center_parabola if np.isfinite(center_parabola) else float(x[index])
start_width = fwhm_measured / (2.0 if shape == "lorentz" else GAUSS_TO_FWHM)
if not np.isfinite(start_width) or start_width <= 0:
start_width = 2.0 * step_px
span_px = float(x[upper] - x[lower])
bounds = [
(0.0, 10.0 * candidate["height_adu"] + 1.0),
(x[lower] - span_px, x[upper] + span_px),
(0.4 * step_px, max(4.0 * start_width, 0.5 * span_px)),
(-5.0 * candidate["height_adu"] - 1.0, 5.0 * candidate["height_adu"] + 1.0),
(-1e4, 1e4),
]
fit = fit_profile(x[lower:upper + 1], net[lower:upper + 1],
[candidate["height_adu"], start_center, start_width, 0.0, 0.0],
shape, bounds)
if fit.get("ok") and (fit["amplitude"] < 0.25 * candidate["height_adu"]
or fit["rms"] > 0.2 * fit["amplitude"]):
# Reject a fit that misses the measured peak: keep the robust estimators.
fit["ok"] = False
if fit.get("ok"):
center, fwhm, source = fit["center"], fit["fwhm"], "fit"
elif np.isfinite(center_parabola):
# Model-free vertex of the measured maximum: still a peak position.
center, fwhm, source = center_parabola, fwhm_measured, "apex"
else:
# Flat or saturated top makes the parabola unusable; report the sample
# where the maximum was detected rather than a window-dependent moment.
center, fwhm, source = float(x[index]), fwhm_measured, "sample"
height = float(net[index])
if fit.get("ok") and fit["amplitude"] > height:
height = float(fit["amplitude"])
return {
"sample_index": int(index),
"sensor_x_px": float(x[index]),
"center_px": float(center),
"center_source": source,
"center_parabola_px": float(center_parabola) if np.isfinite(center_parabola) else float("nan"),
"center_fit_px": float(fit["center"]) if fit.get("ok") else float("nan"),
"center_err_px": float(fit["components"][0]["center_err"]) if fit.get("ok") else float("nan"),
"sigma_px": float(fit["width"]) if fit.get("ok") else float("nan"),
"fwhm_px": float(fwhm) if np.isfinite(fwhm) else float("nan"),
"fwhm_halfmax_px": float(fwhm_measured) if np.isfinite(fwhm_measured) else float("nan"),
"height_net_adu": height,
"prominence_adu": float(candidate["prominence_adu"]),
"raw_adu": float(raw[index]),
"baseline_adu": float(baseline_value),
"snr": float(height / noise) if noise > 0 else float("inf"),
"area_net_adu_px": float(np.sum(np.maximum(net[lower:upper + 1], 0.0)) * step_px),
"saturated": bool(saturated),
"fit_ok": bool(fit.get("ok")),
"fit_rms_adu": float(fit["rms"]) if fit.get("ok") else float("nan"),
"fit_shape": shape,
"fit_params": fit.get("params") if fit.get("ok") else None,
"fit_xref": fit.get("xref") if fit.get("ok") else None,
"deblended": False,
"window_lo_px": float(x[lower]),
"window_hi_px": float(x[upper]),
"window_lo_sample": int(lower),
"window_hi_sample": int(upper),
}
def _fit_pair(x: np.ndarray, net: np.ndarray, first: dict, second: dict,
shape: str, step_px: float, noise: float) -> Optional[list[dict]]:
"""Joint two-component fit of a blended pair; None if it is not trustworthy."""
lower = min(first["window_lo_sample"], second["window_lo_sample"])
upper = max(first["window_hi_sample"], second["window_hi_sample"])
xx = x[lower:upper + 1]
yy = net[lower:upper + 1]
if xx.size < 11:
return None
starts: list[float] = []
bounds: list[tuple[float, float]] = []
separation = abs(float(second["sensor_x_px"]) - float(first["sensor_x_px"]))
widths = []
for peak in (first, second):
height = max(float(peak["height_net_adu"]), 1.0)
# Start from the detected maximum, not from the biased single-peak fit.
center = float(peak["sensor_x_px"])
width = float(peak["sigma_px"]) if np.isfinite(peak["sigma_px"]) and peak["sigma_px"] > 0 \
else 2.0 * step_px
widths.append(width)
starts.extend([height, center])
bounds.extend([(0.0, 10.0 * height + 1.0),
(center - 3.5 * step_px, center + 3.5 * step_px)])
# One shared line width: two components of the same blend come from the same
# instrument, and independent widths make the fit ill-conditioned.
shared = max(0.4 * step_px, min(max(widths), 0.5 * max(separation, step_px)))
starts.append(shared)
bounds.append((0.4 * step_px, max(8.0 * max(widths), 4.0 * step_px)))
scale = max(float(first["height_net_adu"]), float(second["height_net_adu"]), 1.0)
starts.extend([0.0, 0.0])
bounds.extend([(-3.0 * scale, 3.0 * scale), (-1e4, 1e4)])
fit = fit_profile(xx, yy, starts, shape, bounds, peaks=2, shared_width=True)
if not fit.get("ok"):
return None
components = fit["components"]
if components[1]["center"] - components[0]["center"] < 0.5 * step_px:
# The two components collapsed onto each other: this is one feature.
return None
if abs(components[0]["center"] - first["sensor_x_px"]) > 3.5 * step_px:
return None
if abs(components[1]["center"] - second["sensor_x_px"]) > 3.5 * step_px:
return None
updated = []
for peak, component in zip((first, second), components):
merged = dict(peak)
merged.update({
"center_px": float(component["center"]),
"center_fit_px": float(component["center"]),
"center_err_px": float(component["center_err"]),
"center_source": "pair",
"sigma_px": float(component["width"]),
"fwhm_px": float(component["fwhm"]),
"height_net_adu": float(component["amplitude"]),
"snr": float(component["amplitude"] / noise) if noise > 0 else float("inf"),
"fit_ok": True,
"fit_rms_adu": float(fit["rms"]),
"fit_params": [component["amplitude"], component["center"],
component["width"], 0.0, 0.0],
"fit_xref": float(fit["xref"]),
"deblended": True,
})
updated.append(merged)
return updated
def deblend_close_peaks(x: np.ndarray, net: np.ndarray, peaks: list[dict], shape: str,
step_px: float, noise: float, blend_factor: float = 1.2) -> list[dict]:
"""Refit unresolved pairs jointly instead of pulling their centres together."""
ordered = sorted(peaks, key=lambda item: item["sensor_x_px"])
result: list[dict] = []
index = 0
while index < len(ordered):
current = ordered[index]
if index + 1 < len(ordered):
neighbour = ordered[index + 1]
width_a = current["fwhm_px"] if np.isfinite(current["fwhm_px"]) else 4.0 * step_px
width_b = neighbour["fwhm_px"] if np.isfinite(neighbour["fwhm_px"]) else 4.0 * step_px
# Use the detected maxima: the fitted centres of a blend can cross.
separation = float(neighbour["sensor_x_px"] - current["sensor_x_px"])
if 0.0 < separation <= blend_factor * 0.5 * (width_a + width_b):
pair = _fit_pair(x, net, current, neighbour, shape, step_px, noise)
if pair is not None:
result.extend(pair)
index += 2
continue
result.append(current)
index += 1
return result
def dedupe_channel_peaks(peaks: list[dict], min_distance_px: float) -> list[dict]:
"""Drop repeated maxima closer than the peak distance, using detection positions.
The fitted centre of a blended line is pulled towards its neighbour, so the
distance is measured between the raw local maxima (``sensor_x_px``) and not
between the refined centres; otherwise a resolved doublet can collapse.
"""
kept: list[dict] = []
for peak in sorted(peaks, key=lambda item: item["height_net_adu"], reverse=True):
if any(abs(peak["sensor_x_px"] - other["sensor_x_px"]) < min_distance_px
for other in kept):
continue
kept.append(peak)
kept.sort(key=lambda item: item["sensor_x_px"])
return kept
def summarise_line(members: dict[str, dict]) -> dict:
"""Combine the per-channel peaks of one line into a line record (no id)."""
for channel, item in members.items():
item["channel"] = channel
items = list(members.values())
strongest = max((item["height_net_adu"] for item in items), default=0.0)
weights = np.array([
max(0.0, item["height_net_adu"]) if item["height_net_adu"] >= 0.25 * strongest else 0.0
for item in items
], dtype=float)
positions = np.array([item["center_px"] for item in items], dtype=float)
center = (float(np.dot(weights, positions) / weights.sum())
if weights.sum() > 0 else float(np.mean(positions)))
primary = max(items, key=lambda item: item["snr"])
return {
"sensor_x_px": center,
"center_spread_px": float(positions.max() - positions.min()),
"channels": members,
"channel_names": "".join(c for c in ("R", "G", "B") if c in members),
"n_channels": len(members),
"primary_channel": primary["channel"],
"height_net_adu": primary["height_net_adu"],
"max_height_net_adu": float(strongest),
"snr": primary["snr"],
"fwhm_px": primary["fwhm_px"],
"area_net_adu_px": primary["area_net_adu_px"],
"saturated": any(item["saturated"] for item in items),
"sample_index": primary["sample_index"],
"sample_step_px": None,
}
def merge_coincident_lines(lines: list[dict], tolerance_px: float = 1.5) -> list[dict]:
"""Join lines that ended up at the same place in complementary channels.
Weak lines can be detected a few pixels apart in different CFA channels; if
the refined centres agree to better than one sample they are one line.
"""
merged: list[dict] = []
for line in sorted(lines, key=lambda item: item["sensor_x_px"]):
target = None
for other in merged:
if abs(line["sensor_x_px"] - other["sensor_x_px"]) < tolerance_px:
target = other
break
if target is None:
merged.append({"sensor_x_px": line["sensor_x_px"],
"channels": dict(line["channels"])})
continue
combined = dict(target["channels"])
for channel, peak in line["channels"].items():
other = combined.get(channel)
if other is None or peak["snr"] > other["snr"]:
combined[channel] = peak
target["channels"] = combined
summaries = [summarise_line(item["channels"]) for item in merged]
summaries.sort(key=lambda entry: entry["sensor_x_px"])
for index, summary in enumerate(summaries, 1):
summary["id"] = f"P{index:03d}"
return summaries
def group_channels(peaks: dict[str, list[dict]], tolerance_px: float,
merge_tolerance_px: float = 1.5) -> list[dict]:
"""Merge per-channel peaks that belong to the same spectral line."""
# Cluster on the detected local maxima, not on the fitted centres: a single
# peak fit inside a blend is pulled towards its neighbour and would wrongly
# merge two resolved lines.
flat = [(item["sensor_x_px"], channel, item)
for channel, items in peaks.items() for item in items]
flat.sort(key=lambda row: row[0])
groups: list[list[tuple[float, str, dict]]] = []
for entry in flat:
if groups and entry[0] - groups[-1][-1][0] <= tolerance_px:
groups[-1].append(entry)
else:
groups.append([entry])
lines = []
for group in groups:
members: dict[str, dict] = {}
for _, channel, item in group:
other = members.get(channel)
# One line cannot have two centres in the same CFA channel.
if other is None or item["snr"] > other["snr"]:
members[channel] = item
summary = summarise_line(dict(members))
summary["id"] = ""
lines.append(summary)
return merge_coincident_lines(lines, merge_tolerance_px)
def attach_wavelength(lines: list[dict], calibration: Optional[Calibration]) -> None:
"""Add wavelength, extrapolation flag and per-channel wavelength data."""
for line in lines:
for item in line["channels"].values():
if calibration is not None:
item["wavelength_nm"] = float(calibration(item["center_px"]))
item["extrapolated"] = bool(
item["center_px"] < calibration.span[0] or item["center_px"] > calibration.span[1])
item["dispersion_nm_per_px"] = float(calibration.dispersion(item["center_px"]))
item["fwhm_nm"] = float(abs(item["fwhm_px"] * item["dispersion_nm_per_px"]))
else:
item["wavelength_nm"] = float("nan")
item["extrapolated"] = False
item["dispersion_nm_per_px"] = float("nan")
item["fwhm_nm"] = float("nan")
if calibration is not None:
line["wavelength_nm"] = float(calibration(line["sensor_x_px"]))
line["extrapolated"] = bool(
line["sensor_x_px"] < calibration.span[0] or line["sensor_x_px"] > calibration.span[1])
line["dispersion_nm_per_px"] = float(calibration.dispersion(line["sensor_x_px"]))
line["fwhm_nm"] = float(abs(line["fwhm_px"] * line["dispersion_nm_per_px"]))
else:
line["wavelength_nm"] = float("nan")
line["extrapolated"] = False
line["dispersion_nm_per_px"] = float("nan")
line["fwhm_nm"] = float("nan")
line.update({"match_species": "", "match_nm": float("nan"),
"match_delta_nm": float("nan"), "match_label": ""})
line["x_plot"] = line["wavelength_nm"] if calibration is not None else line["sensor_x_px"]
+551
View File
@@ -0,0 +1,551 @@
import csv
import json
import os
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
import numpy as np
import peak_finder as pf
import spectro_core as core
HERE = Path(__file__).resolve().parent
TEST_TMP = os.environ.get("PEAK_FINDER_TEST_TMP")
def _find_recording(name: str) -> Path:
"""Look for recordings/ here and one or two levels up.
The recordings often live beside the program folder (Light/recordings vs
Light/Programs), so a fixed path would silently skip the end-to-end tests.
"""
for base in (HERE, HERE.parent, HERE.parent.parent):
candidate = base / "recordings" / name
if candidate.is_dir():
return candidate
return HERE / "recordings" / name
# Optional overrides let the end-to-end tests run against a recording kept elsewhere.
RECORDING = Path(os.environ.get("PEAK_FINDER_TEST_RECORDING",
_find_recording("hq_20261101_114417")))
CALIBRATION = Path(os.environ.get("PEAK_FINDER_TEST_CALIBRATION",
HERE / "wavelength_calibration" / "calibration.json"))
MODULE_DIR = Path(os.environ.get("PEAK_FINDER_TEST_MODULE_DIR", HERE))
AVAILABLE = RECORDING.is_dir() and CALIBRATION.is_file()
def read_rows(path) -> list[dict]:
with open(path, newline="", encoding="utf-8") as handle:
return list(csv.DictReader(handle))
class temp_directory:
"""Scratch directory context manager; PEAK_FINDER_TEST_TMP overrides the location.
A plain ``mkdir`` is used instead of ``tempfile.mkdtemp`` so the directory
keeps the parent's permissions, which matters inside restrictive sandboxes.
"""
def __init__(self):
import shutil
import uuid
self._shutil = shutil
base = Path(TEST_TMP) if TEST_TMP else Path(tempfile.gettempdir())
base.mkdir(parents=True, exist_ok=True)
self.path = base / f"pf_case_{uuid.uuid4().hex[:10]}"
self.path.mkdir()
def __enter__(self) -> str:
return str(self.path)
def __exit__(self, *exc_info) -> bool:
self._shutil.rmtree(self.path, ignore_errors=True)
return False
def gaussian(x, amplitude, center, sigma, base=0.0, slope=0.0, xref=None):
xref = float(np.mean(x)) if xref is None else xref
return base + slope * (x - xref) + amplitude * np.exp(-0.5 * ((x - center) / sigma) ** 2)
def make_peak(center, height, fwhm, snr, channel="G", **extra):
peak = {
"channel": channel,
"sample_index": int(round(center / 2.0)),
"sensor_x_px": float(round(center / 2.0) * 2.0),
"center_px": float(center),
"center_source": "fit",
"center_parabola_px": float(center) + 0.3,
"center_fit_px": float(center),
"sigma_px": fwhm / 2.3548200450309493,
"fwhm_px": fwhm,
"fwhm_halfmax_px": fwhm * 1.02,
"height_net_adu": height,
"prominence_adu": height * 0.9,
"raw_adu": height + 100.0,
"baseline_adu": 10.0,
"snr": snr,
"area_net_adu_px": height * fwhm,
"saturated": False,
"fit_ok": True,
"fit_rms_adu": height * 0.02,
"fit_shape": "gauss",
"fit_params": [height, float(center), fwhm / 2.3548200450309493, 10.0, 0.0],
"fit_xref": float(center),
"window_lo_px": float(center) - 20.0,
"window_hi_px": float(center) + 20.0,
"window_lo_sample": 0,
"window_hi_sample": 10,
}
peak.update(extra)
return peak
def make_lines():
wavelengths = [404.7, 435.8, 546.1, 577.0, 579.1]
return [
{"id": f"P{index:03d}", "wavelength_nm": value, "sensor_x_px": 200.0 + 100.0 * index,
"snr": float(10 * index), "center_spread_px": 0.5, "extrapolated": False}
for index, value in enumerate(wavelengths, 1)
]
class EstimatorTests(unittest.TestCase):
def test_baseline_follows_continuum_not_peaks(self):
x = np.arange(0, 600, 2.0)
y = 100.0 + 0.05 * x + gaussian(x, 500.0, 300.0, 5.0)
baseline = pf.estimate_baseline(y, 50, lambda values, width: values)
error = np.abs(baseline - (100.0 + 0.05 * x))[60:-60]
self.assertLess(float(np.max(error)), 5.0)
def test_baseline_disabled_returns_zero(self):
y = np.arange(10.0)
np.testing.assert_array_equal(pf.estimate_baseline(y, 0, lambda v, w: v), np.zeros(10))
def test_noise_proxy_of_white_noise(self):
rng = np.random.default_rng(7)
y = rng.normal(0.0, 7.0, 4000)
self.assertAlmostEqual(pf.estimate_noise(y), 7.0, delta=0.5)
def test_parabola_and_half_max_width(self):
x = np.arange(0, 200, 2.0)
y = gaussian(x, 1000.0, 101.0, 4.0)
index = int(np.argmax(y))
self.assertAlmostEqual(pf.parabola_center(x, y, index), 101.0, delta=0.5)
left, right = pf.half_max_crossings(x, y, index, 40)
self.assertAlmostEqual(right - left, 2.3548200450309493 * 4.0, delta=0.6)
class FitTests(unittest.TestCase):
def test_gaussian_fit_recovers_parameters(self):
rng = np.random.default_rng(11)
x = np.arange(600.0, 700.0, 2.0)
truth = gaussian(x, 1200.0, 651.7, 3.4, base=40.0, slope=0.8, xref=650.0)
y = truth + rng.normal(0.0, 8.0, x.size)
bounds = [(0.0, 20000.0), (580.0, 720.0), (0.8, 40.0), (-2000.0, 2000.0), (-100.0, 100.0)]
fit = pf.fit_profile(x, y, [1000.0, 650.0, 4.0, 0.0, 0.0], "gauss", bounds)
self.assertTrue(fit["ok"])
self.assertAlmostEqual(fit["center"], 651.7, delta=0.4)
self.assertAlmostEqual(fit["fwhm"], 2.3548200450309493 * 3.4, delta=0.6)
self.assertAlmostEqual(fit["amplitude"], 1200.0, delta=120.0)
def test_lorentzian_fit_recovers_width(self):
rng = np.random.default_rng(3)
x = np.arange(0.0, 120.0, 2.0)
y = 900.0 / (1.0 + ((x - 61.2) / 3.0) ** 2) + rng.normal(0.0, 5.0, x.size)
bounds = [(0.0, 10000.0), (20.0, 100.0), (1.0, 20.0), (-500.0, 500.0), (-50.0, 50.0)]
fit = pf.fit_profile(x, y, [900.0, 60.0, 3.0, 0.0, 0.0], "lorentz", bounds)
self.assertTrue(fit["ok"])
self.assertAlmostEqual(fit["fwhm"], 6.0, delta=0.6)
def test_model_values_round_trip(self):
x = np.arange(0.0, 10.0)
params = [5.0, 4.0, 1.5, 1.0, 0.0]
model = pf.model_values(x, params, 4.0, "gauss")
self.assertAlmostEqual(float(np.max(model)), 6.0, delta=1e-6)
self.assertTrue(np.all(np.isfinite(model)))
class DetectionTests(unittest.TestCase):
def test_three_peaks_found_in_continuum(self):
rng = np.random.default_rng(5)
x = np.arange(0.0, 800.0, 2.0)
y = (0.02 * x + gaussian(x, 1000.0, 200.0, 4.0) + gaussian(x, 400.0, 400.0, 6.0)
+ gaussian(x, 120.0, 600.0, 3.0) + rng.normal(0.0, 2.0, x.size))
baseline = pf.estimate_baseline(y, 50, lambda values, width: values)
net = y - baseline
noise = pf.estimate_noise(net)
candidates = pf.find_peak_candidates(net, 4, 20.0, 20.0, 10)
centers = sorted(x[item["sample"]] for item in candidates)
self.assertEqual(len(centers), 3)
for found, expected in zip(centers, (200.0, 400.0, 600.0)):
self.assertAlmostEqual(found, expected, delta=4.0)
self.assertLess(noise, 5.0)
def test_refined_centres_are_sub_sample(self):
rng = np.random.default_rng(9)
x = np.arange(0.0, 200.0, 2.0)
y = gaussian(x, 800.0, 97.3, 2.6) + rng.normal(0.0, 4.0, x.size)
baseline = pf.estimate_baseline(y, 40, lambda values, width: values)
net = y - baseline
noise = pf.estimate_noise(net)
candidate = pf.find_peak_candidates(net, 4, 20.0, 20.0, 4)[0]
peak = pf.refine_peak(x, net, y, candidate, noise, 2.0, 30, 0, "gauss", False)
self.assertAlmostEqual(peak["center_px"], 97.3, delta=0.6)
self.assertAlmostEqual(peak["fwhm_px"], 2.3548200450309493 * 2.6, delta=0.8)
self.assertTrue(peak["fit_ok"])
def test_silent_profile_has_no_candidates(self):
net = np.zeros(50)
self.assertEqual(pf.find_peak_candidates(net, 4, 1.0, 1.0, 5), [])
class BlendTests(unittest.TestCase):
def test_failed_fit_falls_back_to_the_measured_maximum(self):
from unittest.mock import patch
x = np.arange(400.0, 700.0, 2.0)
net = (gaussian(x, 10000.0, 542.0, 15.0)
+ 4000.0 * np.exp(-np.clip(x - 542.0, 0.0, None) / 120.0))
candidate = {"sample": int(np.argmax(net)), "height_adu": float(net.max()),
"base_adu": 0.0, "prominence_adu": float(net.max())}
with patch.object(core, "fit_profile", return_value={"ok": False}):
peak = pf.refine_peak(x, net, net, candidate, 1.0, 2.0, 30, 0, "gauss", False)
self.assertEqual(peak["center_source"], "apex")
self.assertAlmostEqual(peak["center_px"], 542.0, delta=1.0)
def test_flat_top_falls_back_to_the_sampled_maximum(self):
from unittest.mock import patch
x = np.arange(0.0, 60.0, 2.0)
net = np.full(x.size, 100.0)
net[10:16] = 1000.0 # saturated flat top
candidate = {"sample": 12, "height_adu": 1000.0,
"base_adu": 100.0, "prominence_adu": 900.0}
with patch.object(core, "fit_profile", return_value={"ok": False}):
peak = pf.refine_peak(x, net, net, candidate, 1.0, 2.0, 30, 0, "gauss", False)
self.assertEqual(peak["center_source"], "sample")
self.assertAlmostEqual(peak["center_px"], 24.0) # x[12]
def test_topographic_prominence_rejects_a_shoulder(self):
net = np.array([0.0, 100.0, 200.0, 500.0, 300.0, 400.0, 200.0, 0.0])
prominence, base = pf.peak_prominence(net, 5)
self.assertAlmostEqual(prominence, 100.0)
self.assertAlmostEqual(base, 300.0)
prominence, _ = pf.peak_prominence(net, 3)
self.assertAlmostEqual(prominence, 500.0)
def test_resolved_doublet_survives_detection(self):
rng = np.random.default_rng(21)
x = np.arange(780.0, 830.0, 2.0)
net = (gaussian(x, 3600.0, 800.0, 2.5) + gaussian(x, 4300.0, 808.0, 2.5)
+ rng.normal(0.0, 20.0, x.size))
candidates = pf.find_peak_candidates(net, 2, 200.0, 100.0, 10)
self.assertEqual(len(candidates), 2)
self.assertEqual(sorted(round(float(x[item["sample"]]), 1) for item in candidates),
[800.0, 808.0])
def test_doublet_is_deblended_with_one_shared_width(self):
rng = np.random.default_rng(22)
x = np.arange(780.0, 830.0, 2.0)
net = (gaussian(x, 3600.0, 800.0, 2.5) + gaussian(x, 4300.0, 808.0, 2.5)
+ rng.normal(0.0, 20.0, x.size))
noise = pf.estimate_noise(net)
candidates = pf.find_peak_candidates(net, 2, 200.0, 100.0, 10)
peaks = [pf.refine_peak(x, net, net, item, noise, 2.0, 30, 0, "gauss", False)
for item in candidates]
deduped = pf.dedupe_channel_peaks(peaks, 4.0)
self.assertEqual(len(deduped), 2)
merged = pf.deblend_close_peaks(x, net, deduped, "gauss", 2.0, noise)
self.assertEqual(len(merged), 2)
self.assertTrue(all(peak["deblended"] for peak in merged))
centres = sorted(peak["center_px"] for peak in merged)
self.assertAlmostEqual(centres[0], 800.0, delta=0.8)
self.assertAlmostEqual(centres[1], 808.0, delta=0.8)
self.assertEqual(len({round(peak["fwhm_px"], 6) for peak in merged}), 1)
self.assertAlmostEqual(merged[0]["fwhm_px"], 2.3548200450309493 * 2.5, delta=0.8)
for peak in merged:
self.assertTrue(np.isfinite(peak["center_err_px"]))
def test_coincident_weak_lines_are_merged(self):
first = pf.summarise_line({"B": make_peak(912.0, 430.0, 9.2, 75.0, "B")})
second = pf.summarise_line({"R": make_peak(917.0, 70.0, 8.0, 15.0, "R")})
first["id"] = second["id"] = ""
lines = [dict(first, sensor_x_px=912.0), dict(second, sensor_x_px=917.0)]
self.assertEqual(len(pf.merge_coincident_lines(lines, 1.5)), 2)
lines[1]["sensor_x_px"] = 912.4
combined = pf.merge_coincident_lines(lines, 1.5)
self.assertEqual(len(combined), 1)
self.assertEqual(combined[0]["channel_names"], "RB")
class GroupingTests(unittest.TestCase):
def test_same_line_in_three_channels_merges(self):
peaks = {
"R": [make_peak(500.0, 300.0, 9.0, 40.0, "R")],
"G": [make_peak(501.0, 1200.0, 10.0, 200.0, "G")],
"B": [make_peak(499.5, 800.0, 11.0, 90.0, "B")],
}
lines = pf.group_channels(peaks, 4.0)
self.assertEqual(len(lines), 1)
line = lines[0]
self.assertEqual(line["channel_names"], "RGB")
self.assertEqual(line["primary_channel"], "G")
self.assertEqual(line["id"], "P001")
self.assertAlmostEqual(line["sensor_x_px"], (300 * 500 + 1200 * 501 + 800 * 499.5) / 2300,
places=6)
self.assertAlmostEqual(line["center_spread_px"], 1.5, places=6)
def test_distinct_lines_stay_separate(self):
peaks = {"G": [make_peak(500.0, 100.0, 8.0, 20.0), make_peak(520.0, 90.0, 8.0, 18.0)]}
lines = pf.group_channels(peaks, 4.0)
self.assertEqual([line["id"] for line in lines], ["P001", "P002"])
def test_channel_duplicate_is_dropped(self):
peaks = {"G": [make_peak(500.0, 300.0, 9.0, 50.0), make_peak(500.4, 900.0, 9.0, 90.0)]}
lines = pf.group_channels(peaks, 4.0)
self.assertEqual(len(lines), 1)
self.assertEqual(lines[0]["height_net_adu"], 900.0)
def test_dedupe_keeps_neighbours_further_than_distance(self):
peaks = [make_peak(100.0, 500.0, 8.0, 40.0), make_peak(112.0, 400.0, 8.0, 30.0),
make_peak(103.0, 300.0, 8.0, 20.0)]
kept = pf.dedupe_channel_peaks(peaks, 12.0)
self.assertEqual([peak["center_px"] for peak in kept], [100.0, 112.0])
class OutputNamingTests(unittest.TestCase):
def test_default_output_names_the_frame(self):
single = pf.default_output_directory(r"C:\raw\hq_20261112_200710", [7])
self.assertEqual(single.name, "hq_20261112_200710_f0007")
stacked = pf.default_output_directory(r"C:\raw\hq_20261112_200710", [5, 6, 7])
self.assertEqual(stacked.name, "hq_20261112_200710_f0005-0007_avg3")
self.assertEqual(stacked.parent.name, "peak_exports")
class SelectionTests(unittest.TestCase):
def setUp(self):
self.lines = make_lines()
def test_all_and_none(self):
self.assertEqual(len(pf.parse_selection("all", self.lines)), 5)
self.assertEqual(pf.parse_selection("none", self.lines), set())
self.assertEqual(pf.parse_selection("ALL", self.lines), pf.parse_selection("*", self.lines))
def test_ids_and_ranges(self):
self.assertEqual(pf.parse_selection("2", self.lines), {"P002"})
self.assertEqual(pf.parse_selection("1-2", self.lines), {"P001", "P002"})
self.assertEqual(pf.parse_selection("id:4", self.lines), {"P004"})
def test_wavelength_ranges(self):
self.assertEqual(pf.parse_selection("nm:540-560", self.lines), {"P003"})
self.assertEqual(pf.parse_selection("540-560", self.lines), {"P003"})
self.assertEqual(pf.parse_selection("546.0", self.lines), {"P003"})
self.assertEqual(pf.parse_selection("nm:430-440", self.lines), {"P002"})
def test_top_n_and_union(self):
self.assertEqual(pf.parse_selection("top:2", self.lines), {"P004", "P005"})
self.assertEqual(pf.parse_selection("1,nm:540-560", self.lines), {"P001", "P003"})
def test_bad_tokens_raise(self):
for token in ("9", "0", "abc", "id:3-9", "nm:"):
with self.assertRaises(ValueError):
pf.parse_selection(token, self.lines)
def test_wavelength_selection_needs_calibration(self):
lines = [dict(line, wavelength_nm=float("nan")) for line in self.lines]
with self.assertRaises(ValueError):
pf.parse_selection("nm:500-600", lines)
self.assertEqual(pf.parse_selection("2", lines), {"P002"})
class CalibrationTests(unittest.TestCase):
def write(self, payload):
handle = tempfile.NamedTemporaryFile("w", suffix=".json", delete=False, encoding="utf-8", dir=TEST_TMP)
handle.write(json.dumps(payload))
handle.close()
return handle.name
def test_scale_offset_and_inverse(self):
path = self.write({"coefficients_descending": [0.285, 349.45],
"calibrated_pixel_range": [304.0, 808.5]})
calibration = pf.Calibration(path, offset_nm=0.5)
calibration.set_width(2028)
self.assertAlmostEqual(float(calibration(600.0)), 0.285 * 600 + 349.45 + 0.5, places=6)
self.assertAlmostEqual(float(calibration.dispersion(600.0)), 0.285, places=9)
self.assertAlmostEqual(float(calibration.inverse(calibration(600.0))), 600.0, delta=1.0)
self.assertAlmostEqual(calibration.nm_span[0], 0.285 * 304 + 349.95, places=6)
def test_invalid_calibrations(self):
for payload in ({"coefficients_descending": [0.0, 1.0], "calibrated_pixel_range": [1, 2]},
{"coefficients_descending": [1.0], "calibrated_pixel_range": [1, 2]},
{"coefficients_descending": [1.0, 2.0], "calibrated_pixel_range": [3, 2]},
{"coefficients_descending": [1.0, 2.0]}):
with self.assertRaises(ValueError):
pf.Calibration(self.write(payload))
def test_geometry_checks(self):
path = self.write({"coefficients_descending": [0.285, 349.45],
"calibrated_pixel_range": [304.0, 808.5]})
calibration = pf.Calibration(path)
source = SimpleNamespace(full_size=(2028, 1520), pattern="BGGR")
calibration.validate_geometry(source, lambda *a, **k: None)
bad = pf.Calibration(self.write({"coefficients_descending": [0.001, -2.0, 900.0],
"calibrated_pixel_range": [304.0, 808.5]}))
with self.assertRaises(ValueError):
bad.validate_geometry(source, lambda *a, **k: None)
class LineTableTests(unittest.TestCase):
def test_match_nearest_within_tolerance(self):
table = [{"wavelength_nm": 546.0, "species": "Hg I", "label": "546"},
{"wavelength_nm": 579.1, "species": "Hg I", "label": "579"}]
lines = [{"id": "P001", "wavelength_nm": 546.4, "sensor_x_px": 1.0},
{"id": "P002", "wavelength_nm": 600.0, "sensor_x_px": 2.0}]
pf.match_lines(lines, table, 1.5)
self.assertEqual(lines[0]["match_species"], "Hg I")
self.assertAlmostEqual(lines[0]["match_delta_nm"], 0.4, places=6)
self.assertEqual(lines[1]["match_species"], "")
self.assertTrue(np.isnan(lines[1]["match_delta_nm"]))
def test_csv_table_with_flexible_columns(self):
with temp_directory() as folder:
path = Path(folder) / "lines.csv"
path.write_text("nm,element\n546.0735,Hg I\n435.8328,Hg I\n", encoding="utf-8")
table = pf.load_line_table(path)
self.assertEqual([entry["wavelength_nm"] for entry in table], [435.8328, 546.0735])
def build_spectrum_and_lines():
"""Run the pipeline up to detection, for tests that need real peaks."""
hq = pf.load_hq(str(MODULE_DIR))
args = pf.build_parser().parse_args(["--input", str(RECORDING), "--output", "unused",
"--quiet"])
source = hq.RecordingSource(str(RECORDING))
source.start()
image, original, indices, frame_meta = pf.read_frames(hq, source, args, lambda *a, **k: None)
roi = hq.detect_auto_roi(hq.make_preview(original, source.pattern), *source.full_size)
source.close()
meta = dict(source.base_meta)
meta.update({key: value for key, value in frame_meta.items()
if not str(key).startswith("_")})
black, full = hq.resolve_levels(meta, source.bit_depth, args.black, args.full_scale)
spectrum = pf.SpectrumData(hq, source, image, original, indices, roi, args, black, full)
lines = pf.group_channels(spectrum.detect(), args.group_tol)
pf.attach_wavelength(lines, None)
return spectrum, lines
@unittest.skipUnless(AVAILABLE, "recordings or calibration not available")
class PickerTests(unittest.TestCase):
def test_hover_previews_and_click_selects(self):
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.backend_bases import KeyEvent, MouseEvent
spectrum, lines = build_spectrum_and_lines()
self.assertGreaterEqual(len(lines), 3)
target = lines[2]
seen = {}
def fake_show(block=True):
figure = plt.gcf()
canvas = figure.canvas
axes = figure.axes[0]
x_display = float(axes.transData.transform((target["x_plot"], 0.0))[0])
y_display = float(axes.transAxes.transform((0.0, 0.5))[1])
canvas.callbacks.process(
"motion_notify_event", MouseEvent("motion_notify_event", canvas,
x_display, y_display))
seen["axes"] = len(figure.axes)
seen["images"] = len(figure.axes[1].images)
seen["curves"] = len(figure.axes[2].lines)
seen["title"] = figure.axes[2].get_title()
canvas.callbacks.process(
"button_press_event", MouseEvent("button_press_event", canvas,
x_display, y_display, button=1))
canvas.callbacks.process(
"key_press_event", KeyEvent("key_press_event", canvas, key="e"))
real_show = plt.show
plt.show = fake_show
try:
chosen = pf.pick_peaks(spectrum, lines, None, set())
finally:
plt.show = real_show
self.assertEqual(chosen, {target["id"]})
self.assertEqual(seen["axes"], 3) # spectrum + RAW patch + zoom
self.assertGreaterEqual(seen["images"], 1) # RAW patch rendered
self.assertGreaterEqual(seen["curves"], 3) # R/G/B profiles drawn
self.assertIn(target["id"], seen["title"])
@unittest.skipUnless(AVAILABLE, "recordings or calibration not available")
class EndToEndTests(unittest.TestCase):
def test_export_selected_peaks(self):
with temp_directory() as folder:
code = pf.main(["--input", str(RECORDING), "--calibration", str(CALIBRATION),
"--output", folder, "--select", "nm:530-560",
"--no-images", "--quiet", "--module-dir", str(MODULE_DIR)])
self.assertEqual(code, 0)
output = Path(folder)
rows = read_rows(output / "selected_peaks.csv")
self.assertTrue(rows)
for row in rows:
self.assertGreaterEqual(float(row["wavelength_nm"]), 530.0)
self.assertLessEqual(float(row["wavelength_nm"]), 560.0)
self.assertTrue(0 <= float(row["patch_x0"]) < float(row["patch_x1"]) <= 2028)
self.assertTrue(0 <= float(row["patch_y0"]) < float(row["patch_y1"]) <= 1520)
self.assertTrue(0 <= float(row["sensor_x_px"]))
self.assertTrue((output / row["shape_csv"]).is_file())
manifest = json.loads((output / "peaks.json").read_text(encoding="utf-8"))
all_rows = read_rows(output / "peaks_all.csv")
self.assertGreaterEqual(len(all_rows), len(manifest["peaks"]))
self.assertEqual(manifest["calibration"]["wavelength_offset_nm"], 0.0)
self.assertTrue(manifest["coordinate_convention"]["sensor_x_px"])
def test_without_calibration_only_pixels_are_exported(self):
with temp_directory() as folder:
code = pf.main(["--input", str(RECORDING), "--output", folder, "--select", "top:1",
"--no-images", "--quiet", "--module-dir", str(MODULE_DIR)])
self.assertEqual(code, 0)
row = read_rows(Path(folder) / "selected_peaks.csv")[0]
self.assertEqual(row["wavelength_nm"], "")
self.assertGreater(float(row["sensor_x_px"]), 0.0)
manifest = json.loads((Path(folder) / "peaks.json").read_text(encoding="utf-8"))
self.assertIsNone(manifest["calibration"])
def test_bad_frame_is_rejected(self):
code = pf.main(["--input", str(RECORDING), "--frame", "100000", "--quiet",
"--output", str(Path(tempfile.gettempdir()) / "pf_unused"),
"--no-images", "--module-dir", str(MODULE_DIR)])
self.assertEqual(code, 2)
def test_wavelength_offset_shifts_the_scale(self):
with temp_directory() as folder:
code = pf.main(["--input", str(RECORDING), "--calibration", str(CALIBRATION),
"--wavelength-offset", "-10", "--output", folder,
"--select", "top:1", "--no-images", "--quiet",
"--module-dir", str(MODULE_DIR)])
self.assertEqual(code, 0)
row = read_rows(Path(folder) / "selected_peaks.csv")[0]
self.assertAlmostEqual(float(row["wavelength_nm"]), 545.8237 - 10.0, delta=0.2)
if __name__ == "__main__":
unittest.main()
+65
View File
@@ -0,0 +1,65 @@
import json
from pathlib import Path
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import numpy as np
import wavelength_player as wp
def calibration(coeff=(.6, -8), span=(690, 927)):
data = dict(coefficients_descending=list(coeff), calibrated_pixel_range=list(span),
roi=[620, 620, 1034, 848])
with patch.object(Path, 'read_text', return_value=json.dumps(data)):
return wp.SavedCalibration('dummy.json')
class PlayerTests(unittest.TestCase):
def test_exact_saved_coefficients(self):
model = calibration()
np.testing.assert_array_equal(model.coeff, [.6, -8])
np.testing.assert_allclose(model([690, 742, 926.5]), [406, 437.2, 547.9])
def test_invalid_models(self):
for coeff, span in (([0, 3], [1, 2]), ([float('nan'), 1], [1, 2]),
([1], [1, 2]), ([1, 2], [4, 3])):
with self.assertRaises(ValueError):
calibration(coeff, span)
def test_quadratic_and_reverse_geometry(self):
source = SimpleNamespace(full_size=(2028, 1520), pattern='BGGR')
calibration([.00001, .5, 5]).validate_geometry(source)
calibration([-.5, 1000]).validate_geometry(source)
with self.assertRaises(ValueError):
calibration([.001, -1, 500]).validate_geometry(source)
def test_geometry_mismatch(self):
model = calibration()
model.data['recording'] = 'reference'
with patch.object(Path, 'is_file', return_value=True), patch.object(Path, 'read_text',
return_value=json.dumps(dict(width=4056, height=3040))):
with self.assertRaises(ValueError):
model.validate_geometry(SimpleNamespace(full_size=(2028, 1520), pattern='BGGR'))
def test_processing_signed_signal_and_sensor_coordinates(self):
hq = wp.load_player()
model = calibration()
wp.install_hooks(hq, model)
args = hq.parse_args(['--smooth', '1', '--black', '10'])
raw = np.full((16, 64), 8, dtype=np.uint16)
raw[:, 30:32] = 110
roi = hq.make_roi(10, 0, 60, 16, 64, 16)
result = hq.process_frame(raw, {}, roi, 'BGGR', args, model, (64, 16), 'SBGGR12', 12)
for c in hq.CHANNELS:
self.assertEqual(result.signal_raw[c][0], -2)
np.testing.assert_array_equal(result.signal[c], result.signal_raw[c])
np.testing.assert_allclose(result.x_plot[c], model(result.x_px[c]))
self.assertEqual(result.normalized[c].max(), 1)
self.assertGreaterEqual(result.x_px[c][0], 10)
def test_help(self):
self.assertEqual(wp.main(['--help']), 0)
if __name__ == '__main__':
unittest.main()
+276
View File
@@ -0,0 +1,276 @@
#!/usr/bin/env python3
"""Independent Hg wavelength calibration; uses hq_spectrum recording/CFA APIs.
Peak finding uses the same core as peak_finder.py: topographic prominence,
Gaussian centre with a peak-top fallback, and a joint two-line fit for blended
doublets (the Hg 576.96/579.07 pair is only ~6 px apart). Baseline subtraction
is used for detection and peak heights only; `signal_adu` still reports the
black-corrected value at the peak sample.
Sensor coordinates are full-resolution columns.
"""
from __future__ import annotations
import argparse
import csv
import json
from pathlib import Path
import sys
import numpy as np
import hq_spectrum as hq
try:
import spectro_core as core
except ImportError as exc: # pragma: no cover - placement error
raise SystemExit('wavelength_calibration.py needs spectro_core.py in the same directory '
'(or on sys.path)') from exc
HG = (404.6563, 435.8328, 546.0735, 576.9598, 579.0663)
SOURCE = 'https://physics.nist.gov/PhysRefData/Handbook/Tables/mercurytable2.htm'
def parser():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--input', required=True, help='hq_record recording directory')
p.add_argument('--frame', type=int, default=0, help='zero-based frame index (default 0)')
p.add_argument('--roi', default='auto', help='auto or x0,y0,x1,y1; same as hq_spectrum')
p.add_argument('--black', type=float, help='black level in decoded ADU')
p.add_argument('--full-scale', type=float)
p.add_argument('--reduce', choices=['mean', 'median'], default='mean')
p.add_argument('--baseline-window', type=float, default=100,
help='baseline opening width in sensor px; 0 disables the baseline')
p.add_argument('--peak-distance', type=float, default=4,
help='minimum distance in sensor pixels per channel (2 px = one sample)')
p.add_argument('--peak-prominence', type=float, default=0.01,
help='topographic prominence as a fraction of the channel maximum')
p.add_argument('--peak-height', type=float, default=0.02,
help='peak height as a fraction of the channel maximum')
p.add_argument('--min-snr', type=float, default=15, help='peak height in units of the noise')
p.add_argument('--max-peaks', type=int, default=40, help='per channel')
p.add_argument('--fwhm-span', type=float, default=60, help='half-width search range in px')
p.add_argument('--fit-window', type=float, default=0, help='fit window half width in px; 0 auto')
p.add_argument('--shape', choices=['gauss', 'lorentz'], default='gauss')
p.add_argument('--assign', help='confirmed candidate identities, e.g. B1:404.6563,B2:435.8328,G3:546.0735')
p.add_argument('--order', type=int, choices=[1, 2], default=1)
p.add_argument('--headless', action='store_true', help='save candidates; fit only with explicit --assign')
p.add_argument('--output', default='wavelength_calibration', help='output directory')
return p
def extract(raw, meta, roi, pattern, depth, args):
planes = hq.extract_planes(raw, pattern, roi)
counts = {c: planes[c].shape[1] for c in hq.CHANNELS}
xs = hq.channel_x_positions(roi, pattern, counts['G'], min(counts['R'], counts['B']))
black, full = hq.resolve_levels(meta, depth, args.black, args.full_scale)
window = 0 if args.baseline_window <= 0 else max(3, int(round(args.baseline_window / 2.0)))
curves, candidates = {}, []
for c in hq.CHANNELS:
y = (np.median(planes[c], axis=0) if args.reduce == 'median'
else np.mean(planes[c], axis=0, dtype=np.float64)) - black
x = xs[c][:len(y)]
baseline = core.estimate_baseline(y, window, hq.smooth_vector)
net = y - baseline
noise = core.estimate_noise(net)
top = float(np.max(net)) if net.size else 0.0
positive = np.maximum(net, 0.0)
norm = positive / positive.max() if positive.max() > 0 else np.zeros(net.size)
peaks = []
if top > 0 and norm.size:
min_height = max(args.peak_height * top, args.min_snr * noise)
min_prominence = max(args.peak_prominence * top, 0.4 * args.min_snr * noise)
min_distance = max(1, int(round(args.peak_distance / 2.0)))
found = core.find_peak_candidates(net, min_distance, min_height,
min_prominence, args.max_peaks)
for candidate in found:
index = candidate['sample']
peaks.append(core.refine_peak(
x, net, y, candidate, noise, 2.0,
max(1, int(round(args.fwhm_span / 2.0))),
max(0, int(round(args.fit_window / 2.0))),
args.shape, core.is_saturated(raw, roi, x[index], full),
baseline_value=float(baseline[index])))
# Two lines closer than ~1.2 FWHM are fitted jointly, otherwise a
# single-peak fit collapses them onto one centre.
peaks = core.deblend_close_peaks(
x, net, core.dedupe_channel_peaks(peaks, args.peak_distance),
args.shape, 2.0, noise)
peaks.sort(key=lambda item: item['center_px'])
curves[c] = (x, y, baseline, net, norm)
for j, peak in enumerate(peaks, 1):
index = peak['sample_index']
candidates.append(dict(
id=f'{c}{j}', channel=c, sample=int(index),
pixel=float(peak['center_px']),
signal_adu=float(y[index]),
height_net_adu=float(peak['height_net_adu']),
normalized=float(norm[index]),
fwhm_px=float(peak['fwhm_px']),
snr=float(peak['snr']),
center_source=peak['center_source'],
deblended=bool(peak.get('deblended')),
saturated=bool(peak['saturated'])))
return curves, candidates, black, full
def assignments(text, candidates, order):
lookup = {v['id']: v for v in candidates}
result, used = [], set()
for token in text.split(','):
name, wavelength = token.strip().split(':')
name = name.upper().strip()
if name not in lookup or name in used:
raise ValueError(f'unknown or duplicate candidate: {name}')
nm = float(wavelength)
if not np.isfinite(nm) or nm <= 0:
raise ValueError('wavelength must be finite and positive')
used.add(name)
result.append({**lookup[name], 'wavelength_nm': nm})
if len(result) < order + 1:
raise ValueError(f'order {order} needs at least {order + 1} confirmed lines')
x = np.array([r['pixel'] for r in result])
y = np.array([r['wavelength_nm'] for r in result])
if len(np.unique(x)) != len(x) or len(np.unique(y)) != len(y):
raise ValueError('select one channel candidate per distinct spectral line')
sort = np.argsort(x)
if not (np.all(np.diff(y[sort]) > 0) or np.all(np.diff(y[sort]) < 0)):
raise ValueError('confirmed wavelength order must be monotonic in sensor x')
return result
def fit(confirmed, order, width):
x = np.array([r['pixel'] for r in confirmed])
y = np.array([r['wavelength_nm'] for r in confirmed])
model = np.polynomial.Polynomial.fit(x, y, order).convert()
coeff = model.coef[::-1]
derivative = np.polyval(np.polyder(coeff), [0, width - 1])
if np.any(np.abs(derivative) < 1e-12) or derivative[0] * derivative[1] <= 0:
raise ValueError('mapping is not monotonic across sensor; revise identities/order/ROI')
residual = np.polyval(coeff, x) - y
return coeff, residual
def write_csv(path, fields, rows):
with path.open('w', newline='', encoding='utf-8') as f:
w = csv.writer(f)
w.writerow(fields)
w.writerows(rows)
def run(args):
if args.frame < 0 or args.peak_distance <= 0 or args.max_peaks < 1:
raise ValueError('frame >= 0, peak-distance > 0 and max-peaks >= 1 required')
for v in (args.peak_height, args.peak_prominence):
if not np.isfinite(v) or not 0 <= v <= 1:
raise ValueError('normalized peak thresholds must be between 0 and 1')
import matplotlib
if args.headless:
matplotlib.use('Agg')
import matplotlib.pyplot as plt
out = Path(args.output)
out.mkdir(parents=True, exist_ok=True)
source = hq.RecordingSource(args.input)
try:
source.start()
raw, meta = source.read_at(args.frame)
roi = hq.parse_roi(args.roi, *source.full_size)
preview = hq.make_preview(raw, source.pattern)
if roi is None:
roi = hq.detect_auto_roi(preview, *source.full_size)
if roi is None:
raise ValueError('auto ROI failed; specify --roi x0,y0,x1,y1')
curves, candidates, black, full = extract(raw, meta, roi, source.pattern,
source.bit_depth, args)
finally:
source.close()
fields = ['id', 'channel', 'sample', 'pixel', 'signal_adu', 'height_net_adu',
'normalized', 'fwhm_px', 'snr', 'center_source', 'deblended', 'saturated']
write_csv(out / 'candidates.csv', fields, ([v[k] for k in fields] for v in candidates))
write_csv(out / 'profiles.csv',
['channel', 'pixel', 'black_corrected_adu', 'baseline_adu', 'net_adu',
'normalized_for_detection'],
([c, x, y, b, n, q] for c, arrays in curves.items()
for x, y, b, n, q in zip(*arrays)))
fig, axes = plt.subplots(3, 1, figsize=(13, 10), constrained_layout=True)
axes[0].imshow(preview, extent=(0, source.full_size[0], source.full_size[1], 0))
from matplotlib.patches import Rectangle
axes[0].add_patch(Rectangle((roi.x0, roi.y0), roi.width, roi.height, fill=False, edgecolor='orange'))
axes[0].set_title(f'Frame {args.frame} (zero based); ROI {roi.text()}; black={black:g} ADU')
for c, (x, y, baseline, net, norm) in curves.items():
axes[1].plot(x, y, color=hq.CHANNEL_COLORS[c], label=c)
axes[1].plot(x, baseline, color=hq.CHANNEL_COLORS[c], lw=0.6, ls=':', alpha=0.7)
axes[2].plot(x, norm, color=hq.CHANNEL_COLORS[c], label=c)
for v in candidates:
marker = 's' if v['deblended'] else 'o'
axes[2].plot(v['pixel'], v['normalized'], marker, color=hq.CHANNEL_COLORS[v['channel']],
mfc='none' if v['deblended'] else hq.CHANNEL_COLORS[v['channel']])
axes[2].annotate(v['id'], (v['pixel'], v['normalized']), xytext=(0, 8),
textcoords='offset points')
print(f"{v['id']:5s} x={v['pixel']:9.3f} FWHM={v['fwhm_px']:7.2f} px "
f"SNR={v['snr']:8.0f} normalized={v['normalized']:.4f}"
f"{' [pair]' if v['deblended'] else ''}{' [sat]' if v['saturated'] else ''}")
axes[1].set_ylabel('Black-corrected ADU'); axes[1].legend()
axes[2].set_ylabel('Normalized (detection only)'); axes[2].set_xlabel('Sensor column / px')
axes[2].set_ylim(-0.05, 1.18)
fig.savefig(out / 'candidates.png', dpi=160)
print('Hg I air wavelengths (nm):', HG)
print('Blended pairs such as 576.960/579.066 nm (about 7 px apart) are split by a joint '
'two-line fit and marked [pair]; use one channel per line.')
print('Select ONE channel candidate per line. Review candidates.png before confirming identities.')
text = args.assign
if not args.headless:
print('Inspect/zoom the plot, then CLOSE its window to enter confirmed identities in the terminal.')
plt.show(block=True)
if text is None:
text = input('Confirmed IDs:wavelength (comma separated), blank to export candidates only: ').strip()
plt.close(fig)
if not text:
print(f'Candidates exported to {out.resolve()}; no calibration fitted.')
return 0
confirmed = assignments(text, candidates, args.order)
coeff, residual = fit(confirmed, args.order, source.full_size[0])
x = np.array([r['pixel'] for r in confirmed])
lo, hi = float(x.min()), float(x.max())
info = dict(recording=str(Path(args.input).resolve()), frame_zero_based=args.frame,
roi=[roi.x0, roi.y0, roi.x1, roi.y1], pattern=source.pattern,
raw_format=source.raw_format, black_adu=black, full_scale_adu=full,
reduce=args.reduce, order=args.order, coefficients_descending=coeff.tolist(),
formula='wavelength_nm = numpy.polyval(coefficients_descending, sensor_x_px)',
wavelength_medium='air', reference=SOURCE, confirmed=confirmed,
residual_definition='fitted - reference (nm)', residuals_nm=residual.tolist(),
rms_nm=float(np.sqrt(np.mean(residual ** 2))), max_abs_nm=float(np.max(np.abs(residual))),
residual_degrees_of_freedom=len(x)-args.order-1, calibrated_pixel_range=[lo, hi],
processing='spatial reduction, black subtraction, morphological baseline for '
'peak detection, topographic prominence, Gaussian centres with '
'peak-top fallback and joint fits for blended doublets')
(out / 'calibration.json').write_text(json.dumps(info, indent=2), encoding='utf-8')
write_csv(out / 'residuals.csv', ['id', 'pixel', 'reference_nm', 'fitted_nm', 'residual_nm'],
([r['id'], r['pixel'], r['wavelength_nm'], r['wavelength_nm']+e, e] for r, e in zip(confirmed, residual)))
grid = np.arange(source.full_size[0])
write_csv(out / 'pixel_wavelength.csv', ['sensor_x_px', 'wavelength_nm', 'extrapolated'],
([int(p), float(w), int(p < lo or p > hi)] for p, w in zip(grid, np.polyval(coeff, grid))))
fig, ax = plt.subplots(2, 1, figsize=(10, 7), constrained_layout=True)
ax[0].plot(grid, np.polyval(coeff, grid)); ax[0].scatter(x, [r['wavelength_nm'] for r in confirmed])
ax[0].axvspan(lo, hi, alpha=.1, color='green', label='Confirmed span'); ax[0].legend()
ax[0].set_ylabel('Wavelength / nm (air)')
ax[1].axhline(0, color='gray'); ax[1].scatter(x, residual)
for r, e in zip(confirmed, residual):
ax[1].annotate(r['id'], (r['pixel'], e))
ax[1].set_ylabel('Fit - reference / nm'); ax[1].set_xlabel('Sensor column / px')
fig.savefig(out / 'calibration.png', dpi=160); plt.close(fig)
print('Coefficients (descending):', coeff)
print(f"RMS={info['rms_nm']:.6g} nm; max |residual|={info['max_abs_nm']:.6g} nm")
if len(x) == args.order + 1:
print('Exact interpolation: zero residual does not establish calibration accuracy; add independent lines.')
print(f'Calibration exported to {out.resolve()}')
return 0
def main(argv=None):
args = parser().parse_args(argv)
try:
return run(args)
except (ValueError, RuntimeError, OSError, IndexError, StopIteration, EOFError) as e:
print(f'Calibration error: {e or "frame outside recording"}', file=sys.stderr)
return 2
if __name__ == '__main__':
raise SystemExit(main())
+66
View File
@@ -0,0 +1,66 @@
{
"recording": "D:\\Dev\\Light\\Programs\\recordings\\hq_20261101_114417",
"frame_zero_based": 0,
"roi": [
98,
654,
848,
856
],
"pattern": "BGGR",
"raw_format": "BGGR_PISP_COMP1",
"black_adu": 4096.0,
"full_scale_adu": 65535.0,
"reduce": "mean",
"order": 1,
"coefficients_descending": [
0.2849422617422453,
349.4538237085669
],
"formula": "wavelength_nm = numpy.polyval(coefficients_descending, sensor_x_px)",
"wavelength_medium": "air",
"reference": "https://physics.nist.gov/PhysRefData/Handbook/Tables/mercurytable2.htm",
"confirmed": [
{
"id": "B1",
"channel": "B",
"sample": 103,
"pixel": 304.0,
"signal_adu": 9736.237623762376,
"normalized": 1.0,
"wavelength_nm": 435.8328
},
{
"id": "G1",
"channel": "G",
"sample": 294,
"pixel": 686.5,
"signal_adu": 54219.9801980198,
"normalized": 1.0,
"wavelength_nm": 546.0735
},
{
"id": "G2",
"channel": "G",
"sample": 355,
"pixel": 808.5,
"signal_adu": 17836.9900990099,
"normalized": 0.3289744856760633,
"wavelength_nm": 579.0663
}
],
"residual_definition": "fitted - reference (nm)",
"residuals_nm": [
0.24347127820948344,
-1.0068136053816943,
0.7633423271722677
],
"rms_nm": 0.7428870276938522,
"max_abs_nm": 1.0068136053816943,
"residual_degrees_of_freedom": 1,
"calibrated_pixel_range": [
304.0,
808.5
],
"processing": "spatial reduction, black subtraction; no smoothing; max normalization only for detection"
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

+9
View File
@@ -0,0 +1,9 @@
id,channel,sample,pixel,signal_adu,normalized
R1,R,36,301.0,1066.666666666667,0.2308984047019312
R2,R,89,407.0,1468.606060606061,0.31790512174643165
R3,R,285,799.0,4619.636363636364,1.0
R4,R,345,919.0,1852.121212121212,0.4009235936188077
G1,G,285,798.5,61439.0,1.0
B1,B,35,298.0,8451.878787878788,0.1652635570724308
B2,B,89,406.0,51141.818181818184,1.0
B3,B,285,798.0,9526.30303030303,0.1862722791050436
1 id channel sample pixel signal_adu normalized
2 R1 R 36 301.0 1066.666666666667 0.2308984047019312
3 R2 R 89 407.0 1468.606060606061 0.31790512174643165
4 R3 R 285 799.0 4619.636363636364 1.0
5 R4 R 345 919.0 1852.121212121212 0.4009235936188077
6 G1 G 285 798.5 61439.0 1.0
7 B1 B 35 298.0 8451.878787878788 0.1652635570724308
8 B2 B 89 406.0 51141.818181818184 1.0
9 B3 B 285 798.0 9526.30303030303 0.1862722791050436
Binary file not shown.

After

Width:  |  Height:  |  Size: 820 KiB

File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+4
View File
@@ -0,0 +1,4 @@
id,pixel,reference_nm,fitted_nm,residual_nm
B1,304.0,435.8328,436.0762712782095,0.24347127820948344
G1,686.5,546.0735,545.0666863946183,-1.0068136053816943
G2,808.5,579.0663,579.8296423271722,0.7633423271722677
1 id pixel reference_nm fitted_nm residual_nm
2 B1 304.0 435.8328 436.0762712782095 0.24347127820948344
3 G1 686.5 546.0735 545.0666863946183 -1.0068136053816943
4 G2 808.5 579.0663 579.8296423271722 0.7633423271722677
+137
View File
@@ -0,0 +1,137 @@
#!/usr/bin/env python3
"""Replay hq_record RAW recordings with saved wavelength calibration.
Keep next to hq_spectrum.py and spectro_core.py. All original playback switches
are supported.
"""
from __future__ import annotations
import argparse
from contextlib import redirect_stdout
import io
from pathlib import Path
import sys
import numpy as np
try: # shared core lives beside this script
import spectro_core as core
except ImportError as exc: # pragma: no cover - placement error
raise SystemExit('wavelength_player.py needs spectro_core.py in the same directory '
'(or on sys.path)') from exc
# Kept as the historical name: the player validates a saved calibration exactly
# like every other tool now does.
SavedCalibration = core.Calibration
def load_player():
# A private module instance provides hooks without editing the original file.
return core.load_hq()
def install_hooks(hq, calibration):
original_process = hq.process_frame
def process(raw, meta, roi, pattern, args, model, full_size, raw_format, bit_depth):
result = original_process(raw, meta, roi, pattern, args, model, full_size, raw_format, bit_depth)
# Retain signed black-subtracted ADU; normalized copies are detection only.
planes = hq.extract_planes(raw, pattern, roi)
for c in hq.CHANNELS:
values = (np.median(planes[c], axis=0) if args.reduce == 'median'
else np.mean(planes[c], axis=0, dtype=np.float64)) - result.black
values = values[:len(result.x_px[c])]
positive = np.maximum(values, 0)
norm = positive / positive.max() if positive.size and positive.max() > 0 else np.zeros_like(values)
result.signal_raw[c] = values
result.signal[c] = values.copy()
result.normalized[c] = norm
result.peaks[c] = hq.find_peaks_normalized(norm, result.x_px[c],
max(1, int(np.ceil(args.peak_distance / 2))), args.peak_prominence,
args.peak_height, args.max_peaks)
return result
class CalibratedDashboard(hq.Dashboard):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.ax_norm.set_title('Normalized profiles (detection only; no smoothing)', fontsize=9)
for axis in (self.ax_spec, self.ax_norm, self.ax_water):
axis.axvspan(*calibration.nm_span, alpha=.08, color='green', zorder=0)
def update(self, result, waterfall, fps, status_text):
# imshow needs an equally spaced nm grid, including quadratic/reverse mappings.
if waterfall.size:
gx = result.x_plot['G']
xmin = min(float(x.min()) for x in result.x_plot.values())
xmax = max(float(x.max()) for x in result.x_plot.values())
grid = np.linspace(xmin, xmax, waterfall.shape[1])
order = np.argsort(gx)
waterfall = np.asarray([np.interp(grid, gx[order], row[order]) for row in waterfall])
super().update(result, waterfall, fps, status_text)
def _target_y_limits(self, result):
# Signed signal (black already subtracted) can dip below zero, so the
# range is padded; rounding and hysteresis happen in Dashboard, which
# keeps the cached background valid instead of redrawing every frame.
valid = [v for v in result.signal.values() if v.size]
ymin = min(float(v.min()) for v in valid) if valid else 0.0
ymax = max(float(v.max()) for v in valid) if valid else 1.0
pad = max(1.0, (ymax - ymin) * 0.05)
return (hq._nice_floor(min(0.0, ymin - pad)),
hq._nice_ceiling(max(1.0, ymax + pad)))
hq.process_frame = process
hq.Dashboard = CalibratedDashboard
hq.make_calibration = lambda args: calibration
def main(argv=None):
argv = sys.argv[1:] if argv is None else list(argv)
options = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
options.add_argument('--calibration', default=str(Path(__file__).with_name('wavelength_calibration') / 'calibration.json'))
extra, rest = options.parse_known_args(argv)
try:
hq = load_player()
if '-h' in rest or '--help' in rest:
print('Wavelength recording player: wavelength_player.py --input REC_DIR --calibration calibration.json')
print('--calibration: saved JSON; --roi calibration: use saved ROI; default ROI is auto.')
print('No smoothing. Original player options follow:\n')
buffer = io.StringIO()
with redirect_stdout(buffer):
try:
hq.parse_args(['--help'])
except SystemExit:
pass
print(buffer.getvalue().replace('(default: 3)', '(default: 1)').replace(
'Live R/G/B spectral analysis of Raspberry Pi HQ Camera RAW frames.',
'Saved-calibration playback of HQ Camera RAW recordings.'))
print('Only recording playback is supported; --demo, --list-modes, --calibrate are unavailable.')
return 0
args = hq.parse_args(['--smooth', '1'] + rest)
if not args.input or args.demo or args.list_modes or args.calibrate:
raise ValueError('use --input REC_DIR and --calibration JSON; live/demo/--calibrate are unsupported')
if args.smooth != 1:
raise ValueError('this player preserves unsmoothed signal; --smooth must be 1')
calibration = SavedCalibration(extra.calibration)
source = hq.RecordingSource(args.input)
calibration.validate_geometry(source)
args.width, args.height = source.full_size
if args.roi == 'calibration':
roi = calibration.data['roi']
if len(roi) != 4 or any(not np.isfinite(float(v)) for v in roi):
raise ValueError('invalid saved ROI')
args.roi = ','.join(str(v) for v in roi)
print('Calibration:', calibration.path)
print('Descending coefficients:', calibration.coeff.tolist())
print(calibration.describe())
print('Use unchanged optics and sensor mode; each recording supplies its own black level.')
if args.headless and not args.frames:
print('Headless mode: processing all frames; use --frames N for a shorter run.')
install_hooks(hq, calibration)
return hq.run(args)
except (ValueError, KeyError, TypeError, OSError, RuntimeError) as exc:
print(f'Player error: {exc}', file=sys.stderr)
return 2
if __name__ == '__main__':
raise SystemExit(main())